Should SasModels include its own c compiler for independent use?

[toqduj]

When I install sasmodels, or rather when I ask other users to install sasmodels as a dependency for other code (McSAS3), it now gives problems on windows as it can't find a c compiler. For example:

  File "Y:/Measurements/SAXS002/mcsas3\mcsas3\mcmodel.py", line 412, in loadModel
    self.func = sasmodels.core.load_model(self.modelName, dtype=self.modelDType)
  File "Y:/Measurements/SAXS002/sasmodels\sasmodels\core.py", line 132, in load_model
    return build_model(load_model_info(model_name),
  File "Y:/Measurements/SAXS002/sasmodels\sasmodels\core.py", line 339, in build_model
    return kerneldll.load_dll(source['dll'], model_info, numpy_dtype)
  File "Y:/Measurements/SAXS002/sasmodels\sasmodels\kerneldll.py", line 301, in load_dll
    filename = make_dll(source, model_info, dtype=dtype)
  File "Y:/Measurements/SAXS002/sasmodels\sasmodels\kerneldll.py", line 278, in make_dll
    compile_model(source=filename, output=dll)
  File "Y:/Measurements/SAXS002/sasmodels\sasmodels\kerneldll.py", line 201, in compile_model
    raise RuntimeError("compile failed.\n%s\n%s"%(command_str, output))
RuntimeError: compile failed.
gcc -shared -std=c99 -O2 -Wall C:\Users\bpauw\AppData\Local\Temp\sas64_sphere_CB729A8A_1arakold.c -o C:\Users\bpauw\.sasmodels\compiled_models\sas64_sphere_CB729A8A.so -lm
'gcc' is not recognized as an internal or external command,
operable program or batch file.

I'm sure there are workarounds that even I might get to work at some point, but those are out of reach of a normal user with a simple use case (I can't really ask a casual user to install a c compiler on their system just to be able to try a package out). I know SasView comes with its own lightweight c compiler to support sasmodels. Is it perhaps possible to ship this c compiler with sasmodels itself so a pip install sasmodels or a git clone sasmodels will result in a working sasmodels installation?

I've now had to program a sphere model again internally so that I could at least use McSAS3 on various windows computers, but that really removes the power from having this awesome sasmodels library leveraged in the first place.

Would it be possible? It would greatly help adoption!

[butlerpd]

Good point - though if it is distributed on pypi would it not need to work on all platforms? It would be nice to do something about mac OS also so we don't have to keep telling people to install the tools?

[pkienzle]

On windows you can use pip install tinycc and sasmodels should find the tcc compiler. This won't work on Mac and there may be problems with some antivirus software on Windows. See https://github.com/SasView/sasview/issues/1146.

[pkienzle]

We can resolve this ticket by producing a better error message. Something like:

if os.name == 'nt':
    msg = 'No C compiler found. Try "pip install tinycc".'
elif sys.platform == 'darwin':
    msg = 'No C compiler found. Please install xcode command line tools.'
else:
    msg = 'No C compiler found. Install clang or gcc using your package manager.'
raise RuntimeError(msg)

Of course that requires scanning the error string from the compile command to determine that the compiler is missing and that it is not an error in the model code. We could maybe check that the command exists on the path before trying to compile. See shutil.which() for details.

[toqduj]

hey yea that seems to work fine on the windows systems. Wouldn't it be an idea to make tinycc a requirement for sasmodels install?

[pkienzle]

We opted to make it a dependency for the application (sasview) rather than the library (sasmodels).

tinycc is not an optimizing compiler so I expect it will be slower than the msvc or gcc. I only wanted to install it if there was no compiler available. We could instead check if msvc or gcc is available and use them in preference to tinycc. Then it doesn't matter if tinycc is installed.

[pkienzle]

The code would be something like the following. Somebody with a windows box will need to test it.

elif os.name == "nt":
    if shutil.which("cl") is not None:
        COMPILER = "msvc"
    elif shutil.which("gcc") is not None:
        COMPILER = "mingw"
    elif tinycc is not None:
        COMPILER = "tinycc"
    else:
        # Even if the user installed msvc sasmodels won't see it if it is not on the path. This is done by
        # calling vcvarsall.bat. This has to happen in the same command window used to launch
        # the python application. If using sasmodels from a GUI application it may not find msvc. For
        # MinGW the bin directory needs to be on the path, either local to the command window or on
        # the system path.
        raise RuntimeError("No C compiler found. Try 'pip install tinycc' or install Microsoft Visual C for Python or install MinGW. For MSVC be sure to set the compiler environment with VCVARSALL.bat. For MinGW, be sure to put the MinGW bin directory on the system path.")

This replaces: https://github.com/SasView/sasmodels/blob/26813653223cffe3939c791f1ec92a08d556e055/sasmodels/kerneldll.py#L114-L124

-- [Update: include code comment re: vcvarsall.]

[pkienzle]

Note that the above will affect sasview as well. If there is a broken msvc or gcc on the system then sasmodels may try to use it instead of the tinycc we supply with the package. We can force it by setting the following in sasview startup:

if os.name == "nt" and os.environ.get("SAS_COMPILER", None) is None:
    os.environ["SAS_COMPILER"] = "tinycc"

Users who care can still force the external compiler by setting the SAS_COMPILER environment variable but it will otherwise default to using tinycc regardless of what is installed on the system.

We could extend the GPU selection window in SasView to include testing with cl or gcc if we can find them on the system.

We ought to check whether their is a substantial performance gain from using cl or gcc before doing too much work.

Also note that OpenMP is not used by default. Set SAS_OPENMP=1 in the environment to see if it makes a difference. It would not be surprising to see a 16x speedup with msvc compared to tinycc on a four-core cpu with four-way SSE instructions.

[andyfaff]

You could just use distutils/setuptools to figure out what ccompilers are available on a system: https://docs.python.org/3/distutils/apiref.html, e.g. distutils.ccompiler.show_compilers()

[pkienzle]

This just shows the compilers that distutils knows how to work with, not the ones that are actually available. On my mac it produces:

>>> distutils.ccompiler.show_compilers()
List of available compilers:
  --compiler=bcpp     Borland C++ Compiler
  --compiler=cygwin   Cygwin port of GNU C Compiler for Win32
  --compiler=mingw32  Mingw32 port of GNU C Compiler for Win32
  --compiler=msvc     Microsoft Visual C++
  --compiler=unix     standard UNIX-style compiler

I suspect if it were to report a compiler it would be the compiler used to compile python which may not be the compiler that is available on your system.

[andyfaff]

I use something like this in setup.py to create a compiler:

    try:
        from setuptools._distutils.ccompiler import new_compiler
        from setuptools._distutils.sysconfig import customize_compiler
    except ImportError:
        return False

    ccompiler = new_compiler()
    customize_compiler(ccompiler)
    ccompiler.compile(["myfile.c"])