Deploying freeSASA python library through PyPi or Anaconda

[molsim]

Hi Simon, I'm now trying to streamline deployment of https://github.com/ncbi/HYDROID to different platforms by putting it to PyPi and Anaconda cloud. Do you think that it might be possible to make freesasa python library available through PyPi and/or Anaconda? Thanks in advance, Alex

[mittinatten]

Hi Alex, That would be great, but I have no experience with packaging for PyPi, and only limited experience with Python in general.

I had a quick look at the PyPi guidelines, and it seems the project structure is not compatible with the C-centric one I have here. So probably the most straightforward solution would be to create a separate repo that has the original FreeSASA as a submodule?

I'm very soon off on holidays, but I can have a look at it in January. If you want to have a go, I could create a repo under https://github.com/freesasa, and add you as co-owner.

Simon

[molsim]

Hi Simon, thanks a lot for the answer! if you could look into PyPi (setuptools) in January - this would be great. I have no experience in wrapping C++ dependencies (Extension modules) into setuptools, so I'm not sure how exactly this can be done for FreeSASA - but if it turns out to be possible - I'll be happy to contribute!

Meanwhile, I found that Anaconda may provide an easier solution compatible with the freesasa current structure. Here I'm developing a recipe for making a binary Anaconda module from FreeSASA https://github.com/intbio/freesasa-conda

My goal is to try to make cross-platform installation OSX/Linux/Win straightforward and easy for the end user.

Have nice holidays, Alex

[mittinatten]

Great! I see you use 2.0.1. One of the main changes in 2.0.2 is to update some of the C code that wasn't compatible with the MS C Compiler, so for Windows users it might help to have the latest version. (I don't know which compiler Python uses to build packages on Windows, or how that works at all)

[molsim]

Very nice! I was just trying to figure out how to compile freesasa on/for Windows. Do you have a recipe for that? What to do with autoconf/make tools?

On Tue, Dec 19, 2017 at 11:12 AM Simon Mitternacht notifications@github.com wrote:

Great! I see you use 2.0.1. One of the main changes in 2.0.2 is to update some of the C code that wasn't compatible with the MS C Compiler, so for Windows users it might help to have the latest version. (I don't know which compiler Python uses to build packages on Windows, or how that works at all)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mittinatten/freesasa/issues/34#issuecomment-352669989, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYPlAtsNkPYOc2fUqxQB__8ZtfEw-pVks5tB2_WgaJpZM4RFM6E .

[mittinatten]

I have no experience with development in Windows myself, I only made som adjustments to the code based on what people had problems with (see release notes https://github.com/mittinatten/freesasa/releases/tag/2.0.2, and the discussion here https://github.com/mittinatten/freesasa/issues/22).

So, I'm not sure what to do about autotools, unfortunately. It should be pretty straightforward to simply compile all the files in the src-directory, possibly excluding json.c and xml.c and setting the corresponding defines manually as compiler flags.

Cython does do its own compilation here: https://github.com/mittinatten/freesasa/blob/master/bindings/python/setup.py.in. As far as I understand, at the moment the pyx-file is converted to a C-file, and then compiled and linked with libfreesasa.a. Maybe you can get Cython to compile everything in a separate setup.py? See: https://github.com/cython/cython/wiki/InstallingOnWindows. As I wrote, I am not a Python expert, so I'm uncertain about some of the details here. Any input is welcome!

[mittinatten]

RDKit uses CMake for FreeSASA, perhaps that could be adapted as well.

https://github.com/rdkit/rdkit/blob/master/External/FreeSASA/CMakeLists.txt

[molsim]

Thanks a lot, I’ll dig into that!

[molsim]

So after digging a bit into the problem, here are some findings: 1) FreeSASA executable can be compile on windows by emulating Linux environment (install bash, auto tools, gcc, etc.) through M2 project available through Anaconda https://www.oliversherouse.com/2017/01/03/unix-anaconda-m2.html 2) However, to compile python bindings for FreeSASA in conda one needs to use Microsoft Compiler. I was experimenting with MSVC++ for Python 2.7 https://www.microsoft.com/en-us/download/details.aspx?id=44266 But trying to compile select files in src dirs gives this kind of error: \AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Include\stdlib.h(610) : error C2485: '__restrict' : unrecognized extended attribute classifier.c(9) : fatal error C1083: Cannot open include file: 'strings.h': No such file or directory

[mittinatten]

Hmm, sounds like we got the version numbers wrong here https://github.com/mittinatten/freesasa/blob/master/src/freesasa_internal.h#L31

Which version of MSVC are you using? Perhaps the latest version includes the standard restrict?

[molsim]

I'm using Microsoft (R) C/C++ Optimizing Compiler Version 15.00.30729.01 for x64

[mittinatten]

What happens if you change line 31 in src/freesasa_internal.h from

#if defined(_MSC_VER) && _MSC_VER >= 1400

to

#if defined(_MSC_VER) && _MSC_VER >= 1400 && _MSC_VER < 1500

It is strange that you get the strings.h error, since that header is only included if HAVE_STRINGS_H is defined and evaluates to true.

[molsim]

Ok, that was my mistake, I tried to substitute config.h generate on linux for the one I cannot generate on windows.

[mittinatten]

I've been experimenting a bit, and managed to create a distutils file that builds without any involvement of autotools, so it will use whatever C compiler is associated with Python. Since the xml and json libraries aren't used anyway by the Python module, I've simply excluded them by default.

I'm not sure how I want to structure the package yet (one idea could be to let the Python bindings be a separate repo, and include the C code as a submodule), but this might be of interest to you too, so here goes:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

sources = map(lambda file: "../../src/" + file,
              ["classifier.c",
               "classifier_protor.c", "classifier_oons.c", "classifier_naccess.c",
               "coord.c", "freesasa.c", "lexer.c", "log.c",
               "nb.c", "node.c", "parser.c", "pdb.c", "rsa.c",
               "sasa_lr.c", "sasa_sr.c", "selection.c", "structure.c",
               "util.c"])

sources.append("*.pyx")

extensions = [
    Extension("*", sources,
              include_dirs = ["../../src"],
              language='c',
              extra_compile_args = ["-DUSE_THREADS=1", "-DUSE_XML=0",
                                    "-DUSE_JSON=0", "-DUSE_CHECK=0"]
          )
]

setup(
    name='FreeSASA',
    description='Calculate solvent accessible surface areas of proteins',
    version= '2.0.2',
    author='Simon Mitternacht',
    url='http://freesasa.github.io/',
    license='MIT',
    ext_modules = cythonize(extensions)
)

[mittinatten]

There are probably some compiler flags I forgot, and this just happened to work on my machine, so a better solution could possibly be to provide a a custom config.h, like you did, instead of the extra_compile_args, and add a separate directory for it in include_dirs. Or let Python detect somehow what libraries are available and give compiler flags based on that.

[satary]

Hello Simon, I am trying to create python bindings for freesasa and python2.7 using Microsoft Visual C++ Compiler for Python 2.7 (that is version 15.00.x, which is from visual studio 2008). I have to use it as only this version is compatible with windows version of python 2.7 (i guess...). But I can't compile bindings or standalone version of freesasa due to tons of syntax errors (see below). Google suggests that it is an incompatibility between C dialects (msvc2008 understands C89). What do you think about this problem? Thanks, Grisha P.S. I've managed to compile freesasa using visual studio 2015 (there were some problems with getopt), but really need 2008 for python 2.7

>cl freesasa.c

Microsoft (R) C/C++ Optimizing Compiler Version 15.00.30729.01 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

freesasa.c
freesasa.c(39) : error C2059: syntax error : '.'
freesasa.c(87) : error C2143: syntax error : missing ';' before 'type'
freesasa.c(88) : error C2143: syntax error : missing ';' before 'type'
freesasa.c(90) : error C2065: 'result' : undeclared identifier
freesasa.c(90) : warning C4047: '==' : 'int' differs in levels of indirection fr
om 'void *'
freesasa.c(99) : error C2065: 'ret' : undeclared identifier
freesasa.c(99) : error C2065: 'result' : undeclared identifier
freesasa.c(99) : error C2223: left of '->sasa' must point to struct/union
freesasa.c(99) : warning C4090: 'function' : different 'const' qualifiers
freesasa.c(99) : warning C4133: 'function' : incompatible types - from 'const do
uble *' to 'const coord_t *'
freesasa.c(99) : warning C4133: 'function' : incompatible types - from 'const fr
eesasa_parameters *' to 'const double *'
freesasa.c(99) : error C2198: 'freesasa_shrake_rupley' : too few arguments for c
all
freesasa.c(102) : error C2065: 'ret' : undeclared identifier
freesasa.c(102) : error C2065: 'result' : undeclared identifier
freesasa.c(102) : error C2223: left of '->sasa' must point to struct/union
freesasa.c(102) : warning C4090: 'function' : different 'const' qualifiers
freesasa.c(102) : warning C4133: 'function' : incompatible types - from 'const d
ouble *' to 'const coord_t *'
freesasa.c(102) : warning C4133: 'function' : incompatible types - from 'const f
reesasa_parameters *' to 'const double *'
freesasa.c(102) : error C2198: 'freesasa_lee_richards' : too few arguments for c
all
freesasa.c(108) : error C2065: 'ret' : undeclared identifier
freesasa.c(109) : error C2065: 'result' : undeclared identifier
freesasa.c(109) : warning C4047: 'function' : 'freesasa_result *' differs in lev
els of indirection from 'int'
freesasa.c(109) : warning C4024: 'freesasa_result_free' : different types for fo
rmal and actual parameter 1
freesasa.c(113) : error C2065: 'result' : undeclared identifier
freesasa.c(113) : error C2223: left of '->total' must point to struct/union
freesasa.c(114) : error C2143: syntax error : missing ';' before 'type'
freesasa.c(114) : error C2143: syntax error : missing ';' before 'type'
freesasa.c(114) : error C2143: syntax error : missing ')' before 'type'
freesasa.c(114) : error C2143: syntax error : missing ';' before 'type'
freesasa.c(114) : error C2065: 'i' : undeclared identifier
freesasa.c(114) : warning C4552: '<' : operator has no effect; expected operator
 with side-effect
freesasa.c(114) : error C2059: syntax error : ')'
freesasa.c(114) : error C2065: 'i' : undeclared identifier
freesasa.c(114) : error C2143: syntax error : missing ';' before '{'
freesasa.c(115) : error C2065: 'result' : undeclared identifier
freesasa.c(115) : error C2223: left of '->total' must point to struct/union
freesasa.c(115) : error C2065: 'result' : undeclared identifier
freesasa.c(115) : error C2223: left of '->sasa' must point to struct/union
freesasa.c(115) : error C2065: 'i' : undeclared identifier
freesasa.c(117) : error C2065: 'result' : undeclared identifier
freesasa.c(117) : error C2223: left of '->parameters' must point to struct/union

freesasa.c(119) : error C2065: 'result' : undeclared identifier
freesasa.c(119) : warning C4047: 'return' : 'freesasa_result *' differs in level
s of indirection from 'int'
freesasa.c(132) : error C2143: syntax error : missing ';' before 'type'
freesasa.c(133) : error C2275: 'freesasa_result' : illegal use of this type as a
n expression

[mittinatten]

Hmm, I use some C99 specific things, such as // comments, and declaring variables inside for loops. I guess those could cause such widespread errors. Are there any flags in visual studio to select C99?

On Thu, 11 Jan 2018 at 18:35, satary notifications@github.com wrote:

Hello Simon, I am trying to create python bindings for freesasa and python2.7 using Microsoft Visual C++ Compiler for Python 2.7 (that is version 15.00.x, which is from visual studio 2008). I have to use it as only this version is compatible with windows version of python 2.7 (i guess...). But I can't compile bindings or standalone version of freesasa due to tons of syntax errors (see below). Google suggests that it is an incompatibility between C dialects (msvc2008 understands C89). What do you think about this problem? Thanks, Grisha P.S. I've managed to compile freesasa using visual studio 2015 (there were some problems with getopt), but really need 2008 for python 2.7

cl freesasa.c

Microsoft (R) C/C++ Optimizing Compiler Version 15.00.30729.01 for x64 Copyright (C) Microsoft Corporation. All rights reserved.

freesasa.c freesasa.c(39) : error C2059: syntax error : '.' freesasa.c(87) : error C2143: syntax error : missing ';' before 'type' freesasa.c(88) : error C2143: syntax error : missing ';' before 'type' freesasa.c(90) : error C2065: 'result' : undeclared identifier freesasa.c(90) : warning C4047: '==' : 'int' differs in levels of indirection fr om 'void ' freesasa.c(99) : error C2065: 'ret' : undeclared identifier freesasa.c(99) : error C2065: 'result' : undeclared identifier freesasa.c(99) : error C2223: left of '->sasa' must point to struct/union freesasa.c(99) : warning C4090: 'function' : different 'const' qualifiers freesasa.c(99) : warning C4133: 'function' : incompatible types - from 'const do uble ' to 'const coord_t ' freesasa.c(99) : warning C4133: 'function' : incompatible types - from 'const fr eesasa_parameters ' to 'const double ' freesasa.c(99) : error C2198: 'freesasa_shrake_rupley' : too few arguments for c all freesasa.c(102) : error C2065: 'ret' : undeclared identifier freesasa.c(102) : error C2065: 'result' : undeclared identifier freesasa.c(102) : error C2223: left of '->sasa' must point to struct/union freesasa.c(102) : warning C4090: 'function' : different 'const' qualifiers freesasa.c(102) : warning C4133: 'function' : incompatible types - from 'const d ouble ' to 'const coord_t ' freesasa.c(102) : warning C4133: 'function' : incompatible types - from 'const f reesasa_parameters ' to 'const double ' freesasa.c(102) : error C2198: 'freesasa_lee_richards' : too few arguments for c all freesasa.c(108) : error C2065: 'ret' : undeclared identifier freesasa.c(109) : error C2065: 'result' : undeclared identifier freesasa.c(109) : warning C4047: 'function' : 'freesasa_result ' differs in lev els of indirection from 'int' freesasa.c(109) : warning C4024: 'freesasa_result_free' : different types for fo rmal and actual parameter 1 freesasa.c(113) : error C2065: 'result' : undeclared identifier freesasa.c(113) : error C2223: left of '->total' must point to struct/union freesasa.c(114) : error C2143: syntax error : missing ';' before 'type' freesasa.c(114) : error C2143: syntax error : missing ';' before 'type' freesasa.c(114) : error C2143: syntax error : missing ')' before 'type' freesasa.c(114) : error C2143: syntax error : missing ';' before 'type' freesasa.c(114) : error C2065: 'i' : undeclared identifier freesasa.c(114) : warning C4552: '<' : operator has no effect; expected operator with side-effect freesasa.c(114) : error C2059: syntax error : ')' freesasa.c(114) : error C2065: 'i' : undeclared identifier freesasa.c(114) : error C2143: syntax error : missing ';' before '{' freesasa.c(115) : error C2065: 'result' : undeclared identifier freesasa.c(115) : error C2223: left of '->total' must point to struct/union freesasa.c(115) : error C2065: 'result' : undeclared identifier freesasa.c(115) : error C2223: left of '->sasa' must point to struct/union freesasa.c(115) : error C2065: 'i' : undeclared identifier freesasa.c(117) : error C2065: 'result' : undeclared identifier freesasa.c(117) : error C2223: left of '->parameters' must point to struct/union

freesasa.c(119) : error C2065: 'result' : undeclared identifier freesasa.c(119) : warning C4047: 'return' : 'freesasa_result *' differs in level s of indirection from 'int' freesasa.c(132) : error C2143: syntax error : missing ';' before 'type' freesasa.c(133) : error C2275: 'freesasa_result' : illegal use of this type as a n expression

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mittinatten/freesasa/issues/34#issuecomment-357002910, or mute the thread https://github.com/notifications/unsubscribe-auth/ACteHC5bXF_hDC5dUatIDr41rIFjet-mks5tJkZkgaJpZM4RFM6E .

[satary]

Thanks for the reply!

Are there any flags in visual studio to select C99?

I have not managed to find anything related to c99 in MS documentation. Forcing C++ for code treatment changed the amount of errors =) But the first errors are still there.

cl /Tp freesasa.c
Microsoft (R) C/C++ Optimizing Compiler Version 15.00.30729.01 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.
freesasa.c
freesasa.c(39) : error C2143: syntax error : missing '}' before '.'
freesasa.c(39) : error C2143: syntax error : missing ';' before '.'
freesasa.c(39) : error C2059: syntax error : '.'
freesasa.c(44) : error C2143: syntax error : missing ';' before '}'
freesasa.c(44) : error C2059: syntax error : '}'
freesasa.c(49) : error C2440: 'initializing' : cannot convert from 'void *' to 'freesasa_result *'   Conversion from 'void*' to pointer to non-'void' requires an explicit cast
freesasa.c(56) : error C2440: '=' : cannot convert from 'void *' to 'double *'    Conversion from 'void*' to pointer to non-'void' requires an explicit cast

[mittinatten]

Here’s a relevant article I found, although it doesn’t help much I guess: https://herbsutter.com/2012/05/03/reader-qa-what-about-vc-and-c99/

Have you tried using MinGW as recommended by Cython? https://github.com/cython/cython/wiki/InstallingOnWindows

[satary]

I am a bit lost with that compiler differences as I am a linux user. I can try and use MinGW, but can it be used to create a compatible library? There is also no 64 bit version of MinGW for Windows. In Python documentation it is stated that Visual C++ 9.0 should be used for python 2.7 https://wiki.python.org/moin/WindowsCompilers Christoph Gohlke also use it to build python extensions for python 2.7 https://www.lfd.uci.edu/~gohlke/pythonlibs/

The majority of errors i get with Visual C++ 9.0 (if i force C++) is due to designated initializers and implicit casts, I will attempt to remedy that...

[mittinatten]

I also do all my development on unix-like systems (linux and mac os) so my knowledge is limited to what I can find with Google :)

I realize now that the link I sent you was for building Cython, so MinGW probably won't work for Python 2.7. For Cython extensions this is the relevant documentation: https://github.com/cython/cython/wiki/CythonExtensionsOnWindows. It seems there are two options, the one with VC++ 9.0, and another one with the full Visual Studio SDK (see https://github.com/cython/cython/wiki/CythonExtensionsOnWindows#using-windows-sdk-cc-compiler-works-for-all-python-versions), could be possible.

The last 2 errors in your log above are expected when compiling C code with C++ I think, since C++ is more picky about type conversions. The other two errors are also incompatibilities between C99 and C++ https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html (i.e. declaring a struct instance the way that is done in lines 39-44, see the point example in the link).

I also saw you mentioned you had problems with getopt earlier, to build the Python module you don't need to compile src/main.c which is the only file that uses getopt.

Another option could perhaps be to build the library as a dynamicly linked library (.dll-file) and include it that way in Python module. Not sure what's possible and what's not on Windows. These are instructions for Unix: https://stackoverflow.com/questions/16993927/using-cython-to-link-python-to-a-shared-library, can probably be translated to Windows.

[mittinatten]

The extern "C" declaration I have in freesasa.h for C++ (https://github.com/mittinatten/freesasa/blob/master/src/freesasa.h#L84) could possibly help in the source files too. At least for some of the errors.

[mittinatten]

Hi again, Here's something you might be able to use: https://github.com/freesasa/freesasa-python

I have only tested it on my mac, but at least the module compiles and passes the tests. I think all GNU-specific things are gone. Could you test it on windows when you have a chance? And then I'll try to push it to PyPi.

Simon

[mittinatten]

For Python 3.5+ it probably won't be a problem building on Windows (but I guess that won't help you):

• https://wiki.python.org/moin/WindowsCompilers
• https://blog.ionelmc.ro/2014/12/21/compiling-python-extensions-on-windows/

If we could get that to work, we might be able to bundle the Windows binaries with the PyPi package.

[mittinatten]

A test https://test.pypi.org/project/freesasa/

[mittinatten]

The dev branch is now ANSI C, so should theoretically work with Python 2.7 in Windows too. The pypi-test package, mentioned above, points to that version too.

[mittinatten]

This is now the main library for Python bindings, https://github.com/freesasa/freesasa-python, and it deploys binaries for Windows and Mac to PyPi for a release. So pip install freesasa should work on most machines.

Closing this now, reopen if you have any problems.

[satary]

Thank you very much! Everything works like a charm!