Adding initial Python bindings

[MichaelClerx]

From Andre/Hugh:

The idea is to have a "bindings/python" folder which would contain the SWIG config/interface files for generating python bindings for the libCellML API. This would also need to be added to the CMake build system so that the bindings can be built. Hugh has found some issues previous with other projects where CMake won't "notice" changes in the API header files and trigger the bindings to be remade when the API changes, but there are ways to work around that on the CI system and still a chance the CMake community will respond to his issues there.

• [x] Add infra + CMake scripts to generate bindings
• [x] Add infra + CMake scripts to generate tests
• [x] Add CMake scripts to generate Python bindings

Swig interfaces (and Python tests):

• [x] Entity (See #193)
• [x] NamedEntity
• [x] ImportedEntity (See #188)
• [x] Import(Source) (See #189, #194)
• [x] ComponentEntity (See #195, #198)
• [x] Component
• [x] Model
• [x] Units (See #203, #204)
• [x] Variable (See #202)
• [x] Error
• [x] Logger
• [x] Printer
• [x] Parser
• [x] Validator
• [x] version
• [x] enumerations
• [x] specification rule

Useful links:

• https://www.softwariness.com/articles/api-design-for-swig/

[hsorby]

To clarify what we are looking for directory structure wise is:

src/
     - bindings/
                - interface/
                - python/

the interface directory holds the interface files used by swig. The python directory holds just the CMake code required for generating Python bindings. It is desired to support other scripting languages so the interface files will (hopefully) be common to all.

[hsorby]

We could possibly break this issue down into

• [x] Adding SWIG interface files
• [x] Adding CMake Python bindings generation script

As getting started on Python bindings isn't really an issue.

[jonc125]

I've just seen a presentation by the Fenics team at a conference, and they're moving from Swig to pybind11 - might be worth a look?

Just chatted to Chris Richardson after the session. He says it's really easy to use, and well worth spending a couple of hours playing with - we'll "never look back"! So definitely worth some experimentation @MichaelClerx.

[MichaelClerx]

Does it support other languages @jonc125 ?

[jonc125]

I don't think so, just Python. But it therefore does Python very well! And at the moment we don't have anyone asking for bindings other than Python AFAIK.

[nickerso]

The current roadmap for libCellML based on community input (http://libcellml.readthedocs.io/en/latest/roadmap.html) lists the following languages in Milestone 1: C++, Python, Java, MATLAB. Milestone 5 then adds in JS, C#/.NET, C, Fortran[77|90|20XX].

I currently have to maintain my own C# bindings for integration into the SED-ML Web Tools, so I'd be happy to have "native" C# support from libCellML. JS is the other obvious one to help make it easier for the various web interfaces that we're working on.

I'd guess we're better off having a single solution that works "ok" for multiple target bindings than a collection of "perfect" language specific bindings? But we also want to make sure the Python bindings are good enough for python developers to use...

[jonc125]

I don't think SWIG can do all those languages in any case, can it? Pretty sure Fortran isn't supported, although the SWIG website does mention JS, C# and Octave, remarkably! Though how it can manage to do a runs-in-browser wrapper without cross-compiling the C++ to JS I'm not sure. SWIG does tend to produce less 'natural' wrappers, but is a decent starting point so you at least have something. But if we can do nicer wrappers for key languages without much effort (on the order of a few days) then I think it's worth doing.

[agarny]

I am also in favour of trying to get the best binding possible for any given language (or people might simply not bother using it, or very reluctantly; see libSBML with Java people).

[hsorby]

I think in the case of Python the bindings can be massaged to a desired outcome. I think SWIG is the way forward for this project as it can be used to target different languages, I think @nickerso would like to use C# bindings at some point but he can answer that for himself. I don't believe we need to focus on performance for the bindings so a slightly fatter interface between languages won't be much of a concern.

[hsorby]

I think we should have a fairly 1-1 relationship with the C++ header files for the binding and then use the package init file to make it possible to do things like:

import libcellml
u = libcellml.Units()

I have no problem with doing:

from libcellml.units import Units
u = Units()

I don't personally find anything wrong with this. But both ways can be accomodated with maniputlation of the init file (much like what requests does, which is a very good Python library). So in this case I think we can have the best of both worlds.

[MichaelClerx]

Agreed that the Python case is definitely solvable :-)

[hsorby]

As to the camelCase vs. snake_case argument I'm easy. The guidelines do mention that if the project is from a C++ background camelCase is acceptable. Having said that I'm not against working on a way that will convert the C++ camelCase to snake_case, I don't think this will be a big task.

I'm from a C++ background so tend to revert to camelCase naturally.

[MichaelClerx]

As to the camelCase vs. snake_case argument I'm easy.

Same here!

If it can be done automatically then sure, but as you say, camelCase is acceptable for a package that's obviously a wrapper around C++ (like PyQt).

[hsorby]

I think we should go for the option that we think will get the best take up and make libcellml easy to use, I just don't know which this is!

I think I would lean towards snake_case if I could generate the change from the C++ header files and avoid having to manually manipulate the change. But as the API should be fairly small I think this could be manually done without causing stomach ulcers.

[MichaelClerx]

Minor detail, but we'd have to trigger the change from the Python-specific SWIG files (or with a switch in the generic SWIG files) right? Because the Java bindings for example would be camelCased.

Thinking about the other languages, does the current directory structure src/bindings/language make sense for all languages? I'm finding it a bit odd to look for my Python files in build/src/bindings/python (but not very familiar with the cmake/swig combo so happy to accept that's just me!)

[hsorby]

My intention with the layout is to get all the SWIG interface files into one location which is nice and reusable. Don't forget that you can do conditional includes on target language so doing something just for Python isn't a problem.

http://www.swig.org/Doc1.3/Preprocessor.html#Preprocessor_condition_compilation

[MichaelClerx]

PR is ready!

[MichaelClerx]

Pfffft. Tests runs fine on my machine (Fedora), but fail for the PR. Anyone here running OS/X, Windows or Ubuntu 14?

[MichaelClerx]

@hsorby @nickerso et al., how do I go about finding the SWIG versions etc. installed on each of these machines?

I've been using SWIG 3.0.12 and GCC 7.1.1 to compile

[agarny]

I am running macOS 10.12, Windows 10 and Ubuntu 16.04. Let me know if you would like me to test anything specific.

[nickerso]

I've just been trying to build your branch @MichaelClerx and its not even trying to build the bindings. I have SWIG available, and the only thing I can see that is obviously missing is the PYTHON_LIBRARY_DEBUG and I'm trying to build a debug version of libCellML. Is that something I need to have or just an indication that something else has gone wrong with the configuration?

[hsorby]

@MichaelClerx if you go to the waterfall page on the Buildbot (which you can get to from the details link from the failed test notification):

http://autotest.bioeng.auckland.ac.nz/libcellml-buildbot/waterfall

you can look at the stdio log of the configure step. The information that you are looking for is in there.

[MichaelClerx]

@agarny could you have a look and see if the branch compiles and the tests run on your ubuntu machine?

[MichaelClerx]

@nickerso I've no idea what you mean, sorry! Is PYTHON_LIBRARY_DEBUG a CMAKE thing?

[agarny]

@agarny could you have a look and see if the branch compiles and the tests run on your ubuntu machine?

It's all working fine for me. Here is what I did:

• Clone https://github.com/MichaelClerx/libcellml.git;
• Check out your issue161 branch;
• Configure libCellML using the default settings;
• Build libCellML; and
• Run the tests.

[MichaelClerx]

Thanks! Seems it's only 14.04 with issues then... How do I go about finding out which SWIG version etc. is uses?

[agarny]

I guess you could temporarily tweak the main CMakeLists.txt file to show the SWIG version on Ubuntu 14.04?...

[MichaelClerx]

https://packages.ubuntu.com/trusty/swig

Says here it's 2.0.11, which is a shame as we're up to 3.0.12 now. I suppose it's a requirement that the code can be compiled on Ubuntu 14.04 without updates?

[agarny]

Personally, I would support the latest LTS (i.e. 16.04 at the moment) and that is it, which means that I would indeed update 14.04. I don't have access to that machine myself, but hopefully @hsorby does and he can update the version of SWIG on that machine.

[MichaelClerx]

One of the key elements of Swig 3.0.0 is that it adds C++11 support, so it might not be possible at all to do the bindings on 2.0 (since it won't be able to analyse the C++ code, regardless of how we write the interface).

[MichaelClerx]

Ok, can now confirm that if we leave the offending bits out of the interface files, swig instead starts complaining about the C++ header files themselves (it doesn't understand the syntax).

[MichaelClerx]

From the Ubuntu machine's cmake logs:

-- Found SWIG: /usr/bin/swig2.0 (found version "2.0.11") 

OS/X:

-- Found SWIG: /usr/local/bin/swig (found version "3.0.12")

Windows:

-- Found SWIG: C:/swigwin-3.0.12/swig.exe (found version "3.0.12")

So we know why the Ubuntu build fails so quickly, but still a mystery about the windows one!

[MichaelClerx]

Fixed the windows warnings, now trying to find out what these guys mean:

"D:\ALL_BUILD.vcxproj" (default target) (1) ->
"D:\src\bindings\python\_variable.vcxproj" (default target) (20) ->
  variablePYTHON_wrap.obj : error LNK2019: unresolved external symbol __imp__invalid_parameter_noinfo_noreturn referenced in function "public: char const & __cdecl std::_String_const_iterator<class std::_String_val<struct std::_Simple_types<char> > >::operator*(void)const " (??D?$_String_const_iterator@V?$_String_val@U?$_Simple_types@D@std@@@std@@@std@@QEBAAEBDXZ) [D:\src\bindings\python\_variable.vcxproj]
  D:\src\bindings\python\Debug\_variable.pyd : fatal error LNK1120: 1 unresolved externals [D:\src\bindings\python\_variable.vcxproj]

where I replaced D:\buildslave_libcellml\Unit_Tests_Builder_Windows_8_64_Bit\libcellml-build with D: for clarity

[MichaelClerx]

Something to do with the linked python binaries?

http://swig.10945.n7.nabble.com/link-error-in-debug-mode-td1247.html

[agarny]

Yes, it looks like you might not be linking some Python library. @dbrnz?

[MichaelClerx]

There are some lines in CMakeLists for the bindings for this, which @hsorby added originally:

https://github.com/cellml/libcellml/pull/205/commits/bb08e2fb50f06bf1bd605b84f708b81c7fa13453

I added a check (see above) and it seems that the windows machine definitely doesn't have the Python debug libraries (+ the detection works). However, the fix doesn't work! Maybe the machine has no Python libraries at all?

[agarny]

Something for @hsorby to confirm and for @dbrnz to give us more insight, if needed.

[MichaelClerx]

On Ubuntu 14, it should be possible to install swig 3.0 with just sudo apt-get install swig3.0

[hsorby]

I have updated the SWIG on Ubuntu 14.04. If this version fails I will shift the libcellml build to the Ubuntu 16.04 machine.

[hsorby]

I have tested on my own windows 10 machine and have found that:

With Python 2.7 without debug python libraries

• everything builds fine with no errors or warnings.
• one test fails (python_test_units) due to attempt to dereference a map/set iterator.

With Python 3.5 with debug python libraries

• everything builds fine with no errors or warnings.
• all the python tests fail because a dll load failed.

[hsorby]

It turns out that the CMake FindPythonInterp module does not find the debug python executable on windows (or on any platform actually). This means the python tests linked to the debug python libraries will never run when they try to use the release version of the python interpreter.

[MichaelClerx]

When would you use a special debug executable for Python?

Is this related? https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html (see deprecation notice for DEBUG_LIBS)

[MichaelClerx]

The Ubuntu machine can compile now, but the tests all fail due to a change between SWIG versions 3.0.2 (used on our machine) and 3.0.3 (released in december 2014).

I can think of a workaround, or we can just set the minimum version to 3.0.3 and update the machine... @hsorby ?

[hsorby]

You use the debug version of the executable when you want to debug :). In our case you would use it when you want to debug through the bindings.

No this is nothing to do with the Python libraries, those are found and used correctly it is the Python interpreter that is not working for us.

I think the workaround is to move the libcellml build to Ubuntu 16.04 and set that as the minimum version required for building libcellml. It is generally not easy to install packages that are not in the database and certainly not helpful for us to tell users that they need this particular package from an external source. Better just to upgrade the minimum version of Ubuntu I think.

[hsorby]

Sadly Ubuntu 14.04 is due to be supported right up until April 2019 which is a bit of a bummer, but oh well can't win them all.

https://www.ubuntu.com/info/release-end-of-life

[MichaelClerx]

Well, SWIG 3.0.2 converts enums such as enumerations.Format.XML to enumerations.XML, while later versions (starting at 3.0.3 I think) use enumerations.Format_XML. I can work around this by checking in the Python __init__ which constants exist and then renaming accordingly (I already do this to convert Format_XML to Format.XML for consistency with the C++ code and neater namespacing).

It wouldn't be perfect, because we still couldn't have A.X and B.X (since they'd both become X), but it would probably be ok.

Another option would be to stop using class enums altogether because they create more issues for the bindings elsewhere... (I'm usually a big fan of enums, esp. the way Java has them, but they're a feature that vary a lot from language to language, so...)

[hsorby]

If we do follow up the idea of removing the enumerations.h header and pushing enumerations elsewhere this wouldn't then apply to us? All enumerations would be namespaced.

[MichaelClerx]

No as long as they're class enums SWIGs behaviour will differ between version 3.0.2 and version 3.0.3 in the way described above.

For example Units.StandardUnit.AMPERE (C++) would become Units.AMPERE (Python, SWIG 3.0.2) and Units.StandardUnit_AMPERE (Python, SWIG 3.0.3+)

[MichaelClerx]

And the problems with languages that don't have typed enums will also persist as long as we use them

(Python can't distinguish an enum from an int, so if you have an overloaded method x(StandardUnit) and x(int) we can only make one of those methods available to the Python users)

[hsorby]

There has been some discussion around this int for enum or rather std::string for enum. It could well be that if we are using enumerations then that is the only option available through the API this would tidy up the conversion a little.

We can write some tests and see what it looks like then decide which style is preferred. I don't think there would be an issues around moving to a strictly no overloaded enumeration API.