Make Python module installable via 'pip install'

[seanmcleod]

I'd like to be able to pip install the JSBSim python module.

In particular I've been trying to install it as part of an Azure Notebooks which is a Linux based container.

It appears that the only methods they support for installing additional Python modules is via either pip install or conda install.

Installing Python Packages to Azure Notebooks

I had tried via shell access to download the .deb and unpackage the .so file into the notebook directory since that had worked in the past for me on Windows, i.e. simply having the .pyd in the same directory as the notebook. But it didn't work. Also tried copying it to a couple of other locations without success.

And lastly I tried installing the .deb package using dpkg but that requires root\sudo which isn't supported in Azure Notebooks.

So I think the ideal would be to have a Wheel .whl file available that can be used by pip install?

[seanmcleod]

Some potentially useful info on how to go about producing a Python Wheel package - https://stackoverflow.com/questions/24071491/how-can-i-make-a-python-wheel-from-an-existing-native-library

[bcoconni]

I just pushed the commit cd9ae08 that allows to build a wheel package for JSBSim. Run the following command in the build directory

> python python/setup.py bdist_wheel --skip-build

After the command has completed, you should find a wheel package (extension .whl) in the directory dist. The command pip install can be applied on this package

> pip install dist/JSBSim*.whl

[seanmcleod]

@bcoconni how is Make a Python package installable by pip different to the support you previously added for generating a .whl which you can then install via pip?

[bcoconni]

how is Make a Python package installable by pip different to the support you previously added for generating a .whl which you can then install via pip?

It is no longer building a wheel but a source distribution. The idea is to upload the package to PyPI so that anyone can install JSBSim via the simple command

> pip install jsbsim

This will trigger the download of the package from PyPI, the build and the installation locally. That way, Python users will no longer need to download the source code on GitHub, use CMake, etc.

Once the process will be stabilized, the idea is to automatize the process with Travis CI :smiley: However for that we will need some versioning.

[seanmcleod]

Will the option to generate a wheel file still be available?

[bcoconni]

Will the option to generate a wheel file still be available?

Yes. The commands to generate the wheel file are exactly the same and are still functional.

> cd build
> python python/setup.py bdist_wheel --skip_build

However, at the moment, the process to build the PyPI package is a bit convoluted since you need to go in build\python\package and run python setup.py sdist from there. The package is then built in build\python\package\dist

> cd build/python/package
> python setup.py sdist

So there are 2 setup.py, one in python for wheel files and one in python/package for PyPI packages.

I hope to unify the process later.

[bcoconni]

@seanmcleod I have just uploaded JSBSim to PyPI. So if you type pip install jsbsim from anywhere with an internet connection, the package should download, build and install. This package needs a C++ compiler to install successfully but I read that wheel packages can also be uploaded to PyPI.

With just a short command all the process takes place (download, build, install), Wow !! :smile:

If you create an account in PyPI, I will add you as a member of the project so that you can experiment with binary wheels in addition to the current source distribution.

[seanmcleod]

Excellent. My thinking is that there will be use cases where the wheel file will be more convenient than a source distribution.

In particular in using Azure notebooks I'd like the startup of the notebook to be as fast as possible so a pip install of a local wheel file in the notebook project directory when the notebook starts up takes only a second or two. But a source compile especially on the low performance freebie VM a number of minutes.

[seanmcleod]

@bcoconni does the source install from PyPI also require CMake to be installed?

I've created an account at PyPI - seanmcleod

[bcoconni]

By the way and for the record all the changes I made today are based on the excellent blog post from Martin Sosic Wrapping C/C++ library as a Python extension module.

That's an excellent introduction.

[bcoconni]

does the source install from PyPI also require CMake to be installed?

No they don't. That's part of the beauty of the thing :smile:

I've created an account at PyPI - seanmcleod

OK I've just added you as a maintainer. You should now get access to the project.

[bcoconni]

@seanmcleod If finally managed to modify our CI to build, check and release the Python wheel packages for the following platforms:

• Python 3.7 Windows 64 bits
• Python 2.7 Linux 64 bits
• Python 3.4 Linux 64 bits

They have all been tested to pip install successfully :smiley: Could you please test and let me know if that answer your request ?

[seanmcleod]

@bcoconni I fired up a clean/bare Windows virtual machine, installed python 3.7 x64 and then did a pip install of the latest wheel from our release. The pip install detected the numpy dependency and downloaded and installed numpy as well as JSBSim.

However import jsbsim fails with ImportError: DLL load failed: The specified module could not be found.

Will need to try and figure out which DLL is failing to load.

PS C:\Users\jsbsim> pip install https://github.com/JSBSim-Team/jsbsim/releases/download/JSBSim-trusty-v2018a/JSBSim-1.0.
0.dev1-cp37-cp37m-win_amd64.whl
Collecting JSBSim==1.0.0.dev1 from https://github.com/JSBSim-Team/jsbsim/releases/download/JSBSim-trusty-v2018a/JSBSim-1
.0.0.dev1-cp37-cp37m-win_amd64.whl
  Downloading https://github.com/JSBSim-Team/jsbsim/releases/download/JSBSim-trusty-v2018a/JSBSim-1.0.0.dev1-cp37-cp37m-
win_amd64.whl (624kB)
    100% |████████████████████████████████| 624kB 6.6MB/s
Collecting numpy (from JSBSim==1.0.0.dev1)
  Downloading https://files.pythonhosted.org/packages/dd/3e/0d7a914ee6cceef588dd83b18e257dc474ac67028a8d340dfec644878128
/numpy-1.16.0-cp37-cp37m-win_amd64.whl (11.9MB)
    100% |████████████████████████████████| 11.9MB 2.2MB/s
Installing collected packages: numpy, JSBSim
Successfully installed JSBSim-1.0.0.dev1 numpy-1.16.0
PS C:\Users\jsbsim>
PS C:\Users\jsbsim>
PS C:\Users\jsbsim> python
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import jsbsim
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed: The specified module could not be found.
>>>

[seanmcleod]

So I grabbed jsbsim.cp37-win_amd64.pyd from the installation and ran a dumpbin /DEPENDENTS on it and also on python37.dll.

Dump of file jsbsim.cp37-win_amd64.pyd
  Image has the following dependencies:

    python37.dll
    MSVCP140.dll
    WSOCK32.dll
    KERNEL32.dll
    VCRUNTIME140.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll
    api-ms-win-crt-environment-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-utility-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll

Dump of file python37.dll
  Image has the following dependencies:

    VERSION.dll
    SHLWAPI.dll
    WS2_32.dll
    KERNEL32.dll
    ADVAPI32.dll
    VCRUNTIME140.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-locale-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    api-ms-win-crt-environment-l1-1-0.dll
    api-ms-win-crt-process-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-conio-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll

I see VCRUNTIME140.dll is distributed with python, it's in the same directory as python37.dll, but I don't see MSVCP140.dll, so my guess this is the missing DLL preventing the JSBSim module from loading.

[seanmcleod]

Doing a quick search:

https://github.com/cztomczak/cefpython/issues/359

Then most probably this is caused, because you are missing "msvcp140.dll" dependency (for Python 3 for example). This is a dependency for Python C++ extensions

Explanation: msvcp140.dll is the DLL for the C++ runtime library. This dependency is added by Cython when building CEF Python module. It seems that Python 3.6 only ships "vcruntime140.dll" which is the DLL for the C runtime library.

[bcoconni]

@seanmcleod Interesting. If you confirm this is the cause of the bug then we will have to find a way to adjust the wheel package so that msvcp140.dll is automatically included in the package.

[seanmcleod]

I copied msvcp140.dll into the same directory as jsbsim.cp37-win_amd64.pyd and the JSBSim module now loads successfully.

Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import jsbsim
>>> fdm=jsbsim.FGFDMExec('.', None)

     JSBSim Flight Dynamics Model v1.0.0.dev1 [build 467/commit 983b9edb7d1b914c78e183a71ab8d7d7931aaec6] Jan 16 2019 22
:17:13
            [JSBSim-ML v2.0]

JSBSim startup beginning ...

>>> exit()

So as you mentioned we'll need to find out how to include msvcp140.dll in the wheel package. I did see people mentioning that you can grab msvcp140.dll from the AppVeyor environment.

[bcoconni]

So as you mentioned we'll need to find out how to include msvcp140.dll in the wheel package. I did see people mentioning that you can grab msvcp140.dll from the AppVeyor environment.

This has been fixed by the PR #159: msvcp140.dll is now shipped with the Python module. Could you please check that it works i.e. that you need no other installation than the wheel package to be able to import successfully JSBSim in Python ?

[seanmcleod]

Thanks @bcoconni, I've just tested again with a bare/fresh Windows VM with no issues.

Will this change in terms of bundling msvcp140.dll also work for the PyPI package? I took a look at PyPI and I see the last release is from Jan 6 2019 so it won't incorporate any of these changes even assuming they apply to the PyPI package setup.

[bcoconni]

I've just tested again with a bare/fresh Windows VM with no issues.

Excellent ! :tada:

Will this change in terms of bundling msvcp140.dll also work for the PyPI package? I took a look at PyPI and I see the last release is from Jan 6 2019 so it won't incorporate any of these changes even assuming they apply to the PyPI package setup.

At the moment, PyPI only has a source distribution (no binaries) so it is currently not a problem since the package will build locally (if the host is capable of compiling code of course) and use the libraries installed on the platform on which the package is built.

The plan is to add wheel packages to PyPI when JSBSim 1.0.0 will be released (see issue #153). pip will then check if a wheel package exists for the platform and if not, pip will download the source distribution and try to build locally the package.

Once we will start to upload wheel packages to PyPI, of course msvcp140.dll will be shipped with the Windows package.

[seanmcleod]

Yes I wasn't thinking straight. I was thinking in terms of the mixing of a source code PyPI distribution with a binary like msvcp140.dll and wondering whether that was supported. But if you have the msvc compiler to build the source code distribution then you'll also have msvcp140.dll already.

[agodemar]

I guess we should update our README.md adding a new section 'JSBSim usage with Python'.

[bcoconni]

I guess we should update our README.md adding a new section 'JSBSim usage with Python'.

Done with commit f13d8a7 (added a new section to install the Python module)