Upload pygmo2 source to PyPI as well as wheels for recent versions

[astrojuanlu]

(Comes from https://github.com/esa/pagmo/issues/213)

I see that the latest release of pygmo available on PyPI only has precompiled wheels for Python 3.6, 3.7, and 3.8, and no source distribution:

https://pypi.org/project/pygmo/2.16.1/#files

On the other hand, there is a pagmo 2.18.0 version tagged on GitHub, but it doesn't seem to be available on GitHub.

This makes it impossible to install pygmo anywhere else from PyPI. I see that the repository hasn't had much activity recently, but at least would you consider generating a sdist and upload it to PyPI? Having precompiled wheels would be a plus.

I see https://github.com/esa/pagmo2/pull/293 implemented a Circle CI workflow to build manylinux wheels. I also saw that @bluescarni has been creating several conda feedstocks, which is awesome 😊 In the documentation you say

Due to a lack of manpower, we are currently unable to provide pip packages for Windows or OSX. If you are willing to help us out, please get in contact with us on the gitter channel or (even better) open a PR on github.

Are you still open to PRs for PyPI wheels, or focusing mostly on conda? Asking this before doing the work because it will be quite intensive and I don't want to waste effort by not aligning myself with the project maintainers first.

[bluescarni]

Hi @astrojuanlu , sorry for the late reply, I was on vacation.

Yes we are interested in having up-to-date wheels and, possibly, sdist packages on PyPI. I think I now know how to do that, as I have been using https://scikit-build.readthedocs.io/en/latest/index.html at work and it seems to work well as a bridge between CMake and the setup.py machinery, allowing to minimise duplication between different build systems.

We have focused our packaging efforts on conda-forge (where pagmo/pygmo 2.18 packages are available), since pagmo and pygmo are mostly C++ codebases with several non-trivial dependencies which are best managed with a cross-language package manager.

I see that the repository hasn't had much activity recently

Indeed pagmo/pygmo development is in a bit of a pause, and at the moment I am just making sure that the conda-forge packages remain usable and up-to-date. We plan with @darioizzo to return to more active pagmo/pygmo development soon.

[bluescarni]

I am transferring the issue over to the pygmo repo.

[bluescarni]

@astrojuanlu now here.

[astrojuanlu]

Thanks for your response @bluescarni. The reason I opened this issue is because I'm interacting with a project that combines GODOT/GODOTPY (through Space CODEV) and pygmo, and I saw that these are leveraging scikit-build successfully. I suppose your plan is to add a setup.py to the root of https://github.com/esa/pygmo2 with the required instructions, correct?

[bluescarni]

I am actually working on GODOT/GODOTPY :)

Yes the plan would be to add a setup.py. There's some extra steps to make the sdist work (e.g., need to add a manifest listing the C++ source files/dirs).

If you have access to the latest (unreleased) GODOTPY, I rewrote and simplified the build system and the setup.py now is just a handful of lines, and it supports out of the box building both wheels and sdist packages. That would be my starting point to improve and update the pypi packages for pygmo2.

[astrojuanlu]

Yes, I saw :)

I would send a pull request myself but I don't want to get into licensing trouble, so I will keep an eye on the repo for future updates.

[Hvass-Labs]

Hello, I have used the pygmo Python package in a number of projects, some of which can be run on Google Colab. The pygmo package is causing problems because it is only supported until Python 3.8 and hasn't been updated for nearly 2 years.

May I ask what is the reason that you are only making packages available for specific Python versions? Can't you just make a package available for Python 3 in general so we don't have this problem in the future?

Thanks!

[darioizzo]

Dear @Hvass-Labs pygmo is a C++ package wrapped to python using a binding technology (pybind11). As a consequence you will agree it is not possible to make a python 3 generic package and at each python version a new binary must be compiled.

Contrary to what you think pygmo is updated regularly and distributed via conda. Any reason not to use conda in your case? We stopped supporting pip distribution since a while as the package manager pip was not born to support easily C++ packages linking to third party libraries, often causing breaking combinations of packages.

For pure python pip is still a good choice ... but then pure python can be rather limited and slow.

[astrojuanlu]

Thanks for chiming in @darioizzo. If it's official that pygmo will not support pip going forward we can adapt. But my impression was that it was open for contributions somehow.

[darioizzo]

@astrojuanlu absolutely, sorry for the confusion, we are opened to receive a PR restoring the wheel functionality for newer python versions, with the understanding that wheels shipped like this will always have to be used at the risk of the user. If documented this is ok.

We are not working on such a feature though, as we do not have the bandwidth to!

[Hvass-Labs]

Thanks for the quick and detailed explanation!

I have no personal experience combining Python and C++ code and distributing the library, so I will have to take your word for it, that it requires a new compilation for each new Python version. But is this really a big problem? They don't release new major versions of Python that often.

Let me give a few reasons why I think it would be wise of you to continue supporting newer Python versions through pip installation:

• pip is still the standard for installing Python packages in spite of its short-comings. (Personally, although I do use Anaconda to manage my virtual environments, I never install Python packages using Anaconda, I always use pip.)

• The pygmo package gets over 200 installs per day using pip, see here. That's over 6000 pip installs per months, and over 72000 pip installs per year! That is quite a lot and you're probably not aware of its popularity! A while ago it ranged between 500-1500 pip installs per day. My guess is that it dropped significantly to "only" 200 pip installs per day because you don't support newer Python versions and that is causing a lot of problems.

• Just to show you how popular pip installs still are: The numpy package gets 2.5-5 million pip installs per day - and they are still growing! See here.

• A lot of these pip installs are probably automated installations, including other packages that use numpy. This is a lot easier to setup with ordinary Python than with Anaconda. In your personal use-case Anaconda may be a great choice, but for a lot of other use-cases ordinary Python and pip installation is much easier.

• Google Colab also doesn't support Anaconda easily. A lot of my work is made to be run on Google Colab so people don't have to install anything on their local machine just to try my source-code. It looks like they are still running Python 3.7 but eventually they will upgrade to newer Python versions and then it will be a big hassle to install pygmo.

Furthermore, you mention that Python is very slow and that's why you made the engine in C++. But numpy can make a lot of Python code almost as fast as C++, and the code that cannot easily be vectorized using numpy can usually be made almost as fast as C++ using the numba package. So unless there is a good reason why you need a C++ implementation, you might consider porting your source-code to pure Python using numpy and numba - the code would be a lot easier to maintain and it would run nearly as fast as the C++ code.

[darioizzo]

:) it is Impossible to allow all of pagmo's features and speed in pure python. Starting with the parallelization interface and multithreading ... numpy and numba only allow for efficient vectorization of scientific code, but c++ as a programming language in pagmo does way more than that.

So thanks, but no thanks ... pure python would be a suicide here.

Updating the wheels to include support also more recent python on linux is possible and relatively easy .. we just do not have the resources ATM to do it. Contributions from the community are welcome!

[Hvass-Labs]

Since you are employees of ESA, perhaps you could just ask your manager to allow you to spend some of your work-time on updating the pygmo pip package. It is a quite popular package and it probably has far greater impact on the world than most research papers. So it would be a wise investment of time and resources.

[bek0s]

Hello friends. :) Just wanted to share my 2 cents with you.

I have been developing a galaxy kinematics modelling software for some time now in Python. The software used to use pygmo2 as its swarm intelligence optimizer, but unfortunately, I have deprecated it in favour of a different library (pymoo). I am saying "unfortunately" because I find pygmo2 to be the best library for swarm intelligence optimisation. I have tried a lot of different similar libraries, but I haven't found something as good so far.

The only reason for the deprecation of pygmo2 in my software is the fact that it can't be installed using pip. I used to work in academia, and I have been witnessing the same drama again and again: (1) astronomers wasting hours or even days just to install a piece of software. (2) astronomers having a gazillion different python environments on their machine that are all broken. Because of that, I have always been a strong supporter of keeping thing simple and creating software that does not need third party package managers like conda.

Please, don't get me wrong, I am not saying all the above to change your mind. In fact, I totally understand pygmo2 team's reluctance to support pip installations. It needs time. Not because it is a hard problem to solve, but because you need to spend a lot of time reading and learning how to do this. My software, although a Python library, it uses C++ and CUDA for the computationally intensive parts. Because of that I had to go through all the pain of learning how to use pip, cmake, pybind11, the manylinux docker images, and auditwheel together to achieve that. As I said before, looking back now, I realise it is not a hard thing to do, but you need to spend a lot of time to learn how to do it.

Anyway, regardless what direction the pygmo2 team decides take, I have to acknowledge that they have done a great work and I would like to thank them for sharing this amazing library with us.

[astrojuanlu]

I appreciate @Hvass-Labs remarks about pip, but I don't think it's reasonable to expect from the Pygmo2 developers to port their C++ to numba.

Also, it's not needed: there are perfectly working packaging methods to combine C++ and Python. I'll hopefully try to send a PR, since this project is in use by one of my clients.

[astrojuanlu]

Help welcome: https://github.com/esa/pygmo2/pull/110

[darioizzo]

117 (distutils based) is fixing this issue while a stable solution is developed. After merge wheels will be uploaded at the next release.

Source code cannot be uploaded to pip in an architecture independent way, only binary wheel are thus created and uploaded.