Using / collaborating on python ports

[m-kuhn]

I am currently in the process of building an application (QGIS) including python support with various packages bundled.

The tremendous work that has been done within this repository will be very helpful.

I wonder what to best do:

1. Using this (Neumann-A/my-vcpkg-ports) repository from the build
2. Copy the (interesting) ports to another overlay
3. Upstreaming ports to microsoft/vcpkg
4. Creating a new registry [whatever]/vcpkg-python

I think the ideal world would be 3. but I guess you have your reasons for not having these ports upstreamed? 1. is not my favorite, as there is a lot going on in here and occasionally other ports will overlay upstream ports. 2. is duplicating efforts and a lot of forth and back. 4. seems to be a valid thing to do if there is genuine interest.

Opinions?

[Neumann-A]

Main aim would be to get numpy into vcpkg but there a few details which need to be considered.

I think the ideal world would be 3. but I guess you have your reasons for not having these ports upstreamed?

• requires python PATH adjustment from e.g. https://github.com/microsoft/vcpkg/pull/31279. On linux it works if rpath fixup is on by default (https://github.com/microsoft/vcpkg/pull/36056). Otherwise modules are simple not able to find their shared libs.
• One of the python packages needs the package-root to be on a shorter path
• numpy itself needs https://github.com/microsoft/vcpkg/pull/28084 being updated to 1.3.1. That seems to be partly blocked by https://github.com/microsoft/vcpkg/pull/31748 and https://github.com/mesonbuild/meson/issues/12696
• numpy needs a decision on blas/lapack + forcing it to be consistent (https://github.com/microsoft/vcpkg/pull/24327). In the overlay here I default to using MKL instead of openblas/lapack-reference
• python should export it site-packages dir for consumption for downstream ports or there is a simple script to request the correct location from the installed python.
• i currently have problems installing numpy using the port on linux due to either the system python being used or the installed python not finding something. Have not taken the time yet to debug that.
• Ideally there would be a solution for Fortran as well but vcpkg still claims to be a C/CXX packaging system.
• For some stuff using python-maturin it would be interesting to know if cargo should be somehow pinned. Currently it simply downloads whatever which probably is not a good approach for vcpkg. (I mean it works but it is not asset cacheable.)

There is also a small discussion on discord starting at https://discord.com/channels/400588936151433218/687365466422902841/1186421038574227456

So if you want a more or less fast solution (4) is probably the way to go since I don't see any of the PRs being merged any time soon.

There are few things which need to be done for that:

• Each python port should have a usage test in the form import * from <installed_module>. Need to figure out how to do that in/from packages though
• naming of ports needs to be decided. I used python-<name> but that is basically just a suggestion
• needs a decision on [whatever]. [whatever] should probably represent a group instead of a single person.

Note: I am currently unemployed so I have some time at hand to do whatever I like ;)

[m-kuhn]

Thanks for the detailed description.

There is also a small discussion on discord starting at https://discord.com/channels/400588936151433218/687365466422902841/1186421038574227456

Didn't notice, looks promising

So if you want a more or less fast solution (4) is probably the way to go since I don't see any of the PRs being merged any time soon.

I like the pragmatism of this approach, and it doesn't exclude a gradual movement towards 1 either.

There are few things which need to be done for that:

• Each python port should have a usage test in the form import * from <installed_module>. Need to figure out how to do that in/from packages though

Isn't that a nice to have, though?

• naming of ports needs to be decided. I used python-<name> but that is basically just a suggestion

The few I was experimenting with were py3-<name>, inspired by alpine linux. I'm lazy, so I'm putting "less typing" forward as an argument. No strong opinion.

• needs a decision on [whatever]. [whatever] should probably represent a group instead of a single person.

Indeed, I can't think of anything reasonable that exists already, a new org vcpkg-contrib or vcpkg-python?

Note: I am currently unemployed so I have some time at hand to do whatever I like ;)

I didn't expect myself to respond with this line to "I am currently unemployed", but: glad to hear.

[Neumann-A]

I like the pragmatism of this approach, and it doesn't exclude a gradual movement towards 1 either.

You mean 3 ?

Isn't that a nice to have, though?

It is to make sure the python packages are actually working and have the correct dependencies. While doing all the python stuff I often came across missing dependencies in a 2nd run due to me having all other deps already installed by a different port.

Indeed, I can't think of anything reasonable that exists already, a new org vcpkg-contrib or vcpkg-python?

It should probably be more general than vcpkg-python making sure it can be used for other stuff. vcpkg-contrib would be a possibility. open-vcpkg sounds a bit better to me (and is shorter :) ) . We could open a discussion in vcpkg and ask other contributors for a name and then have a vote on it.

The few I was experimenting with were py3-, inspired by alpine linux. I'm lazy, so I'm putting "less typing" forward as an argument. No strong opinion.

Question is: Does it need the version number?

[m-kuhn]

I like the pragmatism of this approach, and it doesn't exclude a gradual movement towards 1 either.

You mean 3 ?

Correct :see_no_evil:

Isn't that a nice to have, though?

It is to make sure the python packages are actually working and have the correct dependencies. While doing all the python stuff I often came across missing dependencies in a 2nd run due to me having all other deps already installed by a different port.

I'm not arguing that the idea is bad

Indeed, I can't think of anything reasonable that exists already, a new org vcpkg-contrib or vcpkg-python?

It should probably be more general than vcpkg-python making sure it can be used for other stuff. vcpkg-contrib would be a possibility. open-vcpkg sounds a bit better to me (and is shorter :) ) . We could open a discussion in vcpkg and ask other contributors for a name and then have a vote on it.

I like this idea :+1:

The few I was experimenting with were py3-, inspired by alpine linux. I'm lazy, so I'm putting "less typing" forward as an argument. No strong opinion.

Question is: Does it need the version number?

Not necessarily

[m-kuhn]

https://github.com/open-vcpkg/ is live 🎉