search

conda-forge / staged-recipes

A place to submit conda recipes before they become fully fledged conda-forge feedstocks
https://conda-forge.org
BSD 3-Clause "New" or "Revised" License
698 stars 4.84k forks source link

New python-implementation? Package pyston #13019

Open h-vetinari opened 3 years ago

h-vetinari commented 3 years ago

Hot off the press, v2 of pyston, claims to be 20% faster than python with lower memory footprint than PyPy.

Thoughts @isuruf @conda-forge/core?

jakirkham commented 3 years ago

Timely! 😄 Was just reading that blogpost earlier. Yeah this seems like a good idea. We might want to confirm, which Python versions they are targeting. Historically this was Python 2, but the blogpost seems to indicate Python 3 support (though unclear as to which one(s)). Also the blogpost seems to indicate this hasn't been open sourced yet, but will be soon (feel free to correct me if I misread). So maybe this buys us some time to figure out how we want to do this when it is open sourced.

h-vetinari commented 3 years ago

Pyston 2.2 has now been open-sourced 🙃 https://github.com/pyston/pyston

wolfv commented 3 years ago

Yeah, this is exciting! the 2x llvm builds in there scare me a bit, though :)

chrisburr commented 3 years ago

the 2x llvm builds in there scare me a bit, though :)

Yup! At a glance it looks like:

h-vetinari commented 3 years ago

The good thing is that BOLT will likely become part of LLVM eventually (though that'll still take month or quarters, realistically).

h-vetinari commented 3 years ago

Here's a tracking issue for LLVM integration of BOLT: https://github.com/facebookincubator/BOLT/issues/183

jakirkham commented 3 years ago

Looks like Anaconda hired the devs working on Pyston. More details in this Pyston blogpost

jakirkham commented 2 years ago

Coming to this a bit late, but it appears pyston has included some packages in their own channel. So maybe that is a simple way to install them and get started

h-vetinari commented 2 years ago

Thanks John! I think it might be worth considering eventually(?) just providing that directly from conda-forge, then also those resolver errors mentioned in the blog post would be much less of an issue (because there'd at least be a migrator to track).

Ping @kmod for visibility (was just conversing with them in https://github.com/pyston/pyston/issues/180).

jakirkham commented 2 years ago

No objection to that. Though there is always a balance to be struck in terms of what is most productive/effective for everyone. If rebuilds are quite frequent, that may mean just using the pyston channel. Whenever things settled down, migrating to conda-forge may make sense. Would be good to hear from Kevin, et al. on what state things are in and when that transition may make sense 🙂

kmod commented 2 years ago

We'd love to be included! We built quite a few of the top conda-forge packages for our channel, and the main issues were not Pyston-related (we had to deal with a couple circular dependencies, as well as some feedstocks that don't build currently because new versions of dependencies broke them). We do also have a small amount of patches for Pyston compatibility, and we haven't submitted them yet to the projects because we are waiting on some discussions with the Python packaging community about how they would like to support alternative implementations.

Here's the script that I wrote to build the packages, which also includes most of the patches that we applied to the feedstocks: https://github.com/pyston/pyston/blob/pyston_main/pyston/conda/build_feedstock.py

How do you all deal with building packages for a new version of CPython? I guess I should have asked that before building these packages ourselves. Hopefully adding Pyston isn't too much harder than that.

We don't anticipate breaking ABI compatibility (causing rebuilds) very often, we're currently hoping that we only have to do that when we retarget to a new Python version.

jakirkham commented 2 years ago

Just to confirm, packages using Pyston need to be rebuilt against it or is there some level of ABI/API compatibility with CPython?

Also curious to get your thoughts on whether C API standard projects like HPy, abi3, etc. might be used in Pyston (admittedly this is likely a longer term thing)?

kmod commented 2 years ago

We are API compatible but not ABI compatible. (We support the stable ABI but don't yet properly flag that.) We rebuilt all the binary packages but didn't need to do anything with the pure-python packages.

We're in an interesting position where we support compiling any package from source, but some packages (like tensorflow, numba, pytorch) are essentially impossible for a user to build on their machine. So we're hoping to get to a point where we have Pyston-native extensions available so people can have the best performance, but as an intermediate step it'd be great to have more stable-abi or HPy-universal-packages!

jakirkham commented 2 years ago

So we do have the python_abi for tracking different Python versions & implementations.

Also we do run migrations for new Python versions and PyPy. Those are currently done globally. Though we have done this in an opt-in basis for some things like ARM & PPC.

Adding a new Python implementation is doable. We might want to discuss how many packages we want to rebuild (IOW do we have a specific list or are we trying to do everything?).

cc @conda-forge/core (guessing others will have thoughts on these and have additional questions as well)

beckermr commented 2 years ago

We didn't rebuild everything for pypy, just what people wanted. We can do the same for pyston.

I'll defer to other core devs for a list of steps to make it happen. I assume building pyston is at the top of the list. We may want to ensure things can be cross-compiled as well.

jakirkham commented 2 years ago

We didn't rebuild everything for pypy, just what people wanted.

Interesting this wasn't my understanding based on the discussion in PR ( https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/225 ) or issue ( https://github.com/regro/cf-scripts/issues/1386 ). For ARM & PPC IIUC this was handled as opt-in as a one-off. Has this changed somewhere for PyPy (or more generally)? If so, where do we maintain the list of things that are opt-in (and how do we opt-out)?

We can do the same for pyston.

This seems reasonable if we have some mechanism for doing that.

I assume building pyston is at the top of the list.

+1

We may want to ensure things can be cross-compiled as well.

To clarify do we care about ARM or PPC here? My temptation would be to skip these on a first pass and stick to x86_64 (unless there is a compelling need for these).

Question for Kevin, et al., are macOS & Windows supported? Or is Pyston Linux only atm?

beckermr commented 2 years ago

Sorry @jakirkham. You are correct that we had a migrator for pypy that tried everything. However large parts of the ecosystem were declared effectively off limits because we did not convert key dependencies.

We can do the same for pyston.

h-vetinari commented 2 years ago

I assume building pyston is at the top of the list.

I don't think we're under huge time pressure here. I'd say that unless there's a recipe by the pyston folks that can be plugged in more or less directly into cf-infra, it would IMO make sense to wait until we can reuse our own LLVM builds, rather than rebuild them as part of pyston (cf. https://github.com/pyston/pyston/issues/180)**, cf. @wolfv's comment above

the 2x llvm builds in there scare me a bit, though :)

Of course this is not a blocker, and all other things being equal, earlier is better, but just wanted to bring that up again.

** while I cannot say how much work this would be on the pyston side, BOLT is already part of LLVM 14, which should reach GA in 2-3 months.

undingen commented 2 years ago

Another Pyston dev here, would be very happy if pyston gets integrated into conda-forge! Our conda packages are currently build against conda-forge and the recipes can be found here: https://github.com/pyston/pyston/tree/pyston_main/pyston/conda Or pyston compilation process is unfortunately quite complicated and involves PGO and BOLT which we run on some external python projects.

We did split it very similar to the PyPy packages and use a python_abi package (that's why we also can't use the anaconda channel which has it not set - but I'm sure you already know that). I hope I didn't make any mistakes and this would make official integration easy.

We have our own BOLT package for now but I think the other dependencies all exist in the correct version in conda-forge (llvm+compiler-rt 13.0.1), I tried at first to base our conda repo against the anaconda channel and that caused more problems. I think when LLVM 14 is released we can try to remove our dependency on a specific BOLT version (will try it out soon). But we run in quite a few issues with BOLT previously so we may need to update it in the future to a unreleased version again/integrate some patches.

The released Pyston is currently only supporting Linux on amd64 (no 32 bit), but we are currently working on Arm64 (no 32 bit) support (which landed a few days ago but we did not try building any packages yet) which we will likely release end of next month. Maybe we will also have MacOS support if there is time and we don't run into big problems but we have not tried yet. For the Arm release we will need to build own packages again but the amd64 we have will be fine because we did not break the ABI. Would be awesome if in the future conda-forge would automatically get build for pyston and not manually like we currently do (which means our packages are not up to date) :).