search

JakobGM / patito

A data modelling layer built on top of polars and pydantic
MIT License
252 stars 23 forks source link

Unable to install with polars-lts-cpu #50

Open lmmx opened 4 months ago

lmmx commented 4 months ago

Hi there, big fan of this project been thinking about the implications, congrats on the recent Pydantic 2.0 re-launch! :tada:

I'm just wondering whether there is an oversight here in regards to the polars-lts-cpu (for hardware without certain CPU instructions) and how that might be represented in package dependencies.

I looked around the Polars repo and came across this issue

In the thread, the following advice is given by Tim Stephenson:

Listing polars as a dependency in a package seems to be an oversight based off of how different x86 binaries for newer/older cpu's are given completely different package names. Potentially the answer from the polars team is that you shouldn't make python packages which depend on polars, only using polars as an end-user. Or you should only compile from source when deploying your package onto an old Xeon server.

I interpret this to mean you should not be putting polars as a package dependency, which is a surprise to say the least!

Perhaps instead it would be possible to make polars an extra, and polars-lts-cpu another extra? I don't suppose it's desirable to ship a patito-lts-cpu package

I'd be interested to see any other suggestions.

If you don't think it's worth the effort to change (and should be left to the user to sort out) then I'd understand too.

thomasaarholt commented 4 months ago

This is an interesting issue! I've explored a similar problem before. I'm a bit hesitant to remove polars a s a direct dependency, but this might make sense. I wish there was a way to specify a "default extra" for pip, with the possibility of overriding it. Let me chew on this for a bit.

lmmx commented 4 months ago

It's a fun one isn't it!

In my case, I decided to handle the transitive dependency exclusion in my package with PDM resolution overrides

As you rightly infer, this does not extend to the distributed package (pip install), but just the development package (pdm install).

I'm interested in making my package pip installable, chewing on this myself!

P.S. the other option would be to copy what Polars did and ship a patito-lts-cpu package