Split optional dependencies in pyproject.toml

zarr-developers / VirtualiZarr

Create virtual Zarr stores from archival data files using xarray syntax

https://virtualizarr.readthedocs.io/en/stable/api.html

Apache License 2.0

123 stars 24 forks source link

Split optional dependencies in pyproject.toml #309

Open TomNicholas opened 6 days ago

TomNicholas commented 6 days ago

Takes the splitting of optional dependencies idea started in https://github.com/zarr-developers/VirtualiZarr/pull/87 to its logical conclusion.

[x] Changes are documented in docs/releases.rst
[x] New functionality has documentation

maxrjones commented 6 days ago

I could change the CI to use pip. I have been using uv a bit recently and it's been super promising for managing deps, but would be a bigger change.

cisaacstern commented 6 days ago

without much context, obligatory pixi endorsement comment

TomNicholas commented 6 days ago

I think the CI failure is because the mypy CI job is apparently the only one that uses pip to install currently

norlandrhagen commented 6 days ago

Also @norlandrhagen fastparquet is in the CI but doesn't seem to be used anywhere?

I think it's a hidden requirement of using the Kerchunk parquet format.

TomNicholas commented 6 days ago

I think it's a hidden requirement of using the Kerchunk parquet format.

So does that mean we should have another writer split out in these dependencies?

maxrjones commented 6 days ago

without much context, obligatory pixi endorsement comment

emerging from a rabbit hole (thanks @cisaacstern 😂) I'm floating around this decision framework for feedback:

If you're starting a new Python package, use uv
If you need to manage multiple Python virtual environments at a time, also use hatch
If you need Conda-Forge in addition to PyPI distributions, ~~also~~ use pixi

More on topic, feel welcome to ping me if you'd like to assign any CI / packaging updates.

norlandrhagen commented 6 days ago

So does that mean we should have another writer split out in these dependencies?

Probably! Also, maybe for reading existing Kerchunk parquet refs.

TomNicholas commented 5 days ago

Probably! Also, maybe for reading existing Kerchunk parquet refs.

Does reading existing Kerchunk parquet refs require the same additional dependencies? If so we could have kerchunk_format = ["fastparquet"] as another group?

norlandrhagen commented 5 days ago

Does reading existing Kerchunk parquet refs require the same additional dependencies? If so we could have kerchunk_format = ["fastparquet"] as another group?

I just checked, both reading the writing of Kerchunk Parquet refs require fastparquet.

TomNicholas commented 5 days ago

In https://github.com/zarr-developers/VirtualiZarr/pull/309/commits/a633a0cadd7aca5ef5bd753f70ccd293f408b568 I embedded the pyproject.toml into the installation docs page so that the list of pip-installable optional dependencies always stays up-to-date. See

https://virtualizarr--309.org.readthedocs.build/en/309/installation.html