Add xoak and fastparquet to environment?

microsoft / PlanetaryComputer

Issues, discussions, and information about the Microsoft Planetary Computer

https://planetarycomputer.microsoft.com/

MIT License

185 stars 8 forks source link

Add xoak and fastparquet to environment? #294

Closed rsignell closed 2 weeks ago

rsignell commented 1 year ago

Any chance to add xoak and fastparquet to the Planetary Computer hub environment? We have a notebook that shows how to explore the USGS CONUS404 simulation, and I'd love to point people to the Planetary Computer hub as a place they could run it (using Dask gateway to speed things up).

The notebook uses:

xoak for lon/lat point selection ( the lon/lat coords are 2D and there is no projected x,y coordinates).
fastparquet to read references generated by Kerchunk

TomAugspurger commented 1 year ago

Hi Rich. I'll look into adding those next time I update the environments.

I think we had fastparquet in the past, but had to remove it thanks to a package conflict. We do have pyarrrow.parquet, if that works for parquet reading.

rsignell commented 1 year ago

Ah, okay -- I'll check with @martindurant about why fsspec is complaining about needing fastparquet to read the references when pyarrow.parquet exists.

martindurant commented 1 year ago

I have used fastparquet for referenceFS explicitly because:

it has helped rapid development to work with an API I know well (and is simpler, at least for me)
it is a much smaller install dependency
it is probably faster for this data (although this is untested and may not be significant)

I am surprised that fastparquet could conflict with anything - it's even been built for wasm.

pandas>=1.5.0
numpy>=1.20.3
cramjam>=2.3  # has no downstream deps
fsspec
packaging

martindurant commented 1 year ago

(having said all that, it may be reasonable at this point to relax the parquet backend requirement)

ghidalgo3 commented 2 weeks ago

The hub is no longer available.