maximedion2
commented
2 months ago Here's a first pass. We won't necessarily take care of all the must-haves before doing some of the nice-to-haves, some of the must-haves are more long term.
Eventual must-haves:
- Metrics to allow filters to be sorted for filter push downs (for optimization)
- Test stores other than local file systems (e.g. AWS s3)
- Add SQL logical tests (see https://github.com/risinglightdb/sqllogictest-rs)
- Implement a proper non-blocking async reader.
Nice-to-haves:
- Geospatial functions (working off of WKT strings in e.g. parquet files, supporting full-on geospatial objects would probably be a ton of work)
- Optimize reading data by supplementing some "columns" with 1D array data (more details on this in the issue I will create specifically for it)
- I think I could rewrite some of the logic in a nicer way, for example how the raw data and metadata are kept separately and only combined when decoding the data, could be nicer to have the meta directly in the
ZarrInMemoryArray.
Maybe won't do:
- Anything related to dumping query results to disk (in the zarr format). Writing in the zarr format is tricky, it's not as simple as writing e.g. a parquet file that contains all the columns, so it'd be a lot of work to implement that (I think).
This will be a list of must-haves, nice-to-haves, and things that we probably won't do, for the overall zarr + DataFusion project, to keep track of the overall progress.