`list_dir` relies on `list_prefix` which traverses all chunks

earth-mover / icechunk

Open-source, cloud-native transactional tensor storage engine

https://icechunk.io

Apache License 2.0

292 stars 17 forks source link

`list_dir` relies on `list_prefix` which traverses all chunks #321

Open paraseba opened 4 weeks ago

paraseba commented 4 weeks ago

This is a problem because it's much slower than it needs to be, and it's used by zarr to implement things like tree. We should make list_dir smarter.