I prototyped DCL supporting Zarr storage on S3 on this branch , which is now outdated, but the backend code changes should still be useful. Zarr storage could potentially be a useful alternative to our zip project storage on S3 since we can cleanly store metadata, store data in chunks, and have a hierarchical storage formatting solution on a remote space. In general this would need to be investigated further to see if the migration would be worth it, and would need to discuss with the lab to make this a lab-wide structural change (it can't just be a DCL thing, and it would probably benefit other things more anyway).
I prototyped DCL supporting Zarr storage on S3 on this branch , which is now outdated, but the backend code changes should still be useful. Zarr storage could potentially be a useful alternative to our zip project storage on S3 since we can cleanly store metadata, store data in chunks, and have a hierarchical storage formatting solution on a remote space. In general this would need to be investigated further to see if the migration would be worth it, and would need to discuss with the lab to make this a lab-wide structural change (it can't just be a DCL thing, and it would probably benefit other things more anyway).