search

dandi / dandi-cli

DANDI command line client to facilitate common operations
https://dandi.readthedocs.io/
Apache License 2.0
22 stars 27 forks source link

Make zarr download begin as soon as initial page of zarr files is obtained #1447

Open yarikoptic opened 5 months ago

yarikoptic commented 5 months ago

This is continuation to work done in

where I thought that interleave() would be a generator which starts yielding without delaying for a full list of zarr files to be retrieved first. @jwodder mentioned that my assumption was wrong and that it would need additional refactoring to make it behave as desired.

While addressing this it might be worth checking if interleave could natively provide support for desired use case where it takes care about running that thread with "producer" iterator. It is somewhat similar to how I have done for ProducerConsumer in datalad where Producer is ran into a separate thread as well IIRC.

yarikoptic commented 4 months ago

ping on this one @jwodder , as today I have tried download of a little zarr and it was odd to stare at non-changing download output for quite a while. Example was

$ dandi download dandi://dandi/000719/sub-1214579789_ses-1214621812_icephys/sub-1214579789_ses-1214621812_icephys.nwb.zarr

which eventually started to download