search

danlamanna / rivet

a tiny cli for syncing a directory with a girder instance
Apache License 2.0
6 stars 1 forks source link

progress status #9

Open bendichter opened 4 years ago

bendichter commented 4 years ago

great tool! It's working like a charm for uploading data to DANDI. It would be nice if there were some progress reports for long uploads. Like a progress bar with a time estimate.

danlamanna commented 4 years ago

Yeah this is a feature we definitely want, and there's actually a half-baked branch for it that I haven't touched in a while.

In the meantime, rivet has a verbose flag that will spam you a bit and give you some non-friendly looking progress:

$ rivet sync -v . girder://35892fe58d777c649a9bd1b7
DEBUG[2020-01-31T14:54:40-05:00] attempting to load config file /Users/dan/.rivet/config.toml 
DEBUG[2020-01-31T14:54:40-05:00] loaded config file /Users/dan/.rivet/config.toml 
DEBUG[2020-01-31T14:54:40-05:00] loaded credentials from configuration file   
DEBUG[2020-01-31T14:54:41-05:00] authenticated with api key (user 35782fe58d577f639a9bc1ab) 
DEBUG[2020-01-31T14:54:41-05:00] scanning . for syncable items                
INFO[2020-01-31T14:54:41-05:00] found 0 dirs, 1 files to potentially sync    
INFO[2020-01-31T14:54:41-05:00] building remote girder directories           
INFO[2020-01-31T14:54:41-05:00] building remote girder items                 
INFO[2020-01-31T14:54:42-05:00] syncing blobs                                
DEBUG[2020-01-31T14:54:42-05:00] detected new file video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 
INFO[2020-01-31T14:54:42-05:00] uploading: video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 
DEBUG[2020-01-31T14:54:42-05:00] video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 - uploading chunk 1/219 
DEBUG[2020-01-31T14:54:55-05:00] video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 - uploading chunk 2/219 
DEBUG[2020-01-31T14:55:07-05:00] video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 - uploading chunk 3/219 
DEBUG[2020-01-31T14:55:19-05:00] video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 - uploading chunk 4/219 
DEBUG[2020-01-31T14:55:31-05:00] video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 - uploading chunk 5/219 
DEBUG[2020-01-31T14:55:44-05:00] video_dh37gs_en 2019-08-25 12_38-35423486816.mp4 - uploading chunk 6/219

Each chunk is 16MB IIRC.

P.S. rivet sync -vv will actually show every single HTTP request that is being performed. Useful for debugging and not much else.

bendichter commented 4 years ago

@danlamanna thanks for the pointers. -v is a step in the right direction, but I am particularly interested in a progress report on a large number of files, e.g. python's tqdm.