search

NDAR / nda-tools

Python package for interacting with NDA web services. Used to validate, submit, and download data to and from NDA.
MIT License
48 stars 22 forks source link

Parallel Submission? #33

Closed JBarsotti closed 8 months ago

JBarsotti commented 3 years ago

Hello!

Sorry to bother you again. I was just wondering if you know whether it would be possible to run nda-tools in parallel across several processors and then submit the resulting files in parallel to NDA? I'm trying to decrease the upload time, but I'm not sure if doing it in parallel would be a good idea.

Thanks,

John

gregmagdits commented 3 years ago

Hey, no worries. We have a setting to increase the number of worker threads used to upload files, which is usually the step that takes the longest amount of time. To adjust the thread count, you can use the '-wt' command line option. For example: vtcmd fmriresults01.csv -b -wt 10 will set the number of threads to 10. The value you should set this to depends on the hardware you're running on, so usually it requires a little trial and error to get the right number - if you set the value too high it will be counterproductive and will probably cause the system to go slower rather than faster.

If the upload takes an unreasonable amount of time to finish even after adjusting the number of threads, let me know.

JBarsotti commented 3 years ago

Thanks so much! Will net you know how it goes.