search

dandi / dandisets-healthstatus

Healthchecks of dandisets and support libraries (pynwb and matnwb)
0 stars 1 forks source link

Add option for setting number of Dandisets to operate on concurrently #11

Closed jwodder closed 1 year ago

jwodder commented 1 year ago

Closes #9.

yarikoptic commented 1 year ago

@candleindark please review and familiarize with the codebase here.

@jwodder -- looks good to my async-ignorant eye. have you tried it on drogon? if needs a merge for that -- please proceed with the merge

jwodder commented 1 year ago

@yarikoptic I ran it on drogon with 8 jobs, and it seems to be working fine. Only Dandisets 6 and 7 have completed so far.

yarikoptic commented 1 year ago

I saw it working and seems to be "in effect" (different dandisets processed in parallel)