Simplify installation ignoring embargoed dandisets (Was: Dandiset 000981 missing?)

[pgleeson]

Has https://github.com/dandisets/000981.git been deleted? I'm currently getting an error with this dandiset when trying:

 datalad install -s https://github.com/dandi/dandisets.git --recursive --recursion-limit 1 --jobs 4

but there is a submodule for 981 here...

[yarikoptic]

Has https://github.com/dandisets/000981.git been deleted?

it is there but it is private. (embargoed dandiset)

I'm currently getting an error with this dandiset when trying:

 datalad install -s https://github.com/dandi/dandisets.git --recursive --recursion-limit 1 --jobs 4

but there is a submodule for 981 here...

for that one and others you will get an error which you would need to ignore for now. Is that a problem? (ie do you have some automation to clone/keep them all)?

[pgleeson]

Yes @yarikoptic, we have a script to pull all dandisets using datalad, and then we run tests on selected nwb files in the dandisets: https://github.com/OpenSourceBrain/DANDIArchiveShowcase/actions/runs/9404051423/job/25902064823#step:4:11094

It would be great if there was a datalad compatible way to ignore these automatically. It used to work fine and it does seem a bit strange that the private ones are referenced here as submodules already even though they're inaccessible.

[yarikoptic]

might take a bit... meanwhile please subscribe to

• https://github.com/datalad/datalad/issues/7567

which I think would be the one to monitor for needed functionality to appear.

[yarikoptic]

may be we could cook up a workaround quickly enough though...

• 386

  which I hope could later be used when solution on datalad level is developed

[yarikoptic]

a workaround recipe here from #386

datalad install -s https://github.com/dandi/dandisets.git
cd dandisets
tools/list-matching-access-status public | xargs datalad install --jobs 4

[pgleeson]

Thanks @yarikoptic this has gotten me past this issue, other have cropped up, but not related to this. Cheers!