Closed slaytonmarx closed 2 years ago
@slaytonmarx - does the behavior change if you just use .
instead of the ./
Unfortunately not :(
@jwodder - any ideas what would cause this behavior? is it doing something but not logging?
@slaytonmarx - can you post which session/folder you are doing this in? i can also try it. (please do kill those processes)
Yes!
ses-20211111h11m08s18
/mnt/beegfs/Lee/dandi/sub-MITU01/ses-20211111h11m08s18/micr
could you try cd /mnt/beegfs/Lee/dandi/; DANDI_DEVEL=1 dandi upload --validation skip --allow-any-path sub-MITU01/ses-20211111h11m08s18/micr
instead?
i can confirm the behavior. in fact even this command hangs:
DANDI_DEVEL=1 dandi -l DEBUG --pdb upload --validation skip --allow-any-path sub-MITU01/ses-20211111h11m08s18/micr/sub-MITU01_ses-20211111h11m08s18_sample-11_photo.jpg
oh... "hangs". It might not be "hanging" but still "navigating" the filesystem to find bids datasets... since network and .zarrs -- may be taking awhile. I have ran also and now observing the thing to be at
leviathan:~
$> ls -l /proc/37865/fd/
total 0
0 lrwx------ 1 yoh chunglab 64 Jul 18 16:54 0 -> /dev/pts/2
0 lrwx------ 1 yoh chunglab 64 Jul 18 16:54 1 -> /dev/pts/2
0 lrwx------ 1 yoh chunglab 64 Jul 18 16:54 2 -> /dev/pts/2
0 lr-x------ 1 yoh chunglab 64 Jul 18 16:54 3 -> /dev/urandom
0 l-wx------ 1 yoh chunglab 64 Jul 18 16:54 4 -> /home/yoh/.cache/dandi-cli/log/20220718203715Z-37865.log
0 lr-x------ 1 yoh chunglab 64 Jul 18 16:54 5 -> /mnt/beegfs/Lee/dandi/sub-MITU01/ses-20220310h10m08s20/micr/sub-MITU01_ses-20220310h10m08s20_sample-19_stain-YO_run-1_chunk-7_SPIM.ome.zarr/sub-MITU01_ses-20220310h10m08s20_sample-19_stain-YO_run-1_chunk-7_SPIM.ngff/0/0/0/14/15/
so -- we should stop navigating deeper than into .ngff (BTW -- how come there is .ngff -- should be .ome.zarr, right?) or .ome.zarr subfolders.... I will propose PR in a minute.
is there a reason why it's navigating anything other than going up to find dandiset.yaml when a specific path is provided (for example the jpg command above).
also why is it trying to find bids datasets with both --validation skip --allow-any-path
. either should prevent any search.
is there a reason why it's navigating anything other than going up to find dandiset.yaml when a specific path is provided (for example the jpg command above).
not yet sure about why it is navigating that much BUT it is not about dandiset.yaml -- we look for BIDS datasets under the path(s) and validate them. E.g. think about 000026.
also why is it trying to find bids datasets with both
--validation skip --allow-any-path
. either should prevent any search.
In principle "yes" with --allow-any-path but currently code finds BIDS datasets to decide either to allow upload files other than .nwb (but there is https://github.com/dandi/dandi-cli/issues/1055 so YMMV ;)).
Anyways -- hold there, I will propose a fix to avoid navigating too deep.
Attempt to run dandi upload leading to indefinite stalling. At first I thought it may be due to output not piping correctly, but logs revealed no activity.
On version 0.45.1
Command
DANDI_DEVEL=1 dandi upload --validation skip --allow-any-path ./
Within directory contain ome.zarr files for uploadLog in it's entirety