search

dandi / dandi-cli

DANDI command line client to facilitate common operations
https://dandi.readthedocs.io/
Apache License 2.0
21 stars 25 forks source link

Hanging dandi upload call #1069

Closed slaytonmarx closed 2 years ago

slaytonmarx commented 2 years ago

Attempt to run dandi upload leading to indefinite stalling. At first I thought it may be due to output not piping correctly, but logs revealed no activity.

On version 0.45.1

Command DANDI_DEVEL=1 dandi upload --validation skip --allow-any-path ./ Within directory contain ome.zarr files for upload

Log in it's entirety

2022-07-18T14:09:44-0400 [INFO    ] dandi 15159:139926543738688 dandi v0.45.1, hdmf v1.5.4, pynwb v1.2.1, h5py v2.10.0
2022-07-18T14:09:44-0400 [INFO    ] dandi 15159:139926543738688 sys.argv = ['/mnt/beegfs/satra/miniconda3/envs/dandi/bin/dandi', 'upload', '--validation', 'skip', '--allow-any-path', './']
2022-07-18T14:09:44-0400 [INFO    ] dandi 15159:139926543738688 os.getcwd() = /mnt/beegfs/Lee/dandi/sub-MITU01/ses-20211111h11m08s18/micr
2022-07-18T14:09:45-0400 [DEBUG   ] urllib3.connectionpool 15159:139926543738688 Starting new HTTPS connection (1): rig.mit.edu:443
2022-07-18T14:09:45-0400 [DEBUG   ] urllib3.connectionpool 15159:139926543738688 https://rig.mit.edu:443 "GET /et/projects/dandi/dandi-cli HTTP/1.1" 200 579
2022-07-18T14:09:45-0400 [DEBUG   ] dandi 15159:139926543738688 No newer (than 0.45.1) version of dandi/dandi-cli found available
2022-07-18T14:10:21-0400 [INFO    ] dandi 15159:139926543738688 Logs saved in /home/smarx/.cache/dandi-cli/log/20220718180944Z-15159.log
satra commented 2 years ago

@slaytonmarx - does the behavior change if you just use . instead of the ./

slaytonmarx commented 2 years ago

Unfortunately not :(

satra commented 2 years ago

@jwodder - any ideas what would cause this behavior? is it doing something but not logging?

@slaytonmarx - can you post which session/folder you are doing this in? i can also try it. (please do kill those processes)

slaytonmarx commented 2 years ago

Yes!

ses-20211111h11m08s18

/mnt/beegfs/Lee/dandi/sub-MITU01/ses-20211111h11m08s18/micr

yarikoptic commented 2 years ago

could you try cd /mnt/beegfs/Lee/dandi/; DANDI_DEVEL=1 dandi upload --validation skip --allow-any-path sub-MITU01/ses-20211111h11m08s18/micr instead?

satra commented 2 years ago

i can confirm the behavior. in fact even this command hangs:

DANDI_DEVEL=1 dandi -l DEBUG --pdb upload --validation skip --allow-any-path sub-MITU01/ses-20211111h11m08s18/micr/sub-MITU01_ses-20211111h11m08s18_sample-11_photo.jpg

yarikoptic commented 2 years ago

oh... "hangs". It might not be "hanging" but still "navigating" the filesystem to find bids datasets... since network and .zarrs -- may be taking awhile. I have ran also and now observing the thing to be at

leviathan:~
$> ls -l /proc/37865/fd/
total 0
0 lrwx------ 1 yoh chunglab 64 Jul 18 16:54 0 -> /dev/pts/2
0 lrwx------ 1 yoh chunglab 64 Jul 18 16:54 1 -> /dev/pts/2
0 lrwx------ 1 yoh chunglab 64 Jul 18 16:54 2 -> /dev/pts/2
0 lr-x------ 1 yoh chunglab 64 Jul 18 16:54 3 -> /dev/urandom
0 l-wx------ 1 yoh chunglab 64 Jul 18 16:54 4 -> /home/yoh/.cache/dandi-cli/log/20220718203715Z-37865.log
0 lr-x------ 1 yoh chunglab 64 Jul 18 16:54 5 -> /mnt/beegfs/Lee/dandi/sub-MITU01/ses-20220310h10m08s20/micr/sub-MITU01_ses-20220310h10m08s20_sample-19_stain-YO_run-1_chunk-7_SPIM.ome.zarr/sub-MITU01_ses-20220310h10m08s20_sample-19_stain-YO_run-1_chunk-7_SPIM.ngff/0/0/0/14/15/

so -- we should stop navigating deeper than into .ngff (BTW -- how come there is .ngff -- should be .ome.zarr, right?) or .ome.zarr subfolders.... I will propose PR in a minute.

satra commented 2 years ago

is there a reason why it's navigating anything other than going up to find dandiset.yaml when a specific path is provided (for example the jpg command above).

satra commented 2 years ago

also why is it trying to find bids datasets with both --validation skip --allow-any-path. either should prevent any search.

yarikoptic commented 2 years ago

is there a reason why it's navigating anything other than going up to find dandiset.yaml when a specific path is provided (for example the jpg command above).

not yet sure about why it is navigating that much BUT it is not about dandiset.yaml -- we look for BIDS datasets under the path(s) and validate them. E.g. think about 000026.

also why is it trying to find bids datasets with both --validation skip --allow-any-path. either should prevent any search.

In principle "yes" with --allow-any-path but currently code finds BIDS datasets to decide either to allow upload files other than .nwb (but there is https://github.com/dandi/dandi-cli/issues/1055 so YMMV ;)).

Anyways -- hold there, I will propose a fix to avoid navigating too deep.