search

dandi / dandi-cli

DANDI command line client to facilitate common operations
https://dandi.readthedocs.io/
Apache License 2.0
22 stars 28 forks source link

`FileExistsError` after restarting Zarr download #1473

Closed kabilar closed 4 months ago

kabilar commented 4 months ago

After restarting the download of a Zarr archive I receive a FileExistsError.

Steps to replicate:

  1. Start initial download (This may a while due to https://github.com/dandi/dandi-cli/issues/1447)

    dandi download https://api.dandiarchive.org/api/assets/9c225928-114b-4a8b-ba33-df1e9c9b8946/download/

  2. Interrupt download (control+C a few times).

  3. Restart download by running command in step 1.

    PATH                                   SIZE      DONE                DONE% CHECKSUM STATUS        MESSAGE          
...8_sample-01_chunk-03_hipCT.ome.zarr 548.7 GB  714.5 MB               0%          error         FileExistsError  
Summary:                               548.7 GB  714.5 MB                           1 error       1 FileExistsError
                                                 0.13%                                                             
                                                 ETA: 2 days
Tail of error log ``` ... 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/50 has correct md5 4bf6d41410bf4b3165b25a22531c6683 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6185463808 Starting new download in new download directory 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/56&download=true 2024-07-30T13:19:24-0500 [INFO ] dandi 87089:6168637440 File 0/0/21/55 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/55 has correct md5 3d1babd7e4f77a523682e2601fc6f272 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6168637440 Starting new download in new download directory 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6168637440 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/57&download=true 2024-07-30T13:19:24-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F4%2F10%2F13 HTTP/1.1" 200 119248 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:24-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/56&download=true HTTP/1.1" 302 0 2024-07-30T13:19:24-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F4%2F35%2F7 2024-07-30T13:19:24-0500 [DEBUG ] urllib3.connectionpool 87089:6168637440 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/57&download=true HTTP/1.1" 302 0 2024-07-30T13:19:25-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181924Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=e942c1c5e48203ad8e0baa15e4a02f078e607260b2028533a53b3ec7f816568d HTTP/1.1" 200 2673285 2024-07-30T13:19:25-0500 [DEBUG ] urllib3.connectionpool 87089:6168637440 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/57?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181924Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=5969a678de7856ab4d02fe531e33ef9d4eb6791332849d2f6d35f65ba0c20d2c HTTP/1.1" 200 2676335 2024-07-30T13:19:25-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F4%2F35%2F7 HTTP/1.1" 200 119128 2024-07-30T13:19:25-0500 [INFO ] dandi 87089:6151811072 File 0/0/21/52 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/52 has correct md5 d5da93a8b524fadecbf969c0d469953a 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6151811072 Starting new download in new download directory 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/58&download=true 2024-07-30T13:19:25-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/58&download=true HTTP/1.1" 302 0 2024-07-30T13:19:25-0500 [INFO ] dandi 87089:6185463808 File 0/0/21/56 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/56 has correct md5 b92bdd5ad8fec1f42c02fd39087200d7 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6185463808 Starting new download in new download directory 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/59&download=true 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:25-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F5%2F28%2F31 2024-07-30T13:19:25-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/59&download=true HTTP/1.1" 302 0 2024-07-30T13:19:26-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181925Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=ac3eb5c395d537378a60238804259a3ff4b87e1b17805fdfef2cb85127cd894d HTTP/1.1" 200 2686299 2024-07-30T13:19:26-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/59?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181925Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=9a73d9b40629fca7852d28321afa4c7157f6cd4164656c542a3ad3eac80b8a9c HTTP/1.1" 200 2674468 2024-07-30T13:19:26-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F5%2F28%2F31 HTTP/1.1" 200 118730 2024-07-30T13:19:26-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:26-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F6%2F20%2F24 2024-07-30T13:19:26-0500 [INFO ] dandi 87089:6185463808 File 0/0/21/59 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:26-0500 [DEBUG ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/59 has correct md5 904f4f16663f28332e3d8e2a8163535d 2024-07-30T13:19:26-0500 [DEBUG ] dandi 87089:6185463808 Starting new download in new download directory 2024-07-30T13:19:26-0500 [DEBUG ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/6&download=true 2024-07-30T13:19:27-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/6&download=true HTTP/1.1" 302 0 2024-07-30T13:19:27-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181926Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=9941ba5054930e984204ecdfec08515e7d0e0d36113e11422c6b70e073e5c363 HTTP/1.1" 200 2613746 2024-07-30T13:19:27-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F6%2F20%2F24 HTTP/1.1" 200 118773 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F13%2F17 2024-07-30T13:19:27-0500 [INFO ] dandi 87089:6185463808 File 0/0/21/6 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/6 has correct md5 b94968e81ae75e8f143da5b5740be73e 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6185463808 Starting new download in new download directory 2024-07-30T13:19:27-0500 [INFO ] dandi 87089:6134984704 File 0/0/21/54 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6185463808 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/60&download=true 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/54 has correct md5 d84164ebd71daa9a0ad5735a50e3e0d8 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6134984704 Starting new download in new download directory 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6134984704 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/61&download=true 2024-07-30T13:19:27-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F13%2F17 HTTP/1.1" 200 119042 2024-07-30T13:19:27-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/60&download=true HTTP/1.1" 302 0 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:27-0500 [DEBUG ] urllib3.connectionpool 87089:6134984704 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/61&download=true HTTP/1.1" 302 0 2024-07-30T13:19:27-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F7%2F1 2024-07-30T13:19:28-0500 [DEBUG ] urllib3.connectionpool 87089:6185463808 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/60?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181927Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=50de1e9f0200ecdb414aa1e5f4b979fa46b6cca1d966341b473eb06a88e7efb9 HTTP/1.1" 200 2681247 2024-07-30T13:19:28-0500 [DEBUG ] urllib3.connectionpool 87089:6134984704 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/61?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181927Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=4bd042babf66bc9ad2f961414c74e6449813d688215637fa5370f7458314a7d5 HTTP/1.1" 200 2656592 2024-07-30T13:19:28-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F7%2F7%2F1 HTTP/1.1" 200 119199 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F8%2F30%2F35 2024-07-30T13:19:28-0500 [INFO ] dandi 87089:6168637440 File 0/0/21/57 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:28-0500 [INFO ] dandi 87089:6134984704 File 0/0/21/61 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/61 has correct md5 a95b9e4c5915bbdf37867fa2a8e8752a 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/57 has correct md5 169080f9518ba092676cdbd0279ce57f 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6168637440 Starting new download in new download directory 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6134984704 Starting new download in new download directory 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6168637440 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/63&download=true 2024-07-30T13:19:28-0500 [DEBUG ] dandi 87089:6134984704 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/62&download=true 2024-07-30T13:19:29-0500 [DEBUG ] urllib3.connectionpool 87089:6168637440 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/63&download=true HTTP/1.1" 302 0 2024-07-30T13:19:29-0500 [DEBUG ] urllib3.connectionpool 87089:6134984704 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/62&download=true HTTP/1.1" 302 0 2024-07-30T13:19:29-0500 [DEBUG ] urllib3.connectionpool 87089:6134984704 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/62?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181928Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=58acbd691b3fd92d46d6d131e52fed9a69f80c29a5d06b55ca97e5339d4d225e HTTP/1.1" 200 2657730 2024-07-30T13:19:29-0500 [DEBUG ] urllib3.connectionpool 87089:6168637440 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/63?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181928Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=dce88a88655339997bf1ce16a1fc02233dacdfc5e86e6fdde21cf647d4b8ea4d HTTP/1.1" 200 2678755 2024-07-30T13:19:29-0500 [INFO ] dandi 87089:6151811072 File 0/0/21/58 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:29-0500 [DEBUG ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/58 has correct md5 f2d7adb84b3f2abae4c9ec1e21d5a5df 2024-07-30T13:19:29-0500 [DEBUG ] dandi 87089:6151811072 Starting new download in new download directory 2024-07-30T13:19:29-0500 [DEBUG ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/64&download=true 2024-07-30T13:19:29-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/64&download=true HTTP/1.1" 302 0 2024-07-30T13:19:29-0500 [INFO ] dandi 87089:6168637440 File 0/0/21/63 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:29-0500 [DEBUG ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/63 has correct md5 b62031d4affb8ad34be5aae93dae7ce1 2024-07-30T13:19:29-0500 [DEBUG ] dandi 87089:6168637440 Starting new download in new download directory 2024-07-30T13:19:29-0500 [DEBUG ] dandi 87089:6168637440 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/65&download=true 2024-07-30T13:19:30-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F8%2F30%2F35 HTTP/1.1" 200 118872 2024-07-30T13:19:30-0500 [DEBUG ] urllib3.connectionpool 87089:6168637440 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/65&download=true HTTP/1.1" 302 0 2024-07-30T13:19:30-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:30-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F9%2F23%2F28 2024-07-30T13:19:30-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/64?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181929Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=364511498060605ef00b5ef99af92730cfe29b6b33668acf1dad857047067a20 HTTP/1.1" 200 2725934 2024-07-30T13:19:30-0500 [DEBUG ] urllib3.connectionpool 87089:6168637440 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/65?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181930Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=e216335107232e0f960675278ad2e56be8e9d9adb3352c9424f9f12cfebabafe HTTP/1.1" 200 2668276 2024-07-30T13:19:30-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=1%2F9%2F23%2F28 HTTP/1.1" 200 118601 2024-07-30T13:19:30-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:30-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F0%2F8%2F4 2024-07-30T13:19:30-0500 [INFO ] dandi 87089:6151811072 File 0/0/21/64 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:30-0500 [DEBUG ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/64 has correct md5 8ad14ae58529eca307d31f2a8a129768 2024-07-30T13:19:30-0500 [DEBUG ] dandi 87089:6151811072 Starting new download in new download directory 2024-07-30T13:19:30-0500 [DEBUG ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/66&download=true 2024-07-30T13:19:31-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/66&download=true HTTP/1.1" 302 0 2024-07-30T13:19:31-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F0%2F8%2F4 HTTP/1.1" 200 118901 2024-07-30T13:19:31-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:31-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F2%2F0%2F12 2024-07-30T13:19:31-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/66?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181930Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=e15aeac9910655184d3c83ecf471a9fb468a0f51f08d88dc81f785fc568eb9a8 HTTP/1.1" 200 2660776 2024-07-30T13:19:31-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F2%2F0%2F12 HTTP/1.1" 200 118489 2024-07-30T13:19:31-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:31-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F5%2F1%2F6 2024-07-30T13:19:31-0500 [INFO ] dandi 87089:6151811072 File 0/0/21/66 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:31-0500 [DEBUG ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/66 has correct md5 2b15a1dbfa01eff8b9c15aac257781cb 2024-07-30T13:19:31-0500 [DEBUG ] dandi 87089:6151811072 Starting new download in new download directory 2024-07-30T13:19:31-0500 [DEBUG ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/67&download=true 2024-07-30T13:19:32-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/67&download=true HTTP/1.1" 302 0 2024-07-30T13:19:32-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F5%2F1%2F6 HTTP/1.1" 200 118561 2024-07-30T13:19:32-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/67?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181931Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=0ac17568c826a86953ae1c251484c57dd50d0b376f26dfc5bc3f87c0cda0c011 HTTP/1.1" 200 2655398 2024-07-30T13:19:32-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:32-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F8%2F11%2F14 2024-07-30T13:19:33-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=2%2F8%2F11%2F14 HTTP/1.1" 200 118273 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:33-0500 [INFO ] dandi 87089:6134984704 File 0/0/21/62 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:33-0500 [INFO ] dandi 87089:6151811072 File 0/0/21/67 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6112997376 GET https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=3%2F5%2F0%2F6 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/62 has correct md5 c368c5d8eab19ba140443a3992f36359 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/67 has correct md5 ed8176f25a2f2748b8228fcbe5e1edb5 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6134984704 Starting new download in new download directory 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6151811072 Starting new download in new download directory 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6134984704 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/68&download=true 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6151811072 Starting download from https://api.dandiarchive.org/api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/69&download=true 2024-07-30T13:19:33-0500 [DEBUG ] urllib3.connectionpool 87089:6134984704 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/68&download=true HTTP/1.1" 302 0 2024-07-30T13:19:33-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?prefix=0/0/21/69&download=true HTTP/1.1" 302 0 2024-07-30T13:19:33-0500 [DEBUG ] urllib3.connectionpool 87089:6112997376 https://api.dandiarchive.org:443 "GET /api/zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/files/?after=3%2F5%2F0%2F6 HTTP/1.1" 200 17657 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6112997376 Response: 200 2024-07-30T13:19:33-0500 [DEBUG ] urllib3.connectionpool 87089:6151811072 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/69?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181933Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=5a8dbcbdec5895f7a48170d17e99ecb5f2078a55f3804bd355ce9521c1f17529 HTTP/1.1" 200 46969 2024-07-30T13:19:33-0500 [DEBUG ] urllib3.connectionpool 87089:6134984704 https://dandiarchive.s3.amazonaws.com:443 "GET /zarr/1904d2d9-ed31-4cf5-b723-eedff28d2c7c/0/0/21/68?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUBRWC5GAEKH3223E%2F20240730%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240730T181933Z&X-Amz-Expires=21600&X-Amz-SignedHeaders=host&X-Amz-Signature=f13e1927a0ff0d3296a9473e857eefa17392a52a446991391a1ae8ad46be92ce HTTP/1.1" 200 500088 2024-07-30T13:19:33-0500 [INFO ] dandi 87089:6151811072 File 0/0/21/69 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:33-0500 [INFO ] dandi 87089:6168637440 File 0/0/21/65 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6151811072 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/69 has correct md5 add4f234b32b690e4b59eaa2e3af3f3e 2024-07-30T13:19:33-0500 [DEBUG ] dandi 87089:6168637440 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/65 has correct md5 9a7ae1fca026f6fd9c6ffedbb8a6e05a 2024-07-30T13:19:34-0500 [INFO ] dandi 87089:6134984704 File 0/0/21/68 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:34-0500 [DEBUG ] dandi 87089:6134984704 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/68 has correct md5 120b2982407d875633da290ea7119418 2024-07-30T13:19:35-0500 [INFO ] dandi 87089:6185463808 File 0/0/21/60 in Zarr 1904d2d9-ed31-4cf5-b723-eedff28d2c7c successfully downloaded 2024-07-30T13:19:35-0500 [DEBUG ] dandi 87089:6185463808 Verified that sub-I58_sample-01_chunk-03_hipCT.ome.zarr/0/0/21/60 has correct md5 63e9b19ddb0ccd436ae221dd5c2465bb 2024-07-30T13:19:44-0500 [ERROR ] dandi 87089:6112997376 Caught while downloading sub-I58_sample-01_chunk-03_hipCT.ome.zarr: Traceback (most recent call last): File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 360, in _download_generator_guard yield from generator File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 918, in _download_zarr for path, status in it: File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 346, in __next__ return self.get() File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 380, in get self._end() File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 400, in _end e.get() File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 110, in get raise e.with_traceback(tb) from None File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/interleave/__init__.py", line 274, in _process x = next(it) File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 981, in pairing for d in gen: File "/Users/kabilar/miniconda3/envs/dandi/lib/python3.10/site-packages/dandi/download.py", line 574, in _download_file raise FileExistsError(f"File {path!r} already exists") FileExistsError: File PosixPath('sub-I58_sample-01_chunk-03_hipCT.ome.zarr/.zattrs') already exists 2024-07-30T13:19:44-0500 [INFO ] dandi 87089:8265534464 Logs saved in /Users/kabilar/Library/Logs/dandi-cli/2024.07.30-18.14.56Z-87089.log ```

Thank you.

cc @satra @aaronkanzer

yarikoptic commented 4 months ago

behaves as expected:

❯ dandi download --help | grep existing
     dandi download https://identifiers.org/DANDI:000027 --existing refresh
  -e, --existing [error|skip|overwrite|overwrite-different|refresh]
                                  What to do if a file found existing locally.
kabilar commented 4 months ago

Thanks Yarik. Apologize for missing this option.