search

dandi / dandi-cli

DANDI command line client to facilitate common operations
https://dandi.readthedocs.io/
Apache License 2.0
21 stars 25 forks source link

Improve feedback when there's a validation error #1469

Open kabilar opened 2 months ago

kabilar commented 2 months ago

Reposting a suggestion that @dstansby provided on the LINC Client (https://github.com/lincbrain/linc-cli/issues/44):

I'm currently trying to upload a dataset, and if there's a validation error it's a bit tricky to understand what went wrong and how I can fix it. The output looks like this:

PATH                                                      SIZE        ERRORS     PROGRESS STATUS           MESSAGE                  
dandiset.yaml                                             1.1 kB                          skipped          should be edited online  
dataset_description.json                                  98 Bytes       2                ERROR            failed validation        
sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json     65 Bytes       1                ERROR            failed validation        
sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.ome.zarr                2                ERROR            failed validation        
Summary:                                                  1.3 kB   3 with errors          1 skipped        1 should be edited online
                                                                                          3 ERROR          3 failed validation      
2024-04-25 13:44:09,332 [ WARNING] One or more assets failed validation.  Consult the logfile for details.
2024-04-25 13:44:09,861 [ WARNING] Upload failed, and you are not using the latest version of lincbrain-cli.  We suggest upgrading lincbrain-cli to v0.9.0 and trying again.
2024-04-25 13:44:09,864 [    INFO] Logs saved in /Users/dstansby/Library/Logs/lincbrain-cli/20240425124405Z-15907.log
Error: failed validation

If I then go into the log, I see messages like

024-04-25T13:44:09+0100 [WARNING ] dandi 15907:6207451136  Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=<Scope.FILE: 'file'>, severity=<Severity.ERROR: 3>, asset_paths=None, within_asset_paths=None, dandiset_path=PosixPath('/Volumes/LaCie/linc_staging/000008'), dataset_path=PosixPath('/Volumes/LaCie/linc_staging/000008'), message='File does not match any pattern known to BIDS.', metadata=None, path=PosixPath('/Volumes/LaCie/linc_staging/000008/sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json'), path_regex=None)
2024-04-25T13:44:09+0100 [ERROR   ] dandi 15907:6207451136 Error uploading /Volumes/LaCie/linc_staging/000008/sub-s45/micr/sub-s45_sample-fullhemisphere_hipCT.json:
Example full log ``` 2024-05-02T14:40:51+0100 [INFO ] dandi 23264:16516 lincbrain v0.5.0+108.g40a8c76b.dirty, hdmf v3.13.0, pynwb v2.6.0, h5py v3.11.0 2024-05-02T14:40:51+0100 [INFO ] dandi 23264:16516 sys.argv = ['C:\\Users\\dstansby\\miniforge3\\envs\\linc\\Scripts\\lincbrain', 'upload'] 2024-05-02T14:40:51+0100 [INFO ] dandi 23264:16516 os.getcwd() = C:\Users\dstansby\data\linc\000005 2024-05-02T14:40:51+0100 [DEBUG ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): rig.mit.edu:443 2024-05-02T14:40:52+0100 [DEBUG ] urllib3.connectionpool 23264:16516 https://rig.mit.edu:443 "GET /et/projects/linc/linc-cli HTTP/1.1" 404 44 2024-05-02T14:40:52+0100 [DEBUG ] dandi 23264:16516 Could not check linc/linc-cli for version updates: 404 Client Error: Not Found for url: https://rig.mit.edu/et/projects/linc/linc-cli 2024-05-02T14:40:52+0100 [DEBUG ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): api.lincbrain.org:443 2024-05-02T14:40:52+0100 [DEBUG ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/info/ HTTP/1.1" 200 371 2024-05-02T14:40:52+0100 [DEBUG ] dandi 23264:16516 GET https://api.lincbrain.org/api/info/ 2024-05-02T14:40:52+0100 [DEBUG ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): api.lincbrain.org:443 2024-05-02T14:40:53+0100 [DEBUG ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/info/ HTTP/1.1" 200 371 2024-05-02T14:40:53+0100 [DEBUG ] dandi 23264:16516 Response: 200 2024-05-02T14:40:53+0100 [DEBUG ] dandi 23264:16516 Using api key from LINCBRAIN_API_KEY environment variable 2024-05-02T14:40:53+0100 [DEBUG ] dandi 23264:16516 GET https://api.lincbrain.org/api/auth/token 2024-05-02T14:40:53+0100 [DEBUG ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/auth/token HTTP/1.1" 301 0 2024-05-02T14:40:53+0100 [DEBUG ] urllib3.connectionpool 23264:16516 https://api.lincbrain.org:443 "GET /api/auth/token/ HTTP/1.1" 200 42 2024-05-02T14:40:53+0100 [DEBUG ] dandi 23264:16516 Response: 200 2024-05-02T14:40:53+0100 [DEBUG ] dandi 23264:16516 Found identifier DANDI:000005 in top level 'identifier' 2024-05-02T14:40:53+0100 [DEBUG ] h5py._conv 23264:16516 Creating converter from 7 to 5 2024-05-02T14:40:53+0100 [DEBUG ] h5py._conv 23264:16516 Creating converter from 5 to 7 2024-05-02T14:40:53+0100 [DEBUG ] h5py._conv 23264:16516 Creating converter from 7 to 5 2024-05-02T14:40:53+0100 [DEBUG ] h5py._conv 23264:16516 Creating converter from 5 to 7 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'zlib' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'gzip' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'bz2' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'lzma' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'blosc' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'zstd' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'lz4' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'astype' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'delta' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'quantize' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'fixedscaleoffset' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'packbits' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'categorize' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'pickle' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'base64' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'shuffle' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'bitround' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'crc32' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'adler32' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'jenkins_lookup3' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'json2' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'vlen-utf8' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'vlen-bytes' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'vlen-array' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'fletcher32' 2024-05-02T14:40:53+0100 [DEBUG ] numcodecs 23264:16516 Registering codec 'n5_wrapper' 2024-05-02T14:40:54+0100 [INFO ] dandi 23264:16516 Found 6 files to consider 2024-05-02T14:40:55+0100 [WARNING ] bids-schema 23264:17416 No BIDS reference root provided. 2024-05-02T14:40:55+0100 [INFO ] bids-schema 23264:17416 No schema path specified, defaulting to the bundled schema, `C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\bidsschematools\data\schema`. 2024-05-02T14:40:55+0100 [WARNING ] bids-schema 23264:17416 The `C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr/` file was not matched by any regex schema entry. 2024-05-02T14:40:56+0100 [WARNING ] dandi 23264:17416 'C:\\Users\\dstansby\\data\\linc\\000005\\dataset_description.json' had 1 validation errors preventing its upload: 2024-05-02T14:40:56+0100 [WARNING ] dandi 23264:17416 Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=, severity=, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None) 2024-05-02T14:40:56+0100 [ERROR ] dandi 23264:17416 Error uploading C:\Users\dstansby\data\linc\000005\dataset_description.json: Traceback (most recent call last): File "C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\lincbrain\upload.py", line 280, in process_path raise UploadError("failed validation") lincbrain.exceptions.UploadError: failed validation 2024-05-02T14:40:56+0100 [DEBUG ] fscacher.cache 23264:29792 Calling memoized version of for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json 2024-05-02T14:40:56+0100 [DEBUG ] fscacher.cache 23264:22488 Calling memoized version of for C:\Users\dstansby\data\linc\000005\samples.tsv 2024-05-02T14:40:56+0100 [DEBUG ] fscacher.cache 23264:30128 Calling memoized version of for C:\Users\dstansby\data\linc\000005\participants.tsv 2024-05-02T14:40:56+0100 [DEBUG ] fscacher.cache 23264:29792 Running original on WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json') 2024-05-02T14:40:56+0100 [DEBUG ] fscacher.cache 23264:29792 Calling memoized version of for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json 2024-05-02T14:40:56+0100 [DEBUG ] dandi 23264:22488 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/ 2024-05-02T14:40:56+0100 [DEBUG ] dandi 23264:30128 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/ 2024-05-02T14:40:56+0100 [DEBUG ] urllib3.connectionpool 23264:30128 Starting new HTTPS connection (2): api.lincbrain.org:443 2024-05-02T14:40:56+0100 [DEBUG ] fscacher.cache 23264:29792 Running original on WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json') 2024-05-02T14:40:56+0100 [DEBUG ] dandi 23264:29792 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/ 2024-05-02T14:40:56+0100 [DEBUG ] urllib3.connectionpool 23264:29792 Starting new HTTPS connection (3): api.lincbrain.org:443 2024-05-02T14:40:56+0100 [DEBUG ] urllib3.connectionpool 23264:22488 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=samples.tsv HTTP/1.1" 200 274 2024-05-02T14:40:56+0100 [DEBUG ] dandi 23264:22488 Response: 200 2024-05-02T14:40:56+0100 [DEBUG ] dandi 23264:22488 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/d5c15a56-afd6-4b3b-a37b-b052849dbfc1/ 2024-05-02T14:40:56+0100 [DEBUG ] urllib3.connectionpool 23264:22488 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/d5c15a56-afd6-4b3b-a37b-b052849dbfc1/ HTTP/1.1" 200 1911 2024-05-02T14:40:56+0100 [DEBUG ] dandi 23264:22488 Response: 200 2024-05-02T14:40:56+0100 [DEBUG ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=rawdata%2Fsub-S45%2Fsub-S45_sample-fullhemisphere_PC.json HTTP/1.1" 200 52 2024-05-02T14:40:56+0100 [DEBUG ] dandi 23264:29792 Response: 200 2024-05-02T14:40:57+0100 [DEBUG ] urllib3.connectionpool 23264:30128 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/?path=participants.tsv HTTP/1.1" 200 278 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:30128 Response: 200 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:30128 GET https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/aef8d0c9-0cd3-46d6-a2ac-3dfdee3fa06b/ 2024-05-02T14:40:57+0100 [DEBUG ] fscacher.cache 23264:29792 Calling memoized version of for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:29792 Calculated dandi-etag of df515b2ad6855b779e78e4ec58fdd98f-1 for C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Beginning upload 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/initialize/ 2024-05-02T14:40:57+0100 [DEBUG ] urllib3.connectionpool 23264:30128 https://api.lincbrain.org:443 "GET /api/dandisets/000005/versions/draft/assets/aef8d0c9-0cd3-46d6-a2ac-3dfdee3fa06b/ HTTP/1.1" 200 1915 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:30128 Response: 200 2024-05-02T14:40:57+0100 [DEBUG ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/initialize/ HTTP/1.1" 200 649 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:29792 Response: 200 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:29792 Uploading C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.json in 1 parts 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:17160 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Uploading part 1/1 (104 bytes) 2024-05-02T14:40:57+0100 [DEBUG ] dandi 23264:17160 PUT https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=content-length%3Bhost&X-Amz-Signature=c431ce73c38b9c8c2ffe2ebbfb08e06ee5b5274731c6b0b06bd56ec39efa48b9 2024-05-02T14:40:57+0100 [DEBUG ] urllib3.connectionpool 23264:17160 Starting new HTTPS connection (1): linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443 2024-05-02T14:40:58+0100 [DEBUG ] urllib3.connectionpool 23264:17160 https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443 "PUT /blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=content-length%3Bhost&X-Amz-Signature=c431ce73c38b9c8c2ffe2ebbfb08e06ee5b5274731c6b0b06bd56ec39efa48b9 HTTP/1.1" 200 0 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:17160 Response: 200 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:17160 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Part upload finished ETag=6a057fca28c6d756dae00c6160579d31 Content-Length=0 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Completing upload 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/complete/ 2024-05-02T14:40:58+0100 [DEBUG ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/complete/ HTTP/1.1" 200 768 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 Response: 200 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Announcing completion to https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 POST https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1 2024-05-02T14:40:58+0100 [DEBUG ] urllib3.connectionpool 23264:29792 https://linc-brain-mit-prod-us-east-2.s3.amazonaws.com:443 "POST /blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed?uploadId=Z2kNpaZNkweXvXF89FWh.LbEXpd2k8DDWQkao682wEbuzwEh0KyCM53smXHSRhUjZP6N4ws8RzVP.cuG2rPVAV1Yf4H_ESgTNh.cZW0b89lFQDgo0UTWRU2puiaw7Wk2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIASGOXNDXVRY2HBTF3%2F20240502%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240502T134058Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=3b46212653ca7ccabaeae1a69ddce210dde5e5b6f29e186495fad5dac065b4b1 HTTP/1.1" 200 None 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 Response: 200 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Upload completed. Response content: b'\n\nhttps://linc-brain-mit-prod-us-east-2.s3.amazonaws.com/blobs%2F082%2Fcda%2F082cda3f-a994-4728-bf34-8108ab122bedlinc-brain-mit-prod-us-east-2blobs/082/cda/082cda3f-a994-4728-bf34-8108ab122bed"df515b2ad6855b779e78e4ec58fdd98f-1"' 2024-05-02T14:40:58+0100 [DEBUG ] charset_normalizer 23264:29792 Encoding detection: utf_8 is most likely the one. 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 POST https://api.lincbrain.org/api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/validate/ 2024-05-02T14:40:58+0100 [DEBUG ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/uploads/082cda3f-a994-4728-bf34-8108ab122bed/validate/ HTTP/1.1" 200 119 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 Response: 200 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Assigning asset blob to dandiset & version 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 POST https://api.lincbrain.org/api/dandisets/000005/versions/draft/assets/ 2024-05-02T14:40:58+0100 [DEBUG ] urllib3.connectionpool 23264:29792 https://api.lincbrain.org:443 "POST /api/dandisets/000005/versions/draft/assets/ HTTP/1.1" 200 2182 2024-05-02T14:40:58+0100 [DEBUG ] dandi 23264:29792 Response: 200 2024-05-02T14:40:58+0100 [INFO ] dandi 23264:29792 rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.json: Asset successfully uploaded 2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616 'C:\\Users\\dstansby\\data\\linc\\000005\\rawdata\\sub-S45\\sub-S45_sample-fullhemisphere_PC.ome.zarr' had 2 validation errors preventing its upload: 2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616 Error 1: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=, severity=, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None) 2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:1616 Error 2: ValidationResult(id='BIDS.NON_BIDS_PATH_PLACEHOLDER', origin=ValidationOrigin(name='bidsschematools', version='0.7.2', bids_version='1.9.0-dev'), scope=, severity=, asset_paths=None, within_asset_paths=None, dandiset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), dataset_path=WindowsPath('C:/Users/dstansby/data/linc/000005'), message='File does not match any pattern known to BIDS.', metadata=None, path=WindowsPath('C:/Users/dstansby/data/linc/000005/rawdata/sub-S45/sub-S45_sample-fullhemisphere_PC.ome.zarr'), path_regex=None) 2024-05-02T14:41:05+0100 [ERROR ] dandi 23264:1616 Error uploading C:\Users\dstansby\data\linc\000005\rawdata\sub-S45\sub-S45_sample-fullhemisphere_PC.ome.zarr: Traceback (most recent call last): File "C:\Users\dstansby\miniforge3\envs\linc\Lib\site-packages\lincbrain\upload.py", line 280, in process_path raise UploadError("failed validation") lincbrain.exceptions.UploadError: failed validation 2024-05-02T14:41:05+0100 [WARNING ] dandi 23264:16516 One or more assets failed validation. Consult the logfile for details. 2024-05-02T14:41:05+0100 [DEBUG ] urllib3.connectionpool 23264:16516 Starting new HTTPS connection (1): rig.mit.edu:443 2024-05-02T14:41:06+0100 [DEBUG ] urllib3.connectionpool 23264:16516 https://rig.mit.edu:443 "GET /et/projects/lincbrain/linc-cli HTTP/1.1" 200 578 2024-05-02T14:41:06+0100 [WARNING ] dandi 23264:16516 Upload failed, and you are not using the latest version of lincbrain-cli. We suggest upgrading lincbrain-cli to v0.9.0 and trying again. 2024-05-02T14:41:06+0100 [DEBUG ] dandi 23264:16516 Caught exception failed validation 2024-05-02T14:41:06+0100 [INFO ] dandi 23264:16516 Logs saved in C:\Users\dstansby\AppData\Local\lincbrain\lincbrain-cli\Logs\20240502134051Z-23264.log ```

These messages are hard to parse by eye, and even then the error is just "File does not match any pattern known to BIDS". It would be super useful if there was some hint (maybe at least a link to the BIDS spec?) on how the file pattern can be fixed.

Related issue:

  1. https://github.com/dandi/dandi-cli/issues/1397

Thank you

cc @dstansby @aaronkanzer

yarikoptic commented 4 weeks ago

These messages are hard to parse by eye, and even then the error is just "File does not match any pattern known to BIDS". It would be super useful if there was some hint (maybe at least a link to the BIDS spec?) on how the file pattern can be fixed.

I don't think there is an easy generic way to address this reliably. Only may be for some cases to look at what entities used and either all are known and suggest closest matching legit ones if some nonlegit ones used.

May be if you provide some specific suggestions, we could see if could be coded up.