ImagingDataCommons / idc-index

Python package to simplify access to the data available from NCI Imaging Data Commons
https://idc-index.readthedocs.io/
MIT License
11 stars 5 forks source link

Unable to download from manifest produced with IDC Portal #91

Closed fedorov closed 5 months ago

fedorov commented 5 months ago

Follow up on #72

@vkt1414 the attached manifest was created using portal, after selecting LIDC collection.

When I try to download the content, I am getting this error. Can you please investigate?

$ idc download-from-manifest --manifest-file ~/Downloads/aws_manifest.s5cmd --download-dir ./test_download
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/bin/idc", line 8, in <module>
    sys.exit(main())
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/idc_index/cli.py", line 246, in download_from_manifest
    client.download_from_manifest(
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/idc_index/index.py", line 1167, in download_from_manifest
    ) = self._validate_update_manifest_and_get_download_size(
  File "/usr/local/Caskroom/miniconda/base/envs/idc-index/lib/python3.9/site-packages/idc_index/index.py", line 619, in _validate_update_manifest_and_get_download_size
    raise ValueError(
ValueError: Either GCS bucket path is invalid or manifest has a mix of GCS and AWS urls. If so, please use urls from one provider only

file_manifest_aws.s5cmd.zip

fedorov commented 5 months ago

I think I messed this one up - I probably used a wrong manifest!