search

Project-MONAI / model-zoo

MONAI Model Zoo that hosts models in the MONAI Bundle format.
Apache License 2.0
190 stars 67 forks source link

sharing schema.json to avoid repeated downloading #111

Closed wyli closed 2 years ago

wyli commented 2 years ago

Is your feature request related to a problem? Please describe. it may run out of API request limit to download the schema file

schema.json: 0.00B [00:05, ?B/s]2022-08-31 06:41:00,599 - ERROR - Download failed from https://github.com/Project-MONAI/MONAI-extra-test-data/releases/download/0.8.1/meta_schema_20220324.json to /tmp/tmppgkh7ytr/schema.json.
[2022-08-31T06:41:01.862Z] 
[2022-08-31T06:41:01.862Z] Traceback (most recent call last):
[2022-08-31T06:41:01.862Z]   File "/home/jenkins/agent/workspace/MONAI-zoo-premerge/ci/verify_bundle.py", line 175, in <module>
[2022-08-31T06:41:01.862Z]     verify(bundle)
[2022-08-31T06:41:01.862Z]   File "/home/jenkins/agent/workspace/MONAI-zoo-premerge/ci/verify_bundle.py", line 153, in verify
[2022-08-31T06:41:01.862Z]     verify_metadata_format(bundle_path)
[2022-08-31T06:41:01.862Z]   File "/home/jenkins/agent/workspace/MONAI-zoo-premerge/ci/verify_bundle.py", line 87, in verify_metadata_format
[2022-08-31T06:41:01.862Z]     verify_metadata(
[2022-08-31T06:41:01.862Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/bundle/scripts.py", line 429, in verify_metadata
[2022-08-31T06:41:01.862Z]     download_url(url=url, filepath=filepath_, hash_val=hash_val_, hash_type=hash_type_, progress=True)
[2022-08-31T06:41:01.862Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/apps/utils.py", line 203, in download_url
[2022-08-31T06:41:01.862Z]     _download_with_progress(url, tmp_name, progress=progress)
[2022-08-31T06:41:01.862Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/apps/utils.py", line 114, in _download_with_progress
[2022-08-31T06:41:01.862Z]     raise e
[2022-08-31T06:41:01.862Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/apps/utils.py", line 107, in _download_with_progress
[2022-08-31T06:41:01.862Z]     urlretrieve(url, filepath, reporthook=t.update_to)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 247, in urlretrieve
[2022-08-31T06:41:01.862Z]     with contextlib.closing(urlopen(url, data)) as fp:
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
[2022-08-31T06:41:01.862Z]     return opener.open(url, data, timeout)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 531, in open
[2022-08-31T06:41:01.862Z]     response = meth(req, response)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
[2022-08-31T06:41:01.862Z]     response = self.parent.error(
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 563, in error
[2022-08-31T06:41:01.862Z]     result = self._call_chain(*args)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
[2022-08-31T06:41:01.862Z]     result = func(*args)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 755, in http_error_302
[2022-08-31T06:41:01.862Z]     return self.parent.open(new, timeout=req.timeout)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 531, in open
[2022-08-31T06:41:01.862Z]     response = meth(req, response)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
[2022-08-31T06:41:01.862Z]     response = self.parent.error(
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 569, in error
[2022-08-31T06:41:01.862Z]     return self._call_chain(*args)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
[2022-08-31T06:41:01.862Z]     result = func(*args)
[2022-08-31T06:41:01.862Z]   File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
[2022-08-31T06:41:01.862Z]     raise HTTPError(req.full_url, code, msg, hdrs, fp)
[2022-08-31T06:41:01.862Z] urllib.error.HTTPError: HTTP Error 503: Egress is over the account limit.
script returned exit code 1

the schema.json could be shared across the model zoo items to avoid repeated downloading https://github.com/Project-MONAI/model-zoo/blob/db6ce75b4df89c76117d2b8eccc94e0495cf24ed/ci/verify_bundle.py#L89

yiheng-wang-nv commented 2 years ago

Thanks @wyli , does the error happen when there are multiple bundles are changed in a PR (thus the same schema is downloaded multiple times) ?

wyli commented 2 years ago

@yiheng-wang-nv, I'm not able to always replicate this issue, sometimes it happens

yiheng-wang-nv commented 2 years ago

Updates:

I met the same issue when running blossom CI tests for multiple bundles (https://github.com/Project-MONAI/model-zoo/actions/runs/3209040893/jobs/5245429001#step:2:784)

I will create a PR to fix this issue first.

yiheng-wang-nv commented 2 years ago

related topic: https://github.com/community/community/discussions/8535

yiheng-wang-nv commented 2 years ago

This ticket is addressed within #141