Ouranosinc / pavics-vdb

Store virtual netCDF file aggregations and metadata fixes
0 stars 0 forks source link

Add try except to catch http errors due to datasets requiring authentication #70

Closed huard closed 1 year ago

tlvu commented 1 year ago

Successful run, took less than 1 minute, merging this PR:

$ deployment/launch_container catalog/gen_catalog.conf
==========
launch_container 'gen_catalog' START_TIME=2023-02-08T18:10:48-05:00
+ realpath catalog/gen_catalog.conf
+ CONFIG_FILE_PATH=/zstore/repos/pavics-vdb/catalog/gen_catalog.conf
+ dirname /zstore/repos/pavics-vdb/catalog/gen_catalog.conf
+ CONFIG_FILE_DIR=/zstore/repos/pavics-vdb/catalog
+ realpath /zstore/repos/pavics-vdb/catalog/.
+ WORK_DIR=/zstore/repos/pavics-vdb/catalog
+ id -u
+ USER_ID=1000
+ id -g
+ GROUP_ID=1000
+ eval echo $CATALOG_OUTPATH
+ echo
+ [ -z  ]
+ eval CATALOG_OUTPATH=$CATALOG_OUTPATH_DEFAULT
+ CATALOG_OUTPATH=/data/catalog
+ eval mkdir -p $CATALOG_OUTPATH
+ mkdir -p /data/catalog
+ eval echo --volume $CATALOG_OUTPATH:$CATALOG_OUTPATH:rw
+ echo --volume /data/catalog:/data/catalog:rw
+ VOLUME_MOUNT_OPTS= --volume /data/catalog:/data/catalog:rw
+ eval echo --env CATALOG_OUTPATH=$CATALOG_OUTPATH
+ echo --env CATALOG_OUTPATH=/data/catalog
+ ENV_OPTS= --env CATALOG_OUTPATH=/data/catalog
+ docker run --rm --name gen_catalog --volume /data/catalog:/data/catalog:rw --volume /zstore/repos/pavics-vdb/catalog/gen_catalog.conf:/zstore/repos/pavics-vdb/catalog/gen_catalog.conf:ro --volume /zstore/repos/pavics-vdb/catalog:/zstore/repos/pavics-vdb/catalog:
ro --workdir /zstore/repos/pavics-vdb/catalog --env CATALOG_OUTPATH=/data/catalog --user 1000:1000 --user 1000:1000 pavics/workflow-tests:210527.1-update20210705 python cli.py
ERROR:tds:401 Client Error: Unauthorized for url: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip6/ouranos/ESPO-G/ESPO-G6v1.0.0/catalog.xml
Traceback (most recent call last):
  File "/zstore/repos/pavics-vdb/catalog/tds.py", line 27, in walk
    child = ref.follow()
  File "/opt/conda/envs/birdy/lib/python3.7/site-packages/siphon/catalog.py", line 450, in follow
    return TDSCatalog(self.href)
  File "/opt/conda/envs/birdy/lib/python3.7/site-packages/siphon/catalog.py", line 282, in __init__
    resp.raise_for_status()
  File "/opt/conda/envs/birdy/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip6/ouranos/ESPO-G/ESPO-G6v1.0.0/catalog.xml
ERROR:tds:401 Client Error: Unauthorized for url: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip5/ouranos/ESPO-R/ESPO-R5v1.0.0/catalog.xml
Traceback (most recent call last):
  File "/zstore/repos/pavics-vdb/catalog/tds.py", line 27, in walk
    child = ref.follow()
  File "/opt/conda/envs/birdy/lib/python3.7/site-packages/siphon/catalog.py", line 450, in follow
    return TDSCatalog(self.href)
  File "/opt/conda/envs/birdy/lib/python3.7/site-packages/siphon/catalog.py", line 282, in __init__
    resp.raise_for_status()
  File "/opt/conda/envs/birdy/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip5/ouranos/ESPO-R/ESPO-R5v1.0.0/catalog.xml
+ cleanup_on_exit
+ set +x

launch_container 'gen_catalog' finished START_TIME=2023-02-08T18:10:48-05:00
launch_container 'gen_catalog' finished   END_TIME=2023-02-08T18:11:31-05:00