search

ESGF / esgf-download

ESGF data transfer and replication tool
https://esgf.github.io/esgf-download/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Downloaded file is larger than expected #13

Closed AtefBN closed 1 year ago

AtefBN commented 1 year ago

[2023-06-17 12:41:47] DEBUG root Locals: { 'self': <httpcore.AsyncConnectionPool object at 0x7f60354c67d0>, 'requests_still_in_flight': 1, 'connection': <AsyncHTTPConnection ['https://esgf-data.ucar.edu:443', HTTP/1.1, CLOSED, Request Count: 1]> }

[2023-06-17 12:41:47] ERROR root

Traceback (most recent call last): File "/gpfscmip/gpfsdata/esgf/miniconda/envs/esgpull/lib/python3.11/site-packages/esgpull/processor.py", line 88, in stream raise DownloadSizeError(ctx.completed, ctx.file.size) esgpull.exceptions.DownloadSizeError: Downloaded file is larger than expected: 5605850 > 5605846

During handling of the above exception, another exception occurred:

svenrdz commented 1 year ago
I found 4 files with a size of 5605846 bytes: file data_node
CMIP6.DCPP.EC-Earth-Consortium.EC-Earth3-CC.dcppB-forecast.s2020-r2i1p1f1.Amon.pr.gr.v20220503.pr_Amon_EC-Earth3-CC_dcppB-forecast_s2020-r2i1p1f1_gr_202611-202710.nc esgf-data1.llnl.gov
CMIP6.DCPP.EC-Earth-Consortium.EC-Earth3-CC.dcppB-forecast.s2020-r2i1p1f1.Amon.pr.gr.v20220503.pr_Amon_EC-Earth3-CC_dcppB-forecast_s2020-r2i1p1f1_gr_202611-202710.nc esgf.bsc.es
CMIP6.DCPP.EC-Earth-Consortium.EC-Earth3.dcppA-hindcast.s1960-r2i1p1f1.Amon.pr.gr.v20200213.pr_Amon_EC-Earth3_dcppA-hindcast_s1960-r2i1p1f1_gr_196211-196310.nc esgf.bsc.es
CMIP6.DCPP.EC-Earth-Consortium.EC-Earth3.dcppA-hindcast.s1960-r2i1p1f1.Amon.pr.gr.v20200213.pr_Amon_EC-Earth3_dcppA-hindcast_s1960-r2i1p1f1_gr_196211-196310.nc esgf.nci.org.au

I cannot download the files from esgf.bsc.es, the data_node seems to be down at the moment.

For the other two, downloading went fine.

Since I can't reproduce your issue, can you provide a bit more context ? Was the file you tried to download one of the above ?

AtefBN commented 1 year ago

File(file_id='CMIP6.DCPP.EC-Earth-Consortium.EC-Earth3.dcppA-hindcast.s1992-r3i1p1f1.Amon.psl.gr.v20200213.psl_Amon_EC-Earth3_dcppA-hindcast_s1992 -r3i1p1f1_gr_199611-199710.nc', dataset_id='CMIP6.DCPP.EC-Earth-Consortium.EC-Earth3.dcppA-hindcast.s1992-r3i1p1f1.Amon.psl.gr.v20200213', master_id='CMIP6.DCPP.EC-Earth-Consortium.EC-Earth3.dcppA-hindcast.s1992-r3i1p1f1.Amon.psl.gr.psl_Amon_EC-Earth3_dcppA-hindcast_s1992-r3i1p1f1gr 199611-199710.nc', url='https://esgf.bsc.es/thredds/fileServer/esg_dataroot/a1ua-DCPP-r3/CMIP6/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/s1992-r3i1p1f1/Amon/ psl/gr/v20200213/psl_Amon_EC-Earth3_dcppA-hindcast_s1992-r3i1p1f1_gr_199611-199710.nc', version='v20200213', filename='psl_Amon_EC-Earth3_dcppA-hindcast_s1992-r3i1p1f1_gr_199611-199710.nc', local_path='CMIP6/DCPP/EC-Earth-Consortium/EC-Earth3/dcppA-hindcast/s1992-r3i1p1f1/Amon/psl/gr/v20200213', data_node='esgf.bsc.es', checksum='9d5bc9c4d16cf738601a2fcd5120f5d6470bcca98f7e45f25fe87de122051836', checksum_type='SHA256', size=4398846, status=<FileStatus.Starting: 'starting'>)

svenrdz commented 1 year ago

This should now be fixed with 8e0de5679d8a62526c899e20d9671a2b5a6f5962, I'm currently making the 0.5.5 tag and conda package.