search

geoschem / geos-chem

GEOS-Chem "Science Codebase" repository. Contains GEOS-Chem science routines, run directory generation scripts, and interface code. This repository is used as a submodule within the GCClassic and GCHP wrappers, as well as in other modeling contexts (external ESMs).
http://geos-chem.org
Other
167 stars 162 forks source link

[BUG/ISSUE] Unable to download HTAPv3 v2022-12 data #1640

Closed kilicomu closed 1 year ago

kilicomu commented 1 year ago

What institution are you from?

Wolfson Atmospheric Chemistry Laboratories

Description of the problem

I can't seem to download the new HTAPv3 data (ExtData/HEMCO/HTAPv3/v2022-12/) from geoschemdata.wustl.edu - I can get the directory structure fine, but trying to download any files returns an HTTP 403 forbidden error.

Description of troubleshooting performed

I tried downloading some other HEMCO data and it worked as expected. I double-checked that nothing is different to how I've been downloading from geoschemdata.wustl.edu - it isn't.

GEOS-Chem version

N/A

Description of modifications

N/A

Log files

$ wget http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-12/2018/HTAPv3_SO2_0.1x0.1_2018.nc

--2023-02-06 17:25:54--  http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-12/2018/HTAPv3_SO2_0.1x0.1_2018.nc
Resolving geoschemdata.wustl.edu (geoschemdata.wustl.edu)... 35.209.233.133
Connecting to geoschemdata.wustl.edu (geoschemdata.wustl.edu)|35.209.233.133|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-02-06 17:25:54 ERROR 403: Forbidden.

Software versions

N/A

msulprizio commented 1 year ago

@SaptSinha @Jourdan-He Can either of you confirm the status of the WashU site?

msulprizio commented 1 year ago

I just checked and I can access http://geoschemdata.wustl.edu/ExtData/HEMCO/ in a browser, so it seems to be operational for me.

Jourdan-He commented 1 year ago

Same here, everything seem normal when I checked just now. Could you test downloading it again?

SaptSinha commented 1 year ago

I can confirm that the WashU ExtData path is working fine. Regarding this issue, which shows the message as "ERROR 403: Forbidden.", it seems that these particular files have been uploaded to the WashU site without download permissions for the community.

kilicomu commented 1 year ago

Yes, sorry, I should have been more clear. I can access the data portal and download most things as normal, just not the HTAPv3 data which seems have different permissions.

Jourdan-He commented 1 year ago

The permission has been changed yesterday, it should work now

kilicomu commented 1 year ago

Sorry @Jourdan-He, I still receive a 403 forbidden error when trying to download the data. Also, it appears that the top-level folder has changed from v2022-12 to v2022-17 and the internal directory structure has changed - is that intentional? See example errors below:

--2023-02-08 09:39:08--  http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-07/._HTAPv3_BC_0.1x0.1_2000.nc
Reusing existing connection to geoschemdata.wustl.edu:80.
HTTP request sent, awaiting response... 403 Forbidden
2023-02-08 09:39:08 ERROR 403: Forbidden.

--2023-02-08 09:39:08--  http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-07/._HTAPv3_NH3_0.1x0.1_2013.nc
Reusing existing connection to geoschemdata.wustl.edu:80.
HTTP request sent, awaiting response... 403 Forbidden
2023-02-08 09:39:08 ERROR 403: Forbidden.

--2023-02-08 09:39:08--  http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-07/._HTAPv3_OC_0.1x0.1_2018.nc
Reusing existing connection to geoschemdata.wustl.edu:80.
HTTP request sent, awaiting response... 403 Forbidden
2023-02-08 09:39:08 ERROR 403: Forbidden.

--2023-02-08 09:39:08--  http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-07/._HTAPv3_SO2_0.1x0.1_2015.nc
Reusing existing connection to geoschemdata.wustl.edu:80.
HTTP request sent, awaiting response... 403 Forbidden
2023-02-08 09:39:08 ERROR 403: Forbidden.

I am able to download some of the files:

HTAPv3_BC_0.1x0.1_2006.nc
HTAPv3_CO_0.1x0.1_2018.nc
HTAPv3_NH3_0.1x0.1_2010.nc
HTAPv3_NH3_0.1x0.1_2011.nc
HTAPv3_NMVOC_0.1x0.1_2000.nc
HTAPv3_NMVOC_0.1x0.1_2003.nc
HTAPv3_NMVOC_0.1x0.1_2007.nc
HTAPv3_NO_0.1x0.1_2002.nc
HTAPv3_NO_0.1x0.1_2005.nc
HTAPv3_NO_0.1x0.1_2009.nc
HTAPv3_NO_0.1x0.1_2013.nc
HTAPv3_NO_0.1x0.1_2014.nc
HTAPv3_SO2_0.1x0.1_2018.nc

but everything else still gives me 403.

msulprizio commented 1 year ago

I'm also tagging @1Dandan as she is the current owner of those files on the WashU site.

1Dandan commented 1 year ago

Thanks for tagging me and sorry for the confusion. I checked the directory is till v2022-12. v2022-07 is a pre-release version. I will delete it to avoid any confusion. I do see the permission is not granted for the direcory v2022-12 and has fixed it now. All files should be accessible now. I also test downloading a file by:

wget http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-12/2018/HTAPv3_BC_0.1x0.1_2018.nc
--2023-02-08 09:05:02--  http://geoschemdata.wustl.edu/ExtData/HEMCO/HTAPv3/v2022-12/2018/HTAPv3_BC_0.1x0.1_2018.nc
Resolving geoschemdata.wustl.edu (geoschemdata.wustl.edu)... 35.209.233.133
Connecting to geoschemdata.wustl.edu (geoschemdata.wustl.edu)|35.209.233.133|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 363820823 (347M) [application/x-netcdf]
Saving to: ‘HTAPv3_BC_0.1x0.1_2018.nc’

HTAPv3_BC_0.1x0.1_2018.nc                      100%[===================================================================================================>] 346.97M  9.75MB/s    in 43s     

2023-02-08 09:05:46 (7.98 MB/s) - ‘HTAPv3_BC_0.1x0.1_2018.nc’ saved [363820823/363820823]

and it works fine.

kilicomu commented 1 year ago

Thanks for that @1Dandan - just started downloading v2022-12 again and it seems to be working!