Syncing gcgrid S3 bucket with Dalhousie's FTP server?

geoschem / geos-chem-cloud

Run GEOS-Chem easily on AWS cloud

http://cloud.geos-chem.org

MIT License

39 stars 9 forks source link

Syncing gcgrid S3 bucket with Dalhousie's FTP server? #19

Closed JiaweiZhuang closed 1 year ago

JiaweiZhuang commented 5 years ago

Maggie Marvin at the University of Edinburgh asked whether the data on Dalhousie's FTP can be also uploaded to S3.

My questions are:

Which part of Dalhousie's data are not on S3 right now?

My reference is http://wiki.seas.harvard.edu/geos-chem/index.php/Downloading_GEOS-Chem_source_code_and_data#Dalhousie_data_directory_archive

Looks like most of them are metfields and should largely duplicate s3://gcgrid/ ?

Further, custom nested fields can be cropped from the global data: s3://gcgrid/GEOS_0.25x0.3125/GEOS_FP/

How large are those data? If it is small we can probably just upload them. If not then it involves talking to Amazon again.
Who is managing Dalhousie's FTP server currently? Is it possible to coordinate?

yantosca commented 5 years ago

Dal is storing the native-resolution GEOS-FP (0.25 x 0.3125) and MERRA-2 data (0.5 x 0.625) on the ComputeCanada server. I believe this is too large to store on Randall's server rain.ucis.dal.ca.

Jun Meng would be the best one to talk to re: Dal server support.

MaggieMarvin commented 5 years ago

I'm happy to use s3://gcgrid/GEOS_0.25x0.3125/GEOS_FP/ and crop down to my domain (SE Asia), but I'm intending to run a nested simulation at 0.25x0.3125 resolution for 2017-present, and I was not able to find met fields for this resolution and timeframe on S3. However, it looks like met fields for these years might be available for some of the nested domains from the Dalhousie server at rain.ucis.dal.ca?

JiaweiZhuang commented 5 years ago

@MargaretMarvin as a temporary solution you can transfer those data from rain.ucis.dal.ca to EC2, and even upload them to your own S3 bucket for later use. Transferring data into AWS is free. The storage cost on S3 is $23/TB/month and I don't think nested metfields would exceed 1 TB.

FeiYao-Edinburgh commented 5 years ago

I am interested in running a nested simulation at 0.25x0.3125 over China. I found that s3://gcgrid/GEOS_0.25x0.3125_CH/GEOS_FP/ only contains 2013 and 2016, whereas ftp://rain.ucis.dal.ca/ctm/GEOS_0.25x0.3125_CH.d/GEOS_FP/ contains met data from 2012 to now. Yes, it is not hard to transfer those data to EC2 or our own S3, but would be great if Amazon agree with you to upload those missing data.

yantosca commented 1 year ago

Closing this issue since the Dalhousie server is no longer in use.