search

ocean-transport / pleiades_llc_recipes

Examples of analysis of MITgcm LLC simulations of NASA Pleiades supercomputer
MIT License
8 stars 2 forks source link

Transfer subdomains from LLC4320 #11

Open abodner opened 1 year ago

abodner commented 1 year ago

Hi team, Thanks so much for including me on your efforts to transfer the data. Here is a spreadsheet I've made to help keep track of all the different regions I'm after (with 10 remaining to be transferred). I'll be adding the appropriate indices to each of the lat/lon coordinates as well.

abodner commented 1 year ago

And here is a repository with the code I am using to transfer and preprocess the llc4320 data: https://github.com/abodner/llc4320_preprocess

cspencerjones commented 1 year ago

Thanks @abodner . It seems as though we should first try to cut out and transfer a small amount of data for one of the regions you haven't tried yet. We will eventually want to transfer hourly data and perform benchmarking on that, but I think it makes sense first to transfer one region of data with one slice every 24hours, so that you can identify any problems with the workflow before we move forward.

Do you have a preference about which one to start with?Then @rsaim can try cutting it out, let us know how big it is and we can work on moving it to OSN.

abodner commented 1 year ago

Thanks @cspencerjones, that sounds like a good plan. Let's start with "South Atlantic, middle". I colored it in orange to make it easier to find.

If you would like to transfer the hourly data that might be helpful for me as well since I was hoping to test the sensitivity of downsampling in my NN results.

abodner commented 1 year ago

Also, it seems I will have my NASA account approved within one week or so. I am happy to contribute to the effort once I am all set up.

cspencerjones commented 1 year ago

The South Atlantic region Abigail is asking for has indices isel(face=1, i=(slice(600,1350), j=slice(1950,2850),k=slice(0,45)) (I added a small margin around the outside to be safe). A single timeslice with all five variables (T,S,U,V,W) should be ~0.5GB.