search

NOAA-GFDL / MOM6-examples

Example configurations for MOM6 and SIS2
Other
87 stars 147 forks source link

rethink use of /nbhome for cloning mom6 #4

Closed nikizadehgfdl closed 9 years ago

nikizadehgfdl commented 9 years ago

/nbhome/$USER is limited to 10GB and can easily fill up and when that happens the analysis scripts will fail. I already went over quota on /nbhome because of the figures and the following items even though I do not have too many files there. I imagine this is going to be a problem for real model runners.

Can we try /ptmp instead?

jkrasting commented 9 years ago

I can provide evidence that I already have 25+ GB of CM4i analysis in /nbhome and I've hit my quota.

Dr. John Krasting Physical Scientist (NOAA Federal) NOAA/Geophysical Fluid Dynamics Laboratory Biogeochemistry, Ecosystems, and Climate Group Princeton University Forrestal Campus 201 Forrestal Road Princeton, NJ 08540

P. (609) 452-5359 F. (609) 987-5063

On Mon, Oct 27, 2014 at 6:41 PM, Niki Zadeh notifications@github.com wrote:

/nbhome/$USER is limited to 10GB and can easily fill up and when that happens the analysis scripts will fail. I already went over quota on /nbhome because of the figures and the following items even though I do not have too many files there. I imagine this is going to be a problem for real model runners.

-

Some analysis tool try to build (MIDAS) libraries in that space.

Some analysis scripts (TS_depth_integrals.py) save larger .nc files in analysis figure destination in addition to small .png.

Can we try /ptmp instead?

— Reply to this email directly or view it on GitHub https://github.com/CommerceGov/NOAA-GFDL-MOM6-examples/issues/4.

adcroft commented 9 years ago

Should we add a cleanup of the .nc files to the TS_depth_integrals.py script?

nikizadehgfdl commented 9 years ago

Yes. Also, one way to save space on /nbhome is to put mom6 clone (or anything not needed "permanently") in /ptmp and link it to /nbhome for any given experiment (I have tested that out of necessity since my /nbhome kept filling up).

adcroft commented 9 years ago

Is this still an issue?

adcroft commented 9 years ago

This also relates to #25 (+x permissions of gcp'd scripts).

The reason we need particular versions of the analysis scripts is they can/do/will evolve with the code and diag_tables. I agree, however, that it is very wasteful to be installing MOM6-examples for each run. Here's some break downs:

We can reduce the footprint by an order of magnitude by doing a sparse clone but this is a little involved. :-1:

Here's a different suggestion: IF we knew the commit id used in the remote repository (think tag) then instead of cloning/copying to the experiment we could clone to /nbhome/$user/mom/<id> so that there is only one copy per user for all their experiments using the same tag/source. Bonnie runs dozens of runs per tag so the above saves an order of magnitude. :+1:

I think the question then becomes how to get the commit id or tag back to GFDL from the remote platform. I believe you said that FRE does not know it at the frepp stage? Which is why you have to gcp the entire directory? If so, then here's an alternative way to get the tag, still using gcp but this time only transferring 41bytes: gcp <path_to_mom>/.git/HEAD gfdl:/<nbhome_path_to_expt>/HEAD and then use the contents of HEAD to check for the existence of /nbhome/$user/mom/dev/master/2015.09.21a/. If missing, clone MOM6-examples into it.

In the .frepp files, we have

set script_dir = ${out_dir}/mom6/tools/analysis

which would need to become

set mom_dir = `cat ${out_dir}/HEAD`
set script_dir = /nbhome/$user/mom/${mom_dir}/mom6/tools/analysis

:cake: (As in "piece of" - I can't find an emoji for :Genius!:)

nikizadehgfdl commented 9 years ago

Most of the space is used up by the analysis figures, mom6 clone takes only a minute part. This is really a workflow issue and not a MOM6 issue. It is more about finding a permanent home for the analysis figures. E.g., a 60 year run takes almost 10G of space (even after .nc files are removed) with mom6 clone being only 178M.

I think we can close this ticket.

/nbhome/Niki.Zadeh/ulm_201505_awg_v20150702_mom6sis2_2015.08.06b_ncrc2.intel15/CM4_c96L32_am4g5r2_2000_sis2_low_mixing3 % 

216M    atmos_0001_0005                                                                                                                
194M    atmos_0001_0010                                                                                                                
199M    atmos_0001_0015                                                                                                                
1.1G    atmos_0001_0020                                                                                                                
209M    atmos_0001_0025                                                                                                                
214M    atmos_0001_0030                                                                                                                
219M    atmos_0001_0035                                                                                                                
223M    atmos_0001_0040                                                                                                                
227M    atmos_0001_0045                                                                                                                
232M    atmos_0001_0050                                                                                                                
236M    atmos_0001_0055                                                                                                                
237M    atmos_0001_0060                                                                                                                
25M     atmos_0006_0010                                                                                                                
25M     atmos_0011_0015                                                                                                                
25M     atmos_0016_0020                                                                                                                
25M     atmos_0021_0025                                                                                                                
849M    atmos_0021_0040                                                                                                                
25M     atmos_0026_0030                                                                                                                
17M     atmos_0031_0035                                                                                                                
17M     atmos_0036_0040                                                                                                                
17M     atmos_0041_0045                                                                                                                
1.1G    atmos_0041_0060                                                                                                                
17M     atmos_0046_0050                                                                                                                
17M     atmos_0051_0055                                                                                                                
17M     atmos_0056_0060                                                                                                                
3.7M    atw_atmos_av_annual                                                                                                            
2.4G    atw_atmos_ts_monthly                                                                                                           
439M    cvdp                                                                                                                           
804K    db                                                                                                                             
532K    ice_0001_0005                                                                                                                  
528K    ice_0001_0010                                                                                                                  
528K    ice_0001_0015                                                                                                                  
524K    ice_0001_0040                                                                                                                  
524K    ice_0001_0045                                                                                                                  
524K    ice_0001_0050                                                                                                                  
524K    ice_0001_0055                                                                                                                  
524K    ice_0001_0060                                                                                                                  
528K    ice_0021_0040                                                                                                                  
364K    kap_land_0001_0005                                                                                                             
364K    kap_land_0001_0010                                                                                                             
368K    kap_land_0001_0015                                                                                                             
196M    kap_land_0001_0020                                                                                                             
380K    kap_land_0001_0025
376K    kap_land_0001_0030
380K    kap_land_0001_0035
380K    kap_land_0001_0040
388K    kap_land_0001_0045
388K    kap_land_0001_0050
388K    kap_land_0001_0055
388K    kap_land_0001_0060
196M    kap_land_0021_0040
196M    kap_land_0041_0060
2.3M    mdt_xml
178M    mom6
6.9M    ocean_0001-0005
6.8M    ocean_0001-0020
7.0M    ocean_0006-0010
6.5M    ocean_0011-0015
7.5M    ocean_0016-0020
7.5M    ocean_0021-0025
6.9M    ocean_0021-0040
7.5M    ocean_0026-0030
7.5M    ocean_0031-0035
7.5M    ocean_0036-0040
7.5M    ocean_0041-0045
6.9M    ocean_0041-0060
7.5M    ocean_0046-0050
7.5M    ocean_0051-0055
7.5M    ocean_0056-0060
1.2M    ocean_monthly_0001_0005
1.2M    ocean_monthly_0001_0020
1.2M    ocean_monthly_0006_0010
1.2M    ocean_monthly_0011_0015
1.2M    ocean_monthly_0016_0020
1.2M    ocean_monthly_0021_0025
1.2M    ocean_monthly_0021_0040
1.2M    ocean_monthly_0026_0030
12K     ocean_monthly_0031_0035
1.2M    ocean_monthly_0036_0040
1.2M    ocean_monthly_0041_0045
1.2M    ocean_monthly_0041_0060
1.2M    ocean_monthly_0046_0050
1.2M    ocean_monthly_0051_0055
1.2M    ocean_monthly_0056_0060
8.9M    refineDiag_ocean_annual
adcroft commented 9 years ago

Done