search

pernak18 / g-point-reduction

Jupyter Notebook evolution of RRTMGP g-point reduction (AKA k-distribution optimization) that started with Menno's [k-distribution-opt](https://github.com/MennoVeerman/k-distribution-opt) repo
0 stars 0 forks source link

Backup on NERSC HPSS #29

Open pernak18 opened 9 months ago

pernak18 commented 9 months ago

Using the Globus Web App, back up what i currently have on perlmutter scratch to the HPSS tape backup system to avoid any purging of our work

pernak18 commented 9 months ago

credentials: NERSC endpoints: NERSC Permutter to NERSC HPSS Paths: /pscratch/sd/p/pernak18/ and /home/p/pernak18/g-point-redux/

and the transfer was the entire RRTMGP directory, which contains:

cori_test_abs_val/ # where we left off when cori reached end-of-life eli_RRTMGP-profile-stats-plots/ # directory eli used to make forcing plots for the 80, 128, and 256 g-point k distributions garand_atmos/ # version controlled source directory (probably not necessary) quad_regress_abs_val/ # MY FINISHED QUADRATIC REGRESSION ABSOLUTE VALUE COST FUNCTION that ended at iteration 176 (80 g-points) reference_netCDF/ # netCDF files that can also be found in /global/cfs/projectdirs/e3sm/pernak18/reference_netCDF/ RRTMGP-profile-stats-plots/ # my plotting code repository, which is version controlled and likely not important

pernak18 commented 9 months ago

i attempted the transfer and got maybe 60% of the way through it before i got an error that was caused by the volume i tried to transfer. HPSS systems -- tape archives -- struggle with a large number of small files, and the recommendation is for everything to be tarred up.

i've tarred up what we need, but then i could not connect to the NERSC HPSS endpoint in the Globus web app. i was getting some authentication error but also could not try to re-login.

instead, i've stored the files in project space, where we have a higher quota than /home and no time constraints like on /pscratch:

(nersc-python) % pwd /global/cfs/projectdirs/e3sm/pernak18/baseline_g080 (nersc-python) % ls g080_iter176_quad_regress_abl_val_code_nc_pickle.tgz profz_statz_4xCO2.tgz profz_statz_plotz_PD_Garand.tgz RRTMGP_g128_g256_perlmutter.tgz profz_statz_4xCO2_corrected_perlmutter.tgz profz_statz_corrected_perlmutter.tgz profz_statz_update_perlmutter_PD_Garand.tgz

so that's all the code and output we used to get to a baseline answer with 80 g-points, plus the associated profile and statistics plots (plus the 128- and 256-point distributions for comparison, and the forcing scenario that started to go wacky quickly)

storing in project space is a workaround. eventually i'd like to push at least g080_iter176_quad_regress_abl_val_code_nc_pickle.tgz to tape backup