MPAS-Dev / MPAS-Analysis

Provides analysis for the MPAS components of E3SM
Other
52 stars 49 forks source link

Simplify ncremap output? #142

Closed pwolfram closed 6 years ago

pwolfram commented 7 years ago

Currently, ncremap produces a ton of output each operation, e.g.,

Started processing at Thu Mar 16 14:38:10 MDT 2017.
Running remap script ncremap from directory /net/scratch3/pwolfram/miniconda2/envs/test_analysis/bin
NCO binaries version "4.6.4" from directory /net/scratch3/pwolfram/miniconda2/envs/test_analysis/bin
Input files in or relative to directory /users/pwolfram/MPAS-Analysis
Intermediate/temporary files written to directory /lustre/scratch3/turquoise/pwolfram/tmp
Output files to directory /users/pwolfram/MPAS-Analysis/tmp/clim/mpas/regridded
Input data shaped in "mpas"-order, will permute with "ncpdq -a Time,nVertLevels,maxEdges,MaxEdges2,nCategories,ONE,nEdges,nCells"
Input assumed to be MPAS-O/I data: will apply renormalized regridding (--rnr=0.0)
Source and destination grids will both be read from supplied map-file
Input #00: /users/pwolfram/MPAS-Analysis/tmp/clim/mpas/iceThickness_EC60to30_FM_years0002-0003.nc
PDQ(in)  : /users/pwolfram/MPAS-Analysis/tmp/clim/mpas/iceThickness_EC60to30_FM_years0002-0003.nc
PDQ(out) : /lustre/scratch3/turquoise/pwolfram/tmp/ncremap_tmp_pdq.nc.pid28005
Map/Wgt  : /users/pwolfram/MPAS-Analysis/tmp/mapping/map_EC60to30_to_0.5x0.5degree_bilinear.nc
Regridded: /users/pwolfram/MPAS-Analysis/tmp/clim/mpas/regridded/iceThickness_EC60to30_to_0.5x0.5degree_FM_years0002-0003.nc
ncks: INFO nco_fl_open() reports current extended filetype = NC_FORMATX_NC3 does not equal previous extended filetype = NC_FORMATX_HDF5. This is expected when NCO is instructed to convert filetypes, i.e., to read from one type and write to another. And when NCO generates grids or templates (which are always netCDF3) when the input file is netCDF4. It is also expected when multi-file operators receive files known to be of different types. However, it could also indicate an unexpected change in input dataset type of which the user should be cognizant.
Cleaning-up intermediate files...
Quick plots of results from last regridded file:
ncview  /users/pwolfram/MPAS-Analysis/tmp/clim/mpas/regridded/iceThickness_EC60to30_to_0.5x0.5degree_FM_years0002-0003.nc &
panoply /users/pwolfram/MPAS-Analysis/tmp/clim/mpas/regridded/iceThickness_EC60to30_to_0.5x0.5degree_FM_years0002-0003.nc &
Elapsed time 0m0s

Would it be best to simplify this output and only return the complete set of output if an error occurs? I would argue that we don't need all this output for standard analysis runs.

xylar commented 7 years ago

@pwolfram, we've already had a lengthy discussion about this while you were away in proposal land ;-) : https://github.com/MPAS-Dev/MPAS-Analysis/pull/124#issuecomment-283455450

pwolfram commented 7 years ago

Ok, sounds good. Thanks @xylar. I think this is something we will want to deal with long-term but following that thread it doesn't appear to be a short-term issue. I recommend we leave this issue open for now.

xylar commented 7 years ago

I think we want to wait for a future version of nco that allows us to suppress some of this verbose output and live with it in the meantime. I'm concerned that relevant warnings that aren't full-blown errors would be missed if I bury the output entirely for error-free runs.

xylar commented 7 years ago

By all means leave this issue here until we get around to updating nco to an appropriate version and setting the needed flags.

xylar commented 6 years ago

Addressed by #224