xylar
commented
5 years ago The issues with one-year analysis runs affect climatologies as well. These issues were first seen on cooley but also can be reproduced on my laptop:
mpasClimatologyOcean.log
``` Computing MPAS climatologies from files: mpaso.hist.am.timeSeriesStatsMonthly.0001-01-01.nc through mpaso.hist.am.timeSeriesStatsMonthly.0001-12-01.nc running: ncclimo -4 --clm_md=mth -a sdd -m mpaso -p bck -v timeMonthly_avg_dThreshMLD,timeMonthly_avg_activeTracers_temperature,timeMonthly_avg_activeTracers_salinity,timeMonthly_avg_pressureAdjustedSSH,timeMonthly_avg_layerThickness,timeMonthly_avg_potentialDensity,timeMonthly_avg_velocityZonal,timeMonthly_avg_velocityMeridional,timeMonthly_avg_landIceFreshwaterFlux,timeMonthly_avg_meridionalHeatTransportLat,timeMonthly_avg_meridionalHeatTransportLatZ,timeMonthly_avg_normalVelocity,timeMonthly_avg_vertVelocityTop,timeMonthly_avg_normalGMBolusVelocity,timeMonthly_avg_vertGMBolusVelocityTop --seasons=JFM,JAS,ANN -s 0001 -e 0001 -i /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/source_symlinks -o /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI Climatology operations invoked with command: /home/xylar/miniconda3/envs/mpas_analysis_py3.7/bin/ncclimo -4 --clm_md=mth -a sdd -m mpaso -p bck -v timeMonthly_avg_dThreshMLD,timeMonthly_avg_activeTracers_temperature,timeMonthly_avg_activeTracers_salinity,timeMonthly_avg_pressureAdjustedSSH,timeMonthly_avg_layerThickness,timeMonthly_avg_potentialDensity,timeMonthly_avg_velocityZonal,timeMonthly_avg_velocityMeridional,timeMonthly_avg_landIceFreshwaterFlux,timeMonthly_avg_meridionalHeatTransportLat,timeMonthly_avg_meridionalHeatTransportLatZ,timeMonthly_avg_normalVelocity,timeMonthly_avg_vertVelocityTop,timeMonthly_avg_normalGMBolusVelocity,timeMonthly_avg_vertGMBolusVelocityTop --seasons=JFM,JAS,ANN -s 0001 -e 0001 -i /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/source_symlinks -o /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI Started climatology splitting at Mon Mar 4 11:29:21 MST 2019 Running climatology script ncclimo from directory /home/xylar/miniconda3/envs/mpas_analysis_py3.7/bin NCO binaries version 4.7.8 from directory /home/xylar/miniconda3/envs/mpas_analysis_py3.7/bin Parallelism mode = Background Producing standard climatology from raw input files in directory /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/source_symlinks Output files to directory /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI Climatology from 1 years of contiguous raw data touching 1 calendar years from YYYYMM = 000101 to 000112 Input filenames will be constructed with CESM'ish or E3SM/ACME'ish conventions Each input file assumed to contain mean of one month Winter statistics based on seasonally discontiguous December (sdd-mode): DJF sequences comprise three months from the same calendar year Annotation for CF climatology attribute and climatology_bounds variable will be performed This climatology will not be regridded Generating climatology... Climatological monthly mean for month 1 ... Climatological monthly mean for month 2 ... Climatological monthly mean for month 3 ... Climatological monthly mean for month 4 ... Climatological monthly mean for month 5 ... Climatological monthly mean for month 6 ... Climatological monthly mean for month 7 ... Climatological monthly mean for month 8 ... Climatological monthly mean for month 9 ... Climatological monthly mean for month 10 ... Climatological monthly mean for month 11 ... Climatological monthly mean for month 12 ... ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_07_000107_000107_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_06_000106_000106_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_01_000101_000101_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ncclimo: ERROR monthly climo cmd_clm[1] failed. Debug this: ncra --cb -O --no_tmp_fl --fl_fmt=netcdf4 -v timeMonthly_avg_dThreshMLD,timeMonthly_avg_activeTracers_temperature,timeMonthly_avg_activeTracers_salinity,timeMonthly_avg_pressureAdjustedSSH,timeMonthly_avg_layerThickness,timeMonthly_avg_potentialDensity,timeMonthly_avg_velocityZonal,timeMonthly_avg_velocityMeridional,timeMonthly_avg_landIceFreshwaterFlux,timeMonthly_avg_meridionalHeatTransportLat,timeMonthly_avg_meridionalHeatTransportLatZ,timeMonthly_avg_normalVelocity,timeMonthly_avg_vertVelocityTop,timeMonthly_avg_normalGMBolusVelocity,timeMonthly_avg_vertGMBolusVelocityTop --hdr_pad=10000 --gaa climo_script=ncclimo --gaa climo_command="'/home/xylar/miniconda3/envs/mpas_analysis_py3.7/bin/ncclimo -4 --clm_md=mth -a sdd -m mpaso -p bck -v timeMonthly_avg_dThreshMLD,timeMonthly_avg_activeTracers_temperature,timeMonthly_avg_activeTracers_salinity,timeMonthly_avg_pressureAdjustedSSH,timeMonthly_avg_layerThickness,timeMonthly_avg_potentialDensity,timeMonthly_avg_velocityZonal,timeMonthly_avg_velocityMeridional,timeMonthly_avg_landIceFreshwaterFlux,timeMonthly_avg_meridionalHeatTransportLat,timeMonthly_avg_meridionalHeatTransportLatZ,timeMonthly_avg_normalVelocity,timeMonthly_avg_vertVelocityTop,timeMonthly_avg_normalGMBolusVelocity,timeMonthly_avg_vertGMBolusVelocityTop --seasons=JFM,JAS,ANN -s 0001 -e 0001 -i /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/source_symlinks -o /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI'" --gaa climo_hostname=eleven --gaa climo_version=4.7.8 --gaa yrs_averaged=0001 -p /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/source_symlinks mpaso.hist.am.timeSeriesStatsMonthly.0001-01-01.nc /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_01_000101_000101_climo.nc ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_04_000104_000104_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_11_000111_000111_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_02_000102_000102_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_10_000110_000110_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_03_000103_000103_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_09_000109_000109_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_08_000108_000108_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_12_000112_000112_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) ERROR: nco__create() unable to create file "/media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI/mpaso_05_000105_000105_climo.nc" nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE) HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140417219843200: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() HDF5-DIAG: Error detected in HDF5 (1.10.4) nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140715765216384: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer HDF5-DIAG: Error detected in HDF5 (1.10.4) minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" thread 139713853748352: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" thread 140615230858368: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140128442057856: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 139674646692992: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed thread 140699057939584nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() : nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140361794962560: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140579133731968: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140054790567040: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140425602651264: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140005091300480: #000: H5F.c line 444 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1364 in H5F__create(): unable to open file major: File accessibilty minor: Unable to open file #002: H5Fint.c line 1615 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #003: H5FD.c line 1640 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #004: H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco__create() nco_err_exit(): ERROR Error code is 13. Translation into English with nc_strerror(13) is "Permission denied" analysis task mpasClimatologyOcean failed during run Traceback (most recent call last): File "/home/xylar/code/mpas-work/analysis/fix_one_year_analysis/mpas_analysis/shared/analysis_task.py", line 320, in run self.run_task() File "/home/xylar/code/mpas-work/analysis/fix_one_year_analysis/mpas_analysis/shared/climatology/mpas_climatology_task.py", line 265, in run_task outDirectory=climatologyDirectory) File "/home/xylar/code/mpas-work/analysis/fix_one_year_analysis/mpas_analysis/shared/climatology/mpas_climatology_task.py", line 450, in _compute_climatologies_with_ncclimo ' '.join(args)) subprocess.CalledProcessError: Command 'ncclimo -4 --clm_md=mth -a sdd -m mpaso -p bck -v timeMonthly_avg_dThreshMLD,timeMonthly_avg_activeTracers_temperature,timeMonthly_avg_activeTracers_salinity,timeMonthly_avg_pressureAdjustedSSH,timeMonthly_avg_layerThickness,timeMonthly_avg_potentialDensity,timeMonthly_avg_velocityZonal,timeMonthly_avg_velocityMeridional,timeMonthly_avg_landIceFreshwaterFlux,timeMonthly_avg_meridionalHeatTransportLat,timeMonthly_avg_meridionalHeatTransportLatZ,timeMonthly_avg_normalVelocity,timeMonthly_avg_vertVelocityTop,timeMonthly_avg_normalGMBolusVelocity,timeMonthly_avg_vertGMBolusVelocityTop --seasons=JFM,JAS,ANN -s 0001 -e 0001 -i /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/source_symlinks -o /media/xylar/bbyates/analysis/output/GMPAS-QU240wLI/fix_one_year_analysis/clim/mpas/unmasked_oQU240wLI' returned non-zero exit status 1. ```
There is an issue with
timeSeriesMOCreading from and writing to the same file.Various tasks (
indexNino34andtimeSeries*Anomaly) that do a running mean fail during the run phase whereas they should fail during setup_and_check()