This bugzilla was created in response to the aqm_forecast_conus failures on Nov 3, 2021 (J date 307). The input files from /gpfs/hps/nco/ops/com/aqm/prod/emission/202111/ (specifically FERT_12km_5x_time202111.ncf) had data inside that could not be read in correctly by the aqm_fcstcs process. The endless search and read caused the job to run out of memory. The TFLAG variable had many lines containing missing data (", _,) instead of a valid j date. In this case, ncdump -v TFLAG FERT_12km_5x_time202111.ncf at the time of the failure showed the missing values within the file. Please create a process that will QC the files within the jaqm_emission_ingest job that copies them from dcom on the 16th of each month. This will help ensure the data is valid and complete before the AQM model tries to use them starting on the 1st of the next month. This will give SPA and ARL time to fix the files before a failure can happen in operations. Thanks, Holly/SPA team
This is another emission related bug for AQMv6, but needs to be addressed by AQM v7
http://www2.spa.ncep.noaa.gov/bugzilla/show_bug.cgi?id=1292
This bugzilla was created in response to the aqm_forecast_conus failures on Nov 3, 2021 (J date 307). The input files from /gpfs/hps/nco/ops/com/aqm/prod/emission/202111/ (specifically FERT_12km_5x_time202111.ncf) had data inside that could not be read in correctly by the aqm_fcstcs process. The endless search and read caused the job to run out of memory. The TFLAG variable had many lines containing missing data (", _,) instead of a valid j date. In this case, ncdump -v TFLAG FERT_12km_5x_time202111.ncf at the time of the failure showed the missing values within the file. Please create a process that will QC the files within the jaqm_emission_ingest job that copies them from dcom on the 16th of each month. This will help ensure the data is valid and complete before the AQM model tries to use them starting on the 1st of the next month. This will give SPA and ARL time to fix the files before a failure can happen in operations. Thanks, Holly/SPA team