search

Unidata / awips2

Weather forecasting display and analysis package developed by NWS/Raytheon, released as open source software by Unidata.
http://unidata.github.io/awips2/
Other
174 stars 67 forks source link

ERROR: duplicate key value violates unique constraint "uk_grid_datauri_fields" #165

Closed mjames-upc closed 7 years ago

mjames-upc commented 7 years ago

2017-06-13 20:57:20 UTC 57479 metadata : 128.117.149.75 : ERROR: duplicate key value violates unique constraint "uk_grid_datauri_fields" 2017-06-13 20:57:20 UTC 57479 metadata : 128.117.149.75 : DETAIL: Key (reftime, forecasttime, info_id, rangestart, rangeend)=(2017-06-13 12:00:00, 3600, 2383, 2017-06-13 12:00:00, 2017-06-13 13:00:00) already exists. 2017-06-13 20:57:20 UTC 57479 metadata : 128.117.149.75 : STATEMENT: insert into grid (forecastTime, refTime, utilityFlags, rangeEnd, rangeStart, insertTime, info_id, id) values ($1, $2, $3, $4, $5, $6, $7, $8) 2017-06-13 20:57:20 UTC 57479 metadata : 128.117.149.75 : ERROR: current transaction is aborted, commands ignored until end of transaction block

mjames-upc commented 7 years ago

grep "already exists" postgresql-Friday.log| cut -c 70-|sort|uniq|grep info_id| cut -d "," -f 7| sort | uniq 2383, 3108, 3120, 3246, 3627, 3629, 4011, 4012, 4013, 4014, 4015, 4016, 4103, 4118, 4380, 4381, 4382, 4385, 4388, 4392, 4393, 4394, 4396, 4400, 4948, 4949, 4950, 5064, 5133, 5135, 5477

 select * from grid_info where id in (2383, 3108, 3120, 3246, 3627, 3629, 4011, 4012, 4013, 4014, 4015, 4016, 4103, 4118, 4380, 4381, 4382, 4385, 4388, 4392, 4393, 4394, 4396, 4400, 4948, 4949, 4950, 5064, 5133, 5135, 5477);
  id  |   datasetid    | ensembleid | secondaryid | level_id | location_id | parameter_abbreviation 
------+----------------+------------+-------------+----------+-------------+------------------------
 2383 | QPE-KRF        |            |             |       37 |         277 | QPE01-KRF1hr
 3108 | MPE-Mosaic-TAR |            |             |       37 |         137 | TP1hr
 3120 | QPE-XNAV-TAR   |            |             |       37 |         137 | TP6hr
 3246 | MPE-Local-STR  |            |             |       37 |         307 | TP1hr
 3627 | QPE-TIR        |            |             |       37 |         326 | QPE01-TIR1hr
 3629 | QPE-FWR        |            |             |       37 |         234 | QPE01-FWR1hr
 4011 | QPE-TUA        |            |             |       37 |         235 | QPE01-TUA1hr
 4012 | MPE-Mosaic-TIR |            |             |       37 |         326 | TP1hr
 4013 | MPE-Local-TIR  |            |             |       37 |         326 | TP1hr
 4014 | QPE-MSR        |            |             |       37 |         182 | QPE01-MSR1hr
 4015 | MPE-Local-TAR  |            |             |       37 |         137 | TP1hr
 4016 | QPE-ORN        |            |             |       37 |         181 | QPE01-ORN1hr
 4103 | QPE-Manual-KRF |            |             |       37 |         277 | QPE24-KRF24hr
 4118 | QPE-Auto-TUA   |            |             |       37 |         235 | TP1hr
 4380 | QPE-XNAV-TIR   |            |             |       37 |         326 | TP6hr
 4381 | MPE-Local-RHA  |            |             |       37 |          86 | TP1hr
 4382 | MPE-Mosaic-RHA |            |             |       37 |          86 | TP1hr
 4385 | QPE-RFC-PTR    |            |             |       37 |           8 | TP6hr
 4388 | QPE-RFC-PTR    |            |             |       37 |           8 | TP24hr
 4392 | QPE-XNAV-FWR   |            |             |       37 |         234 | TP6hr
 4393 | QPE-XNAV-SJU   |            |             |       37 |         312 | TP6hr
 4394 | QPE-XNAV-ALR   |            |             |       37 |         383 | TP6hr
 4396 | QPE-ALR        |            |             |       37 |         383 | QPE01-ALR1hr
 4400 | QPE-RFC-STR    |            |             |       37 |         307 | TP232hr
 4948 | MPE-Mosaic-ALR |            |             |       37 |         383 | TP1hr
 4949 | MPE-Mosaic-SJU |            |             |       37 |         312 | TP1hr
 4950 | MPE-Local-ALR  |            |             |       37 |         383 | TP1hr
 5064 | QPE-RFC-STR    |            |             |       37 |         307 | TP6hr
 5133 | MPE-Mosaic-ORN |            |             |       37 |         181 | TP1hr
 5135 | MPE-Local-SJU  |            |             |       37 |         312 | TP1hr
 5477 | URMA25         |            |             |       37 |          81 | TP6hr
mjames-upc commented 7 years ago

As these are all TP/QPE grid parameters, created post-processing, suspect it's an issue with OverwriteGribPostProcessor:

<!-- Post processor definitions for models which disseminate updated grids 
        containing identical metadata. This data is primarily from RFCs. The overwrite 
        post processor is used to prevent updated data from being discarded as duplicates. -->
    <postProcessedModel id="Overwrite">
        <modelName>HPCqpfNDFD|RFCqpf|HPCqpf|MPE-.*|QPE-.*|URMA25</modelName>
        <processorName>OverwriteGribPostProcessor</processorName>
    </postProcessedModel>

but also look into com.raytheon.edex.plugin.grib.decoderpostprocessors.precipitation.PrecipAccumPostProcessor

mjames-upc commented 7 years ago 
public class OverwriteGribPostProcessor extends DecoderPostProcessor {

    @Override
    public GridRecord[] process(GridRecord record) throws GribException {
        record.setOverwriteAllowed(true);
        return new GridRecord[] { record };
    }
}