search

aodn / harvesters

Harvesters
GNU General Public License v3.0
0 stars 0 forks source link

SOOP_AUSCPR harvester needs to handle null dates #524

Closed mhidas closed 6 years ago

mhidas commented 7 years ago

This harvester has been failing for weeks due to a "null" in a date field:

Jun 15, 2017 6:00:46 PM org.geotools.data.wfs.v1_1_0.WFS_1_1_0_DataStore adaptQueryForSupportedCrs
WARNING: Query does not provide a CRS, using default: Query:
   feature type: imos:AUSCPR_PHYTO
   filter: Filter.INCLUDE
   [properties:  ALL ]
java.lang.NullPointerException
        at routines.system.FastDateParser$DateParser.parse(FastDateParser.java:127)
        at java.text.DateFormat.parse(DateFormat.java:355)
        at routines.TalendDate.parseDate(TalendDate.java:763)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.sWfsInput_1_GIProcess(phyto_harvest.java:3681)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.iPostgresqlDbUpdate_1Process(phyto_harvest.java:930)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.tPostgresqlConnection_1Process(phyto_harvest.java:711)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.iIncludeSdiLibraries_1Process(phyto_harvest.java:593)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.runJobInTOS(phyto_harvest.java:4319)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.runJob(phyto_harvest.java:4150)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.tRunJob_4Process(SOOP_AUSCPR_harvester.java:1430)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.runJobInTOS(SOOP_AUSCPR_harvester.java:3430)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.main(SOOP_AUSCPR_harvester.java:3197)
Exception in component tMap_1
java.lang.RuntimeException: java.text.ParseException: Unparseable date: "null"
        at routines.TalendDate.parseDate(TalendDate.java:777)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.sWfsInput_1_GIProcess(phyto_harvest.java:3681)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.iPostgresqlDbUpdate_1Process(phyto_harvest.java:930)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.tPostgresqlConnection_1Process(phyto_harvest.java:711)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.iIncludeSdiLibraries_1Process(phyto_harvest.java:593)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.runJobInTOS(phyto_harvest.java:4319)
        at soop_auscpr.phyto_harvest_0_1.phyto_harvest.runJob(phyto_harvest.java:4150)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.tRunJob_4Process(SOOP_AUSCPR_harvester.java:1430)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.runJobInTOS(SOOP_AUSCPR_harvester.java:3430)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.main(SOOP_AUSCPR_harvester.java:3197)
Caused by: java.text.ParseException: Unparseable date: "null"
        at java.text.DateFormat.parse(DateFormat.java:357)
        at routines.TalendDate.parseDate(TalendDate.java:763)
        ... 9 more
Exception in component tRunJob_4
java.lang.RuntimeException: Child job running failed
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.tRunJob_4Process(SOOP_AUSCPR_harvester.java:1441)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.runJobInTOS(SOOP_AUSCPR_harvester.java:3430)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.main(SOOP_AUSCPR_harvester.java:3197)
mhidas commented 6 years ago

This is causing the harvester to fail again (still?). I've asked CMAR if those dates are allowed to be null. If not then this change won't be needed.

mhidas commented 6 years ago

CMAR have confirmed that we do need to handle those nulls. Done in #597