search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

soop_auscpr null pointer exception #425

Closed jonescc closed 5 years ago

jonescc commented 5 years ago

While testing this harvester for the upgrade, I notice its skipping the zoop harvest step due to a null pointer exception:

Exception in component tMap_1
java.lang.NullPointerException
        at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1380)
        at java.text.DateFormat.parse(DateFormat.java:355)
        at routines.TalendDate.parseDate(TalendDate.java:763)
        at soop_auscpr.zoop_harvest_0_1.zoop_harvest.sWfsInput_1_GIProcess(zoop_harvest.java:3871)
        at soop_auscpr.zoop_harvest_0_1.zoop_harvest.iPostgresqlDbUpdate_1Process(zoop_harvest.java:935)
        at soop_auscpr.zoop_harvest_0_1.zoop_harvest.tPostgresqlConnection_1Process(zoop_harvest.java:711)
        at soop_auscpr.zoop_harvest_0_1.zoop_harvest.iIncludeSdiLibraries_1Process(zoop_harvest.java:593)
        at soop_auscpr.zoop_harvest_0_1.zoop_harvest.runJobInTOS(zoop_harvest.java:4531)
        at soop_auscpr.zoop_harvest_0_1.zoop_harvest.runJob(zoop_harvest.java:4362)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.tRunJob_5Process(SOOP_AUSCPR_harvester.java:1684)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.runJobInTOS(SOOP_AUSCPR_harvester.java:3713)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.main(SOOP_AUSCPR_harvester.java:3455)
Exception in component tRunJob_5
java.lang.RuntimeException: Child job running failed
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.tRunJob_5Process(SOOP_AUSCPR_harvester.java:1695)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.runJobInTOS(SOOP_AUSCPR_harvester.java:3713)
        at soop_auscpr.soop_auscpr_harvester_0_1.SOOP_AUSCPR_harvester.main(SOOP_AUSCPR_harvester.java:3455)
mhidas commented 5 years ago

This is due to a change in the zoop table columns at the CSIRO end. The updated table can be downloaded as CSV and then harvested by the new PLANKTON_PRODUCTS harvester. This currently needs to be done manually.

Possible solutions:

  1. Deactivate the zoop_harvest subjob -- this is the easiest, but it means we will no longer be getting a daily harvest of the latest cpr zoop data.
  2. As above, plus set up a cron job to download this table to CSV and harvest it into PLANKTON_PRODUCTS.
  3. Update the harvester and the soop_auscpr db schema to the new version -- this is a bit of work and probably a waste of time because the geoserver layer based on that table is no longer used in the portal (it has been made obsolete by the new layers from plankton_products).
mhidas commented 5 years ago

So in the short term I think we should do 1. above, possibly followed by 2. at some point.

mhidas commented 5 years ago

Ok, 1. above was implemented in https://github.com/aodn/harvesters/pull/737 and the harvester is no longer failing.

mhidas commented 5 years ago

The second part of the solution I proposed above can be dealt with later (https://github.com/aodn/PO-Backlog/issues/1), so we can close this.