search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

talend soop_xbt_harvest is failing #167

Closed danfruehauf closed 8 years ago

danfruehauf commented 8 years ago

Not sure what should happen with this harvester after soopxbt* were migrated to per file processing.

Error is:

Iterating over catalog at 'https://thredds.aodn.org.au/thredds/catalog/IMOS/SOOP/SOOP-XBT/catalog.xml'
Excluding any items matching these patterns: [.*aggregated_datasets.*]
 catalog valid = true
  dataset REALTIME valid = true
  dataset DELAYED valid = true
  dataset SOOP-XBT valid = true
Exception in component iThreddsIterator_2
java.lang.NullPointerException
        at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
        at java.util.regex.Matcher.reset(Matcher.java:308)
        at java.util.regex.Matcher.<init>(Matcher.java:228)
        at java.util.regex.Pattern.matcher(Pattern.java:1088)
        at netcdfharvester.indexfiles_1_0.IndexFiles$1.datasetMatches(IndexFiles.java:1437)
        at netcdfharvester.indexfiles_1_0.IndexFiles$1.shouldExclude(IndexFiles.java:1410)
        at netcdfharvester.indexfiles_1_0.IndexFiles$1.findNext(IndexFiles.java:1378)
        at netcdfharvester.indexfiles_1_0.IndexFiles$1.<init>(IndexFiles.java:1345)
        at netcdfharvester.indexfiles_1_0.IndexFiles.iThreddsIterator_2Process(IndexFiles.java:1296)
        at netcdfharvester.indexfiles_1_0.IndexFiles.tRunJob_1Process(IndexFiles.java:592)
        at netcdfharvester.indexfiles_1_0.IndexFiles.runJobInTOS(IndexFiles.java:4790)
        at netcdfharvester.indexfiles_1_0.IndexFiles.runJob(IndexFiles.java:4653)
        at netcdfharvester.indexandharvestattributes_1_0.IndexAndHarvestAttributes.tRunJob_1Process(IndexAndHarvestAttributes.java:497)
        at netcdfharvester.indexandharvestattributes_1_0.IndexAndHarvestAttributes.runJobInTOS(IndexAndHarvestAttributes.java:933)
        at netcdfharvester.indexandharvestattributes_1_0.IndexAndHarvestAttributes.runJob(IndexAndHarvestAttributes.java:768)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.tRunJob_1Process(SoopXbtHarvester.java:1218)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.runJobInTOS(SoopXbtHarvester.java:3205)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.main(SoopXbtHarvester.java:2989)
Exception in component tRunJob_1
java.lang.RuntimeException: Child job running failed
        at netcdfharvester.indexandharvestattributes_1_0.IndexAndHarvestAttributes.tRunJob_1Process(IndexAndHarvestAttributes.java:508)
        at netcdfharvester.indexandharvestattributes_1_0.IndexAndHarvestAttributes.runJobInTOS(IndexAndHarvestAttributes.java:933)
        at netcdfharvester.indexandharvestattributes_1_0.IndexAndHarvestAttributes.runJob(IndexAndHarvestAttributes.java:768)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.tRunJob_1Process(SoopXbtHarvester.java:1218)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.runJobInTOS(SoopXbtHarvester.java:3205)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.main(SoopXbtHarvester.java:2989)
Exception in component tRunJob_1
java.lang.RuntimeException: Child job running failed
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.tRunJob_1Process(SoopXbtHarvester.java:1229)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.runJobInTOS(SoopXbtHarvester.java:3205)
        at netcdfharvester.soopxbtharvester_0_1.SoopXbtHarvester.main(SoopXbtHarvester.java:2989)
finish;2016-01-05 18:33:30+11:00 0 minutes
danfruehauf commented 8 years ago

This is a friendly reminder that if nothing is going to be done about - the harvester will stop being monitored on nagios.

danfruehauf commented 8 years ago

So, what should we do with that? Disable the harvester? Stop monitoring it?

Clearly this is because aggregated_datasets were not uploaded to S3. I don't think it is planned to be uploaded either.