search

gbif / pipelines

Pipelines for data processing (GBIF and LivingAtlases)
Apache License 2.0
40 stars 28 forks source link

Fix for 1037 #1052

Closed vjrj closed 3 weeks ago

vjrj commented 3 months ago

Simple fix for #1037 similar to #1004. With this patch the event interpretation also work with --local:

INFO  [2024-03-28 16:52:17,235+0100] [Executor task launch worker for task 528] au.org.ala.pipelines.transforms.LocationTransform: Close stateProvinceKvStore
INFO  [2024-03-28 16:52:17,235+0100] [Executor task launch worker for task 525] au.org.ala.pipelines.transforms.LocationTransform: Close biomeKvStore
INFO  [2024-03-28 16:52:17,235+0100] [Executor task launch worker for task 528] au.org.ala.pipelines.transforms.LocationTransform: Close biomeKvStore
INFO  [2024-03-28 16:52:17,235+0100] [Executor task launch worker for task 530] au.org.ala.pipelines.transforms.LocationTransform: Close stateProvinceKvStore
INFO  [2024-03-28 16:52:17,236+0100] [Executor task launch worker for task 530] au.org.ala.pipelines.transforms.LocationTransform: Close biomeKvStore
INFO  [2024-03-28 16:52:23,442+0100] [main] au.org.ala.pipelines.beam.ALAVerbatimToEventPipeline: Save metrics into the file and set files owner
INFO  [2024-03-28 16:52:23,442+0100] [main] org.gbif.pipelines.common.beam.metrics.MetricsHandler: Trying to write pipeline's metadata to a file - hdfs://gbif-es-pipelines-2022-1:9000/pipelines-data/dr950/1/interpretation-metrics.yml
INFO  [2024-03-28 16:52:23,443+0100] [main] org.gbif.pipelines.common.beam.metrics.MetricsHandler: Added pipeline metadata - locationRecordsCountAttempted: 12, uniqueIdsCountAttempted: 6, temporalRecordsCountAttempted: 6, metadataRecordsCountAttempted: 1, identifierRecordsCountAttempted: 6, 
INFO  [2024-03-28 16:52:23,588+0100] [main] org.gbif.pipelines.common.beam.metrics.MetricsHandler: Metadata was written to a file - hdfs://gbif-es-pipelines-2022-1:9000/pipelines-data/dr950/1/interpretation-metrics.yml
INFO  [2024-03-28 16:52:23,588+0100] [main] au.org.ala.pipelines.beam.ALAVerbatimToEventPipeline: Deleting beam temporal folders
INFO  [2024-03-28 16:52:23,601+0100] [main] au.org.ala.pipelines.beam.ALAVerbatimToEventPipeline: Pipeline has been finished
28-Mar 16:52:24 [LA-PIPELINES] [dr950] [INFO] Thu Mar 28 16:52:24 CET 2024
28-Mar 16:52:24 [LA-PIPELINES] [dr950] [INFO] END Interpretation of dr950 in [local], took 0 minutes and 22 seconds.