[BUG] [PRO] Metadata catalog empty after ingestion

eroan-marie commented 1 year ago

Environment:

Delivery tag: release/1.7.0-rc1
Platform: IVV Orange Cloud
Configuration: 3 masters / 22 workers

Test:

Name: TST_PRO_WF_S1L0
Traçability (requirements): RS-REQ-SRD-001

Current Behavior:

After copying sessions, the ingestion chain works just fine but the metadata catalog is empty.

Expected Behavior:

The metadata catalog should not be empty

Steps To Reproduce: Example: steps to reproduce the behavior:

In this environment
With the default config
Copy a session to the ESA folder on the mock
See error below in the metadata-extraction pod in the processing namespace and a kafka message produced by the ingestion worker

Test execution artefacts (i.e. logs, screenshots…)

{"header":{"header":{"type":"LOG","timestamp":"2022-11-21T13:47:57.842398Z","level":"ERROR","line":250,"file":"LogAccessor.java","thread":"KafkaConsumerDestination{consumerDestinationName=metadata2-part1.metadata-filter, partitions=1, dlqName=error-warning}.container-0-C-1"},"message":{"content":"org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.cloud.stream.function.FunctionConfiguration$FunctionToDestinationBinder$1@5dd8edbc]; nested exception is java.lang.NullPointerException: Name is null, failedMessage=GenericMessage [payload=byte[878], headers={deliveryAttempt=3, kafka_timestampType=CREATE_TIME, kafka_receivedTopic=metadata2-part1.metadata-filter, target-protocol=kafka, b3=4c7ae8b1656e7498-ecd2d5c11082db9d-0, nativeHeaders={b3=[4c7ae8b1656e7498-ecd2d5c11082db9d-0]}, kafka_offset=294, scst_nativeHeadersPresent=true, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@280638e8, kafka_receivedPartitionId=0, contentType=application/json, kafka_receivedTimestamp=1669038474837, kafka_groupId=metadata2-part1}]
    at org.springframework.integration.support.utils.IntegrationUtils.wrapInHandlingExceptionIfNecessary(IntegrationUtils.java:191)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:65)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
    at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:216)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:397)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:83)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:454)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:428)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0(RetryingMessageListenerAdapter.java:125)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:119)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:42)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2629)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2609)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException: Name is null
    at java.base/java.lang.Enum.valueOf(Enum.java:238)
    at esa.s1pdgs.cpoc.metadata.model.MissionId.valueOf(MissionId.java:3)
    at esa.s1pdgs.cpoc.metadata.extraction.service.ExtractionService.apply(ExtractionService.java:88)
    at esa.s1pdgs.cpoc.metadata.extraction.service.ExtractionService.apply(ExtractionService.java:57)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunctionAndEnrichResultIfNecessary(SimpleFunctionRegistry.java:897)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunction(SimpleFunctionRegistry.java:853)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:708)
    at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)
    at org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:84)
    at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionWrapper.apply(FunctionConfiguration.java:754)
    at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionToDestinationBinder$1.handleMessageInternal(FunctionConfiguration.java:586)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56)
    ... 32 more
"},"custom":{"logger_string":"org.springframework.integration.handler.LoggingHandler"}}

{
  "uid": "7074f449-2a5e-4b29-9290-88e7f8733abc",
  "creationDate": "2022-11-21T13:47:27.687Z",
  "missionId": "S1",
  "satelliteId": "",
  "keyObjectStorage": "S1A_20200120185900030888/DCS_01_S1A_20200120185900030888_ch1_DSDB_00023.raw",
  "storagePath": "s3://rs-session-files/S1A_20200120185900030888/DCS_01_S1A_20200120185900030888_ch1_DSDB_00023.raw",
  "productFamily": "EDRS_SESSION",
  "podName": "ingestion3-part1-ingestion-worker-v4-6698cbbbff-ms5zx",
  "allowedActions": [],
  "retryCounter": 0,
  "additionalFields": {
    "t0PdgsDate": "2022-01-19T17:12:01.000000Z"
  },
  "metadata": {
    "mode": null,
    "relativePath": "MTI_/S1A/S1A_20200120185900030888/DCS_01_S1A_20200120185900030888_ch1_DSDB_00023.raw",
    "productName": "S1A_20200120185900030888/DCS_01_S1A_20200120185900030888_ch1_DSDB_00023.raw"
  },
  "demandType": "NOMINAL",
  "debug": false,
  "timeliness": null,
  "productSizeByte": 314571600,
  "oqcFlag": "NOT_CHECKED",
  "stationName": "MTI_"
}

Whenever possible, first analysis of the root cause

Bug Generic Definition of Ready (DoR)

[X] The affect version in which the bug has been found is mentioned
[X] The context and environment of the bug is detailed
[X] The description of the bug is clear and unambiguous
[X] The procedure (steps) to reproduce the bug is clearly detailed
[X] The failed tests is linked to the bug : failed result % expected result
[X] The tested User Story / features is linked to the bug
[X] Logs are attached if available
[X] A data set attached if available
[X] Category label is link to the bug

Bug Generic Definition of Done (DoD)

[ ] the modification implemented (the solution to fix the bug) is described in the bug.
[ ] Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
[ ] Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
[ ] Code is compliant with coding rules (SONAR Report as evidence)
[ ] Acceptance criteria of the related User story are checked and Passed

eroan-marie commented 1 year ago

I encountered the same bug with a S3 session which makes me think that the issue is not related to the S1 session but to the ingestion worker

Woljtek commented 1 year ago

I checked the COPRS-ICD-ADST-001144532 - ICD Processing Message Format, there is no "Name" field, so I don't expect a metadata extraction for this field. @eroan-marie , could you provide the stream-parameters.properties of RS Core Metadata? @w-fsi , could you have a look to this issue?

eroan-marie commented 1 year ago

RS-CORE-METADATA stream-parameters.properties

# Application properties
app.*.spring.kafka.bootstrap-servers=kafka-cluster-kafka-bootstrap.infra.svc.cluster.local:9092

# Error Management
app.*.spring.cloud.stream.bindings.input.consumer.maxAttempts=3
app.*.spring.cloud.stream.kafka.bindings.input.consumer.enableDlq=true
app.*.spring.cloud.stream.kafka.bindings.input.consumer.dlqName=error-warning

# Prevent auto creation of kafka topics
#app.*.spring.cloud.stream.kafka.binder.autoAddPartitions=false
#app.*.spring.cloud.stream.kafka.binder.autoCreateTopics=false

app.metadata-filter.function.expression=true

app.metadata-extraction.main.banner-mode=off
app.metadata-extraction.management.endpoint.health.show-details=always
app.metadata-extraction.logging.config=/log/log4j2.yml
app.metadata-extraction.spring.cloud.stream.function.bindings.extractMetadata-in-0=input
app.metadata-extraction.spring.cloud.stream.function.bindings.extractMetadata-out-0=output
app.metadata-extraction.spring.cloud.stream.function.definition=extractMetadata
app.metadata-extraction.application.name=coprs-metadata-extraction

app.metadata-extraction.elasticsearch.host=elasticsearch-processing-es-http.database.svc.cluster.local
app.metadata-extraction.elasticsearch.port=9200
app.metadata-extraction.elasticsearch.connect-timeout-ms=2000
app.metadata-extraction.elasticsearch.socket-timeout-ms=10000
app.metadata-extraction.process.manifest-filenames.sen3=xfdumanifest.xml
app.metadata-extraction.process.manifest-filenames.isip=[PRODUCTNAME]_iif.xml
app.metadata-extraction.process.manifest-filenames.safe=manifest.safe
app.metadata-extraction.process.manifest-filenames.s2=[S2PRODUCTNAME].xml
app.metadata-extraction.process.hostname=${HOSTNAME}
app.metadata-extraction.process.num-obs-download-retries=10
app.metadata-extraction.process.sleep-between-obs-retries-millis=3000
app.metadata-extraction.worker.product-categories.auxiliary-files.pattern-config=^([0-9A-Za-z][0-9A-Za-z])([0-9A-Za-z_])(_(OPER|TEST))?_(AMH_ERRMAT|AMV_ERRMAT|AM__ERRMAT|AUX_CAL|AUX_ICE|AUX_INS|AUX_ITC|AUX_OBMEMC|AUX_PP1|AUX_PP2|AUX_POEORB|AUX_PREORB|AUX_RESORB|AUX_SCF|AUX_SCS|AUX_TEC|AUX_TRO|AUX_WAV|AUX_WND|MPL_ORBPRE|MPL_ORBRES|MPL_ORBSCT|MSK_EW_SLC|MSK__LAND_|MSK_OCEAN_|MSK_OVRPAS)_\\w{1,}\\.(XML|EOF|SAFE)(/.*)?$
app.metadata-extraction.worker.product-categories.auxiliary-files.local-directory=/data/local-catalog/auxiliary_files/
app.metadata-extraction.worker.product-categories.edrs-sessions.pattern-config=^([0-9aA-Z-z_]+)/(ch[0|_]?[1-2]/)?(DCS_[0-9]{2}_([a-zA-Z0-9_]*)_ch([12])_(DSDB|DSIB).*\\.(raw|aisp|xml))$
app.metadata-extraction.worker.product-categories.edrs-sessions.local-directory=/data/local-catalog/edrs_sessions/
app.metadata-extraction.worker.product-categories.edrs-sessions.path-pattern=^([0-9A-Za-z_]{4}/)?([0-9A-Za-z_]{2})([0-9A-Za-z_]{1})/([0-9A-Za-z_/]+)/(ch[0|_]?[1-2]/)?(DCS_[0-9]{2}_([a-zA-Z0-9_]*)_ch([12])_(DSDB|DSIB).*\\.(raw|aisp|xml))$
app.metadata-extraction.worker.product-categories.edrs-sessions.path-metadata-elements.missionId=2
app.metadata-extraction.worker.product-categories.edrs-sessions.path-metadata-elements.satelliteId=3
app.metadata-extraction.worker.product-categories.edrs-sessions.path-metadata-elements.sessionId=7
app.metadata-extraction.worker.product-categories.edrs-sessions.path-metadata-elements.channelId=8
app.metadata-extraction.worker.product-categories.plans-and-reports.pattern-config=^(S1[ABCD_]_OPER_REP_MP_MP__PDMC_|S1[AB_]OPER_REP_MP_MPPDMC|S1[ABCD]_OPER_MPL_SP.{4}_PDMC_|S1[ABCD_]_OPER_MPL_FS.{4}_PDMC_|S1[ABCD]_OPER_REP_PASS_[1-9]_.{4}_|S[12]__OPER_SRA_EDRS_[AC]_PDMC_|EDR_OPER_MPL_RQ[1-9]_O[AC]_|EDR_OPER_MPL_[LM]AS_O[AC]_|EDR_OPER_MPL_CR[1-9]_O[AC]_|EDR_OPER_MPL_SS[1-9]_O[AC]_|EDR_OPER_MPL_ER[1-9]_O[AC]_|EDR_OPER_SER_SR[1-9]_O[AC]_|S1[ABCD]_OPER_MPL_ORBOEM_|EDR_OPER_MPL_GOB_P[AC]_|EDR_OPER_MPL_GOB_R[AC]_|S1[ABCD]_OPER_REP__SUP___|S1[ABCD]_OPER_REP_STNACQ_.{4}_|S1[ABCD_]_OPER_REP_STNUNV_.{4}_|S[123][ABCD_]_OPER_SRA_BANSEG_PDMC_|S1[ABCD]_OPER_TLM__REQ_[A-O]_|S1[ABCD]_OPER_REP__SMPR__|S1[ABCD]_OPER_MPL__SSC___|S1[ABCD]_OPER_TLM__PSCAT_|S1[ABCD]_OPER_MPL_OCMSAR_|S1[ABCD]_OPER_REP__MACP__|S1[ABCD]_OPER_REP__MCSF__|S1[ABCD]_OPER_MPL__NPPF__|S1[ABCD]_OPER_MPL__NPIF__|S1[ABCD]_OPER_REP_NPIFCC_|S[123][ABCD_]_OPER_SRA_GSUNAV_PDMC_|S1[ABCD]_OPER_OBS_MIMG___|S1[ABCD]_OPER_AUX_RDB____MPC__|S1[ABCD]_OPER_MPL_SESDB[ABCD]_|S1[ABCD]_OPER_REP__CHF___|S1[AB]_OPER_REP__FCHF__|S1[AB]_OPER_AM[VH_]_FAILUR_MPC__|S1[AB]_OPER_AUX_QCSTDB_|S1[AB_]_OPER_REP_QC...._MPC__|S1[AB]OPER_REPSUP__|S1[AB]OPER_REPMACP_).*(\\.xml|\\.XML|\\.EOF|\\.TGZ|\\.zip|\\.ZIP)?$
app.metadata-extraction.worker.product-categories.plans-and-reports.local-directory=/data/local-catalog/plans_and_reports/
app.metadata-extraction.worker.product-categories.level-segments.pattern-config=^(S1|AS)(A|B)_(S[1-6]|RF|GP|HK|IW|EW|WV|N[1-6]|EN|IM|Z[1-6]|ZE|ZI|ZW)_(SLC|GRD|OCN|RAW)(F|H|M|_)_(0)(A|C|N|S|_)(SH|__|SV|HH|HV|VV|VH|DH|DV)_([0-9a-z]{15})_([0-9a-z]{15})_([0-9]{6})_([0-9a-z_]{6})\\w{1,}\\.(SAFE)(/.*)?$
app.metadata-extraction.worker.product-categories.level-segments.local-directory=/data/local-catalog/level_segments/
app.metadata-extraction.worker.product-categories.level-products.pattern-config=^(S1|AS)(A|B)_(S[1-6]|RF|IW|EW|WV|GP|HK|N[1-6]|EN|IM)_(SLC|GRD|OCN|RAW)(F|H|M|_)_(0|1|2)(A|C|N|S|_)(SH|SV|HH|HV|VV|VH|DH|DV)_([0-9a-z]{15})_([0-9a-z]{15})_([0-9]{6})_([0-9a-z_]{6})\\w{1,}\\.(SAFE)(/.*)?$
app.metadata-extraction.worker.product-categories.level-products.local-directory=/data/local-catalog/level_products/
app.metadata-extraction.worker.product-categories.s2-aux.pattern-config=^(S2)(A|B|_)_(OPER|TEST)_(AUX_[0-9A-Z_]{7})(.*)$
app.metadata-extraction.worker.product-categories.s2-aux.local-directory=/data/local-catalog/s2_aux/
app.metadata-extraction.worker.product-categories.s2-aux.enable-extraction-from-product-name=true
app.metadata-extraction.worker.product-categories.s2-products.pattern-config=^(S2)(A|B|_)_([A-Z0-9]{4})_((MSI)_(L0_|L1A|L1B|L1C)_(GR|DS|TL|TC)|PRD_HKTM__)_\\w{0,4}_?(\\d{8}T\\d{6})(.*)$
app.metadata-extraction.worker.product-categories.s2-products.local-directory=/data/local-catalog/s2_products/
app.metadata-extraction.worker.product-categories.s2-products.enable-extraction-from-product-name=true
app.metadata-extraction.worker.product-categories.s3-aux.pattern-config=^([a-zA-Z0-9][a-zA-Z0-9])(\\w{1})_((OL|SL|SR|DO|MW|GN|SY|TM|AX)_(0|1|2|_)_\\w{4}AX)_(\\d{8}T\\d{6})_(\\d{8}T\\d{6})_(\\d{8}T\\d{6})_(_{17})_(\\w{3})_(\\w{8})\\.(SEN3)\\/?(.+)?$
app.metadata-extraction.worker.product-categories.s3-aux.local-directory=/data/local-catalog/s3_aux/
app.metadata-extraction.worker.product-categories.s3-products.pattern-config=^([a-zA-Z0-9][a-zA-Z0-9])(\\w{1})_((OL|SL|SR|DO|MW|GN|SY|TM|AX)_(0|1|2|_)_\\w{4}(?!AX)\\w{2})_(\\d{8}T\\d{6})_(\\d{8}T\\d{6})_(\\d{8}T\\d{6})_(\\w{17})_(\\w{3})_(\\w{8})\\.(SEN3|ISIP)\\/?(.+)?$
app.metadata-extraction.worker.product-categories.s3-products.local-directory=/data/local-catalog/s3_products/
app.metadata-extraction.worker.product-insertion.max-retries=3
app.metadata-extraction.worker.product-insertion.tempo-retry-ms=1000
app.metadata-extraction.mdextractor.type-overlap.EW=8.2F
app.metadata-extraction.mdextractor.type-overlap.IW=7.4F
app.metadata-extraction.mdextractor.type-overlap.SM=7.7F
app.metadata-extraction.mdextractor.type-overlap.WV=0.0F
app.metadata-extraction.mdextractor.type-slice-length.EW=60.0F
app.metadata-extraction.mdextractor.type-slice-length.IW=25.0F
app.metadata-extraction.mdextractor.type-slice-length.SM=25.0F
app.metadata-extraction.mdextractor.type-slice-length.WV=0.0F
app.metadata-extraction.mdextractor.xslt-directory=xslt/
app.metadata-extraction.mdextractor.fieldTypes.absoluteStartOrbit=long
app.metadata-extraction.mdextractor.fieldTypes.coordinates=string
app.metadata-extraction.mdextractor.fieldTypes.creationTime=date
app.metadata-extraction.mdextractor.fieldTypes.cycleNumber=long
app.metadata-extraction.mdextractor.fieldTypes.instrumentConfigurationId=string
app.metadata-extraction.mdextractor.fieldTypes.instrumentShortName=string
app.metadata-extraction.mdextractor.fieldTypes.missionDataTakeId=long
app.metadata-extraction.mdextractor.fieldTypes.operationalMode=string
app.metadata-extraction.mdextractor.fieldTypes.pass=string
app.metadata-extraction.mdextractor.fieldTypes.platformSerialIdentifier=string
app.metadata-extraction.mdextractor.fieldTypes.platformShortName=string
app.metadata-extraction.mdextractor.fieldTypes.processingDate=date
app.metadata-extraction.mdextractor.fieldTypes.processorName=string
app.metadata-extraction.mdextractor.fieldTypes.processorVersion=string
app.metadata-extraction.mdextractor.fieldTypes.productClass=string
app.metadata-extraction.mdextractor.fieldTypes.productComposition=string
app.metadata-extraction.mdextractor.fieldTypes.productConsolidation=string
app.metadata-extraction.mdextractor.fieldTypes.productType=string
app.metadata-extraction.mdextractor.fieldTypes.qualityDataObjectID=string
app.metadata-extraction.mdextractor.fieldTypes.qualityNumOfCorruptedElements=long
app.metadata-extraction.mdextractor.fieldTypes.qualityNumOfElement=long
app.metadata-extraction.mdextractor.fieldTypes.qualityNumOfMissingElements=long
app.metadata-extraction.mdextractor.fieldTypes.qualityNumOfRSCorrectedElements=long
app.metadata-extraction.mdextractor.fieldTypes.qualityNumOfRSCorrectedSymbols=long
app.metadata-extraction.mdextractor.fieldTypes.qualityNumOfRSIncorrigibleElements=long
app.metadata-extraction.mdextractor.fieldTypes.relativeStartOrbit=long
app.metadata-extraction.mdextractor.fieldTypes.safeTimeliness=string
app.metadata-extraction.mdextractor.fieldTypes.segmentStartTime=date
app.metadata-extraction.mdextractor.fieldTypes.site=string
app.metadata-extraction.mdextractor.fieldTypes.sliceNumber=long
app.metadata-extraction.mdextractor.fieldTypes.sliceProductFlag=boolean
app.metadata-extraction.mdextractor.fieldTypes.startTimeANX=double
app.metadata-extraction.mdextractor.fieldTypes.startTime=date
app.metadata-extraction.mdextractor.fieldTypes.stopTimeANX=double
app.metadata-extraction.mdextractor.fieldTypes.stopTime=date
app.metadata-extraction.mdextractor.fieldTypes.swathIdentifier=string
app.metadata-extraction.mdextractor.fieldTypes.totalNumberOfSlice=long
app.metadata-extraction.mdextractor.fieldTypes.validityStartTime=date
app.metadata-extraction.mdextractor.fieldTypes.validityStopTime=date
app.metadata-extraction.mdextractor.packet-store-types.S1A-0=Emergency
app.metadata-extraction.mdextractor.packet-store-types.S1A-1=Emergency
app.metadata-extraction.mdextractor.packet-store-types.S1A-2=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1A-3=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1A-4=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1A-5=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1A-6=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-7=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-8=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-9=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-10=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-11=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-12=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-13=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-14=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-15=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-16=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-17=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-18=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1A-19=WV
app.metadata-extraction.mdextractor.packet-store-types.S1A-20=WV
app.metadata-extraction.mdextractor.packet-store-types.S1A-21=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-22=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-23=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-24=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-25=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-26=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-27=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-28=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-29=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-30=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-31=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-32=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-33=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-34=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-35=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-36=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1A-37=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-38=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-39=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-40=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-41=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-42=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-43=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-44=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1A-45=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1A-46=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1A-47=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1A-48=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1A-49=GPS
app.metadata-extraction.mdextractor.packet-store-types.S1A-50=Filler
app.metadata-extraction.mdextractor.packet-store-types.S1B-0=Emergency
app.metadata-extraction.mdextractor.packet-store-types.S1B-1=Emergency
app.metadata-extraction.mdextractor.packet-store-types.S1B-2=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1B-3=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1B-4=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1B-5=RFC
app.metadata-extraction.mdextractor.packet-store-types.S1B-6=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-7=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-8=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-9=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-10=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-11=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-12=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-13=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-14=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-15=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-16=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-17=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-18=NRT
app.metadata-extraction.mdextractor.packet-store-types.S1B-19=WV
app.metadata-extraction.mdextractor.packet-store-types.S1B-20=WV
app.metadata-extraction.mdextractor.packet-store-types.S1B-21=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-22=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-23=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-24=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-25=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-26=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-27=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-28=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-29=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-30=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-31=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-32=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-33=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-34=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-35=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-36=Standard
app.metadata-extraction.mdextractor.packet-store-types.S1B-37=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-38=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-39=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-40=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-41=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-42=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-43=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-44=PassThrough
app.metadata-extraction.mdextractor.packet-store-types.S1B-45=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1B-46=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1B-47=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1B-48=HKTM
app.metadata-extraction.mdextractor.packet-store-types.S1B-49=GPS
app.metadata-extraction.mdextractor.packet-store-types.S1B-50=Filler
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.Emergency=PT
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.HKTM=NRT
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.NRT=NRT
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.GPS=NRT
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.PassThrough=PT
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.Standard=FAST24
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.RFC=FAST24
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.WV=FAST24
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.Filler=FAST24
app.metadata-extraction.mdextractor.packetstore-type-timelinesses.Spare=FAST24
app.metadata-extraction.mdextractor.timeliness-priority-from-high-to-low=PT, NRT, FAST24

# Timeliness configuration (in seconds)
app.metadata-extraction.timeliness.S1_SESSION=3600
app.metadata-extraction.timeliness.S1_NRT=10800
app.metadata-extraction.timeliness.S1_FAST24=86400
app.metadata-extraction.timeliness.S1_PT=10800

app.metadata-extraction.timeliness.S2_SESSION=1800
app.metadata-extraction.timeliness.S2_L0=4500
app.metadata-extraction.timeliness.S2_L1=4500
app.metadata-extraction.timeliness.S2_L2=7200

app.metadata-extraction.timeliness.S3_NRT=10800
app.metadata-extraction.timeliness.S3_NTC=2592000
app.metadata-extraction.timeliness.S3_STC=172800

# Mandatory Deployer Settings

# Liveness Probe
deployer.*.kubernetes.namespace=processing
deployer.*.kubernetes.livenessProbeDelay=10
deployer.*.kubernetes.livenessProbePeriod=120
deployer.*.kubernetes.livenessProbeTimeout=20
deployer.*.kubernetes.livenessProbePath=/actuator/health/liveness
deployer.*.kubernetes.livenessProbePort=8080

# Readiness Probe
deployer.*.kubernetes.readinessProbeDelay=10
deployer.*.kubernetes.readinessProbePeriod=120
deployer.*.kubernetes.readinessProbeTimeout=20
deployer.*.kubernetes.readinessProbePath=/actuator/health/readiness
deployer.*.kubernetes.readinessProbePort=8080

deployer.*.kubernetes.maxTerminatedErrorRestarts=20

# Deployer properties
deployer.metadata-filter.kubernetes.imagePullPolicy=IfNotPresent
deployer.metadata-extraction.kubernetes.imagePullPolicy=Always
deployer.metadata-extraction.kubernetes.imagePullSecrets=spring-cloud-dataflow-registry-dockersecret

deployer.metadata-extraction.count=1
deployer.metadata-extraction.kubernetes.environmentVariables=JAVA_TOOL_OPTIONS=-Xmx512m
deployer.metadata-extraction.kubernetes.requests.memory=3500Mi
deployer.metadata-extraction.kubernetes.requests.cpu=300m
deployer.metadata-extraction.kubernetes.limits.memory=4000Mi
deployer.metadata-extraction.kubernetes.limits.cpu=1500m
deployer.metadata-extraction.kubernetes.secretKeyRefs=[{ envVarName: 'OBS_USERNAME', secretName: 'obs', dataKey: 'USER_ID' },{ envVarName: 'OBS_PASSWORD', secretName: 'obs', dataKey: 'USER_SECRET' }]
deployer.metadata-extraction.kubernetes.volumeMounts=[{ name: 'local-cache', mountPath: '/data/local-catalog'}]
deployer.metadata-extraction.kubernetes.volumes=[{ name: 'local-cache', emptyDir: { medium: 'Memory', sizeLimit: '1500Mi' }}]
# OBS properties

app.*.obs.user-id=${OBS_USERNAME}
app.*.obs.user-secret=${OBS_PASSWORD}
app.*.obs.endpoint=http://oss.eu-west-0.prod-cloud-ocb.orange-business.com
app.*.obs.endpoint-region=eu-west-0
app.*.obs.disable-chunked-encoding=false
app.*.obs.bucket.auxiliary-file=rs-s1-aux
app.*.obs.bucket.edrs-session=rs-session-files
app.*.obs.bucket.plan-and-report=rs-s1-plans-and-reports
app.*.obs.bucket.l0-slice=rs-s1-l0-slices
app.*.obs.bucket.l0-acn=rs-l0-acns
app.*.obs.bucket.l0-segment=rs-s1-l0-segments
app.*.obs.bucket.l0-blank=rs-s1-l0-blanks
app.*.obs.bucket.l1-slice=rs-s1-l1-slices
app.*.obs.bucket.l1-acn=rs-s1-l1-acns
app.*.obs.bucket.l2-slice=rs-s1-l2-slices
app.*.obs.bucket.l2-acn=rs-s1-l2-acns
app.*.obs.bucket.spp-mbu=rs-s1-spp-mbu
app.*.obs.bucket.spp-obs=rs-s1-spp
app.*.obs.bucket.invalid=rs-s1-invalid
app.*.obs.bucket.ghost=rs-s1-ghost
app.*.obs.bucket.debug=rs-debug
app.*.obs.bucket.failed-workdir=rs-failed-workdir
app.*.obs.bucket.session-retransfer=rs-session-retransfer
app.*.obs.bucket.auxiliary-file-zip=rs-s1-aux-zip
app.*.obs.bucket.l0-slice-zip=rs-s1-l0-slices-zip
app.*.obs.bucket.l0-acn-zip=rs-s1-l0-acns-zip
app.*.obs.bucket.l0-segment-zip=rs-s1-l0-segments-zip
app.*.obs.bucket.l0-blank-zip=rs-s1-l0-blanks-zip
app.*.obs.bucket.l1-slice-zip=rs-s1-l1-slices-zip
app.*.obs.bucket.l1-acn-zip=rs-s1-l1-acns-zip
app.*.obs.bucket.l2-slice-zip=rs-s1-l2-slices-zip
app.*.obs.bucket.l2-acn-zip=rs-s1-l2-acns-zip
app.*.obs.bucket.spp-obs-zip=rs-s1-spp-zip
app.*.obs.bucket.plan-and-report-zip=rs-s1-plans-and-reports-zip

app.*.obs.bucket.s2-aux=rs-s2-aux
app.*.obs.bucket.s2-l0-gr=rs-s2-l0c
app.*.obs.bucket.s2-l0-ds=rs-s2-l0c
app.*.obs.bucket.s2-hktm=rs-s2-hktm
app.*.obs.bucket.s2-aux-zip=rs-s2-aux-zip
app.*.obs.bucket.s2-l0-gr-zip=rs-s2-l0c-zip
app.*.obs.bucket.s2-l0-ds-zip=rs-s2-l0c-zip
app.*.obs.bucket.s2-hktm-zip=rs-s2-hktm-zip

app.*.obs.bucket.s3-granules=rs-s3-granules
app.*.obs.bucket.s3-aux=rs-s3-aux
app.*.obs.bucket.s3-l0=rs-s3-l0
app.*.obs.bucket.s3-l1-nrt=rs-s3-l1-nrt
app.*.obs.bucket.s3-l1-stc=rs-s3-l1-stc
app.*.obs.bucket.s3-l1-ntc=rs-s3-l1-ntc
app.*.obs.bucket.s3-l2-nrt=rs-s3-l2-nrt
app.*.obs.bucket.s3-l2-stc=rs-s3-l2-stc
app.*.obs.bucket.s3-l2-ntc=rs-s3-l2-ntc
app.*.obs.bucket.s3-cal=rs-s3-cal
app.*.obs.bucket.s3-pug=rs-s3-pug
app.*.obs.bucket.s3-granules-zip=rs-s3-granules-zip
app.*.obs.bucket.s3-aux-zip=rs-s3-aux-zip
app.*.obs.bucket.s3-l0-zip=rs-s3-l0-zip
app.*.obs.bucket.s3-l1-nrt-zip=rs-s3-l1-nrt-zip
app.*.obs.bucket.s3-l1-stc-zip=rs-s3-l1-stc-zip
app.*.obs.bucket.s3-l1-ntc-zip=rs-s3-l1-ntc-zip
app.*.obs.bucket.s3-l2-nrt-zip=rs-s3-l2-nrt-zip
app.*.obs.bucket.s3-l2-stc-zip=rs-s3-l2-stc-zip
app.*.obs.bucket.s3-l2-ntc-zip=rs-s3-l2-ntc-zip
app.*.obs.bucket.s3-cal-zip=rs-s3-cal-zip
app.*.obs.bucket.s3-pug-zip=rs-s3-pug-zip
# Common Application properties
app.*.common.rsChainName=METADATA
app.*.common.rsChainVersion=1.7.0-rc1

Woljtek commented 1 year ago

At first view, all seems OK Could you provide the full logs of the pod?

w-fsi commented 1 year ago

We just had a look at this issue and it seems like the ingested session is not valid. At a first glance it looks fine, but it lacks the channel information in there. By factory default the expected configuration is: ^([0-9aA-Z-z_]+)/(ch[0|_]?[1-2]/)?(DCS_[0-9]{2}_([a-zA-Z0-9_]*)_ch([12])_(DSDB|DSIB).*\\.(raw|aisp|xml))$

Especially note the second group. This seems to be missing and thus the mission id is not extracted correctly from the term and seems to be null.

A valid request should look more like this:

{"uid":"47b4933e-7364-441f-8c19-f32951634c9b","creationDate":"2022-11-22T13:59:37.147Z","missionId":"S3","satelliteId":"","keyObjectStorage":"DCS_01_S3B_20200928074138009060_dat/ch_1/DCS_01_S3B_20200928074138009060_ch1_DSDB_00016.raw","storagePath":"s3://rs-session-files/DCS_01_S3B_20200928074138009060_dat/ch_1/DCS_01_S3B_20200928074138009060_ch1_DSDB_00016.raw","productFamily":"EDRS_SESSION","podName":"ingestion-xbip-part1-ingestion-xbip-worker-v57-6f9dbb6c5d-7v2w6","allowedActions":[],"retryCounter":0,"additionalFields":{"t0PdgsDate":"2022-11-22T13:57:14.000000Z"},"metadata":{"mode":null,"relativePath":"SGS_/S3B/DCS_01_S3B_20200928074138009060_dat/ch_1/DCS_01_S3B_20200928074138009060_ch1_DSDB_00016.raw","productName":"DCS_01_S3B_20200928074138009060_dat/ch_1/DCS_01_S3B_20200928074138009060_ch1_DSDB_00016.raw"},"demandType":"NOMINAL","debug":false,"timeliness":"","productSizeByte":4096,"oqcFlag":"NOT_CHECKED","stationName":"SGS_"}

Woljtek commented 1 year ago

Hi @eroan-marie , what's the current status of this bug? The priority is blocking and we havent got any news since the CCB Yesterday

eroan-marie commented 1 year ago

Hi, it doesn't seem to be the source of the problem, you can find the metadata extraction pod logs and a kafka message generated by the ingestion worker. logs-metadata-extraction.txt kafka-message.txt

w-fsi commented 1 year ago

I am still quite sure that it is something with the way the test data is ingested as it is working fine in our system and within the operational environment.

What is the ingestion backend that is used? Is it an XBIP? In this case my last information regarding the channel id might be wrong as the data is placed flat within, e.g. s3://rs-session-files/S1B_20200120162933019903/DCS_01_S1B_20200120162933019903_ch2_DSDB_00036.raw

However, when having a look at your relative path it seems like the station it is part of it. It shall look like:

"metadata":{"mode":null,"relativePath":"S1B/S1B_20200120162933019903/DCS_01_S1B_20200120162933019903_ch2_DSDB_00036.raw","productName":"S1B_20200120162933019903/DCS_01_S1B_20200120162933019903_ch2_DSDB_00036.raw"}

However within your kafka it looks like:

"metadata":{"mode":null,"relativePath":"MTI_/S1A/S1A_20200120185900030888/ch1/DCS_01_S1A_20200120185900030888_ch1_DSDB_00017.raw","productName":"S1A_20200120185900030888/ch1/DCS_01_S1A_20200120185900030888_ch1_DSDB_00017.raw"}

How does the structure on the pickup point look like?

nleconte-csgroup commented 1 year ago

The problem does not occur any more. It was the combination of two issues :

The data was placed in the wrong arborescence on the mock
- Our procedure shall be updated and/or more detailed
Issue https://github.com/COPRS/rs-issues/issues/716

LAQU156 commented 1 year ago

IVV_CCB_2022_w48 : Closed, Won't fix

COPRS / rs-issues

[BUG] [PRO] Metadata catalog empty after ingestion #725