[BUG] [TRACE] [MONITORING] Trace JobProcessing S3-OL1-RAC-NRT and S3-SL1-NTC with status NOK are not stored on the database.

[pcuq-ads]

Environment:

• Platform: OPS Orange Cloud
• Configuration: OPS
• Infrastructure v1.5.0
• RS core Monitoring 1.5.0-rc1
• RS add-on S3-OL1-RAC-NRT 1.13.1-rc1

Traceability:

• Test(s):
• Requirements:

Current Behavior: Trace JobProcessing S3-OL1-RAC-NRT and S3-SL1-NTC with status NOK are not stored on the database. Trace S3-OL1-RAC-NRT and S3-SL1-NTC OK are well treated by RS core monitoring.

Expected Behavior: All JobProcessing trace shall be analysed and stored on processing and product tables.

Steps To Reproduce: Check database content for chain S3-OL1-RAC-NRT with status NOK.

Test execution artefacts (i.e. logs, screenshots…)

{
  "header": {
    "type": "REPORT",
    "timestamp": "2023-06-01T14:30:06.462000Z",
    "level": "ERROR",
    "mission": "S3",
    "workflow": "NOMINAL",
    "rs_chain_name": "S3-OL1-RAC-NRT",
    "rs_chain_version": "1.13.1-rc1"
  },
  "message": {
    "content": "[code 1] [msg {'bucket': \"ops-rs-cal\", 'key': \"S3A_OL_1_RAC____20230527T190353_20230527T190439_20230601T142943_0045_099_184______LN3_D_NR_002.SEN3\", 'msg': \"Getting number of objects fails: Error: Number of retries has exceeded while performing number of objects ops-rs-cal/S3A_OL_1_RAC____20230527T190353_20230527T190439_20230601T142943_0045_099_184______LN3_D_NR_002.SEN3 after 11 attempts: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 00000188775D73D3900F62BEC5761D26; S3 Extended Request ID: 1t5SDgDWtPwOhAmUgudoBTzO/rurAjpnH4YAOXPPvfm+5IAVRots8IN/DOLzyBsy), S3 Extended Request ID: 1t5SDgDWtPwOhAmUgudoBTzO/rurAjpnH4YAOXPPvfm+5IAVRots8IN/DOLzyBsy\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4266)\n\tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:834)\n\tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:809)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.lambda$getNbObjects$2(S3ObsServices.java:182)\n\tat esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.getNbObjects(S3ObsServices.java:180)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsClient.prefixExists(S3ObsClient.java:193)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsClient.existsWithSameSize(S3ObsClient.java:173)\n\tat esa.s1pdgs.cpoc.obs_sdk.UnrecoverableErrorAwareObsClient.existsWithSameSize(UnrecoverableErrorAwareObsClient.java:102)\n\tat esa.s1pdgs.cpoc.ipf.execution.worker.job.file.OutputProcessor.addToUploadAndPublishIfNotExistsInObs(OutputProcessor.java:335)\n\tat esa.s1pdgs.cpoc.ipf.execution.worker.job.file.OutputProcessor.sortOutputs(OutputProcessor.java:316)\n\tat esa.s1pdgs.cpoc.ipf.execution.worker.job.file.OutputProcessor.processOutput(OutputProcessor.java:695)\n\tat esa.s1pdgs.cpoc.ipf.execution.worker.service.ExecutionWorkerService.processJob(ExecutionWorkerService.java:265)\n\tat esa.s1pdgs.cpoc.ipf.execution.worker.service.ExecutionWorkerService.apply(ExecutionWorkerService.java:223)\n\tat esa.s1pdgs.cpoc.ipf.execution.worker.service.ExecutionWorkerService.apply(ExecutionWorkerService.java:87)\n\tat org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunctionAndEnrichResultIfNecessary(SimpleFunctionRegistry.java:897)\n\tat org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeFunction(SimpleFunctionRegistry.java:853)\n\tat org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:708)\n\tat org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)\n\tat org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:84)\n\tat org.springframework.cloud.stream.function.FunctionConfiguration$FunctionWrapper.apply(FunctionConfiguration.java:754)\n\tat org.springframework.cloud.stream.function.FunctionConfiguration$FunctionToDestinationBinder$1.handleMessageInternal(FunctionConfiguration.java:586)\n\tat org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56)\n\tat org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)\n\tat org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)\n\tat org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)\n\tat org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)\n\tat org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317)\n\tat org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272)\n\tat org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)\n\tat org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)\n\tat org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)\n\tat org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)\n\tat org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:216)\n\tat org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:397)\n\tat org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:83)\n\tat org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:454)\n\tat org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:428)\n\tat org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0(RetryingMessageListenerAdapter.java:125)\n\tat org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)\n\tat org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)\n\tat org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:119)\n\tat org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:42)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2629)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2609)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355)\n\tat org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n\" }]"
  },
  "task": {
    "uid": "07aafd63-43a4-4a92-bf8b-17db9f22c856",
    "name": "JobProcessing",
    "event": "END",
    "status": "NOK",
    "output": {},
    "input": {
      "job_order_id_string": "JobOrder.91634.xml",
      "ipf_release_string": "06.13",
      "filename_strings": [
        "S3A_AX___FRO_AX_20230527T000000_20230606T000000_20230530T065508___________________EUM_O_AL_001.SEN3",
        "S3A_AX___OSF_AX_20160216T192404_99991231T235959_20220330T090651___________________EUM_O_AL_001.SEN3",
        "S3A_TM_0_NAT____20230527T181746_20230527T195805_20230601T141954_6018_099_184______LN3_D_AL_002.SEN3",
        "JobOrder.91634.xml",
        "S3A_OL_1_CAL_AX_20211027T000000_20991231T235959_20220505T120000___________________MPC_O_AL_026.SEN3",
        "S3A_OL_1_RAC_AX_20160425T103700_20991231T235959_20210309T120000___________________MPC_O_AL_006.SEN3",
        "S3A_OL_0_CR0____20230527T190353_20230527T190439_20230601T142635_0045_099_184______LN3_D_NR_002.SEN3",
        "S3A_OL_1_INS_AX_20201030T120000_20991231T235959_20220505T120000___________________MPC_O_AL_009.SEN3",
        "S3A_OL_1_PRG_AX_20160425T095210_20991231T235959_20210309T120000___________________MPC_O_AL_004.SEN3"
      ]
    },
    "quality": {},
    "error_code": 1,
    "duration_in_seconds": 53.781,
    "missing_output": []
  }
}

Whenever possible, first analysis of the root cause Hypothesis :

Bug Generic Definition of Ready (DoR)

• [X] The affect version in which the bug has been found is mentioned
• [X] The context and environment of the bug is detailed
• [X] The description of the bug is clear and unambiguous
• [X] The procedure (steps) to reproduce the bug is clearly detailed
• [ ] The tested User Story / features is linked to the bug if available
• [ ] Logs are attached if available
• [X] A data set attached if available

Bug Generic Definition of Done (DoD)

• [ ] the modification implemented (the solution to fix the bug) is described in the bug.
• [ ] Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
• [ ] Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
• [ ] Code is compliant with coding rules (SONAR Report as evidence)
• [ ] Acceptance criteria of the related User story are checked and Passed

[pcuq-ads]

I observe the same issue for chain S3-SL1-NTC. A processing NOK is not stored on the database.

{
  "header": {
    "type": "REPORT",
    "timestamp": "2023-06-23T16:38:11.034000Z",
    "level": "ERROR",
    "mission": "S3",
    "workflow": "NOMINAL",
    "rs_chain_name": "S3-SL1-NTC",
    "rs_chain_version": "1.13.1-rc1"
  },
  "message": {
    "content": "[code 53] [msg esa.s1pdgs.cpoc.common.errors.obs.ObsParallelAccessException: {'bucket': \"ops-rs-s3-l0\", 'key': \"S3B_TM_0_NAT____20230409T200344_20230409T214417_20230409T224321_6032_078_128______LN3_D_AL_002.SEN3\", 'msg': \"Download in /data/localWD/100602 fails: Error: Number of retries has exceeded while performing download objects ops-rs-s3-l0/S3B_TM_0_NAT____20230409T200344_20230409T214417_20230409T224321_6032_078_128______LN3_D_AL_002.SEN3 to /data/localWD/100602 after 11 attempts: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 00000188E91ED02F901A760AB48750C1; S3 Extended Request ID: a7PBm8O+yFb6Mf1np+ooPAovRRP8UCuUfw/ds7w3Bm6wYt08WpWzffz0oPWI29pA), S3 Extended Request ID: a7PBm8O+yFb6Mf1np+ooPAovRRP8UCuUfw/ds7w3Bm6wYt08WpWzffz0oPWI29pA\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)\n\tat com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1409)\n\tat com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1270)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.getExpectedFiles(S3ObsServices.java:286)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.lambda$downloadObjectsWithPrefix$3(S3ObsServices.java:218)\n\tat esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.downloadObjectsWithPrefix(S3ObsServices.java:212)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsClient.downloadObject(S3ObsClient.java:200)\n\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.downloadObjects(AbstractObsClient.java:109)\n\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.download(AbstractObsClient.java:244)\n\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.lambda$downloadCall$0(AbstractObsClient.java:128)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n\" }]"
  },
  "task": {
    "uid": "3c376864-c67e-4092-bcff-c5eb58ca9132",
    "name": "JobProcessing",
    "event": "END",
    "status": "NOK",
    "output": {},
    "input": {
      "job_order_id_string": "JobOrder.100602.xml",
      "ipf_release_string": "06.19",
      "filename_strings": [
        "S3B_SL_1_O_F2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_N_F1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3",
        "S3B_SL_1_N_F2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3",
        "S3B_SL_1_NAS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_O_S1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_VIC_AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3",
        "S3__AX___MA1_AX_20230408T210000_20230409T090000_20230409T054711___________________ECW_O_SN_001.SEN3",
        "S3__AX___MFA_AX_20230409T150000_20230410T030000_20230409T181335___________________ECW_O_NT_001.SEN3",
        "S3B_SL_1_O_S2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_N_S2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_CDP_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3B_TM_0_NAT____20230409T182323_20230409T200339_20230409T204432_6016_078_127______LN3_D_AL_002.SEN3",
        "S3B_SL_1_N_S3AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3__AX___MA1_AX_20230409T150000_20230410T030000_20230410T055056___________________ECW_O_SN_001.SEN3",
        "S3B_SL_1_N_S8AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_ESSTAX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_RTT_AX_20180425T000000_20991231T235959_20211201T120000___________________MPC_O_AL_002.SEN3",
        "S3__AX___DEM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_O_F1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3",
        "S3B_SL_1_CLO_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_NBS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_AX___FRO_AX_20230409T000000_20230419T000000_20230412T065540___________________EUM_O_AL_001.SEN3",
        "S3B_SL_1_CLP_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_O_S3AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_LCC_AX_20180425T000000_20991231T235959_20201015T120000___________________MPC_O_AL_002.SEN3",
        "S3B_SL_1_VSC_AX_20230409T212600_20500101T000000_20230419T133208___________________LN3_D_NN____.SEN3",
        "S3__AX___MFA_AX_20230409T210000_20230410T090000_20230409T182422___________________ECW_O_NT_001.SEN3",
        "S3__AX___MFA_AX_20230409T030000_20230409T150000_20230409T055759___________________ECW_O_NT_001.SEN3",
        "S3__AX___OOM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_IRE_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3__AX___MFA_AX_20230409T090000_20230409T210000_20230409T055758___________________ECW_O_NT_001.SEN3",
        "S3B_SL_1_OAS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_GEC_AX_20190101T000000_20991231T235959_20191010T120000___________________MPC_O_AL_004.SEN3",
        "S3B_SL_1_OAS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_N_S1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_TM_0_NAT____20230409T182323_20230409T200343_20230409T205001_6020_078_127______LN3_D_AL_002.SEN3",
        "S3B_SL_1_O_S8AX_20180425T000000_20991231T235959_20180712T120000___________________MPC_O_AL_002.SEN3",
        "S3B_SL_0_SLT____20230409T194504_20230409T212603_20230516T095003_6059_078_127______LN3_O_NR_002.SEN3",
        "S3B_SL_0_SLT____20230409T212603_20230409T230702_20230516T094437_6059_078_128______LN3_O_NR_002.SEN3",
        "S3__AX___MA1_AX_20230409T030000_20230409T150000_20230409T174059___________________ECW_O_SN_001.SEN3",
        "S3__AX___TRM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3__AX___MA1_AX_20230410T030000_20230410T150000_20230410T175656___________________ECW_O_SN_001.SEN3",
        "S3B_SL_1_ANC_AX_20180425T000000_20991231T235959_20190912T120000___________________MPC_O_AL_003.SEN3",
        "S3__AX___MFA_AX_20230408T210000_20230409T090000_20230408T180246___________________ECW_O_NT_001.SEN3",
        "S3__AX___CLM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_O_S7AX_20180425T000000_20991231T235959_20200410T120000___________________MPC_O_AL_004.SEN3",
        "S3B_AX___OSF_AX_20180425T191855_99991231T235959_20221110T110324___________________EUM_O_AL_001.SEN3",
        "S3B_SL_1_NBS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_O_S9AX_20180425T000000_20991231T235959_20180712T120000___________________MPC_O_AL_002.SEN3",
        "S3B_SL_1_OAS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_TM_0_NAT____20230409T214431_20230409T232544_20230410T000748_6073_078_129______LN3_D_AL_002.SEN3",
        "S3__AX___MA1_AX_20230409T210000_20230410T090000_20230410T055055___________________ECW_O_SN_001.SEN3",
        "S3B_SL_1_OBS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3__AX___MA1_AX_20230409T090000_20230409T210000_20230409T175145___________________ECW_O_SN_001.SEN3",
        "S3B_SL_1_NBS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_GEO_AX_20180425T000000_20991231T235959_20190912T120000___________________MPC_O_AL_003.SEN3",
        "S3__AX___LWM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3B_TM_0_NAT____20230409T214431_20230409T232603_20230410T000734_6092_078_129______LN3_D_AL_002.SEN3",
        "S3__SL_2_LSTBAX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3B_TM_0_NAT____20230409T200344_20230409T214417_20230409T224321_6032_078_128______LN3_D_AL_002.SEN3",
        "S3B_TM_0_NAT____20230409T200344_20230409T214430_20230409T222557_6045_078_128______LN3_D_AL_002.SEN3",
        "S3B_SL_1_NAS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_OBS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_PCP_AX_20180425T000000_20991231T235959_20211011T120000___________________MPC_O_AL_004.SEN3",
        "S3B_SL_1_ADJ_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3__AX___MFA_AX_20230410T030000_20230410T150000_20230410T055055___________________ECW_O_NT_001.SEN3",
        "S3B_SL_1_N_S9AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3",
        "S3B_SL_1_NAS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3",
        "JobOrder.100602.xml",
        "S3B_SL_0_SLT____20230409T230702_20230409T231930_20230516T103525_0747_078_129______LN3_O_NR_002.SEN3",
        "S3B_SL_1_N_S7AX_20180425T000000_20991231T235959_20190912T120000___________________MPC_O_AL_003.SEN3",
        "S3B_SL_1_OBS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3"
      ]
    },
    "quality": {},
    "error_code": 1,
    "duration_in_seconds": 5.417,
    "missing_output": []
  }
}

[nleconte-csgroup]

I was able to ingest a modified version of these traces on IVV. The problem seems to come from the message.content field. It either contains too many characters or invalid characters for the filter component. I'm still looking at it.

[nleconte-csgroup]

@pcuq-ads Is it possible to provide the complete kafka message from the topic fluentbit.trace ? I can't find them, they are probably already rolled

EDIT : nevermind I found another example

[pcuq-ads]

Here is an other trace :

2023-07-02T17:01:52Z    {"header":{"type":"REPORT","timestamp":"2023-07-02T17:01:52.948000Z","level":"ERROR","mission":"S3","workflow":"NOMINAL","rs_chain_name":"S3-SL1-NTC","rs_chain_version":"1.13.1-rc1"},"message":{"content":"[code 53] [msg esa.s1pdgs.cpoc.common.errors.obs.ObsParallelAccessException: {'bucket': \"ops-rs-l0\", 'key': \"S3B_TM_0_NAT____20230427T135726_20230427T153643_20230427T163249_5956_078_381______LN3_D_AL_002.SEN3\", 'msg': \"Download in /data/localWD/102796 fails: Error: Number of retries has exceeded while performing download objects ops-rs-l0/S3B_TM_0_NAT____20230427T135726_20230427T153643_20230427T163249_5956_078_381______LN3_D_AL_002.SEN3 to /data/localWD/102796 after 11 attempts: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 00000189178D4F8A901A51B27AA6F8FB; S3 Extended Request ID: gFnJ4DB94pnV8eGD58txBDMQJcN4Y/M0Dw2lFb1qLfAcLud+YBB8VfL0dUOHOyJU), S3 Extended Request ID: gFnJ4DB94pnV8eGD58txBDMQJcN4Y/M0Dw2lFb1qLfAcLud+YBB8VfL0dUOHOyJU\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)\n\tat com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1409)\n\tat com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1270)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.getExpectedFiles(S3ObsServices.java:286)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.lambda$downloadObjectsWithPrefix$3(S3ObsServices.java:218)\n\tat esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.downloadObjectsWithPrefix(S3ObsServices.java:212)\n\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsClient.downloadObject(S3ObsClient.java:200)\n\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.downloadObjects(AbstractObsClient.java:109)\n\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.download(AbstractObsClient.java:244)\n\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.lambda$downloadCall$0(AbstractObsClient.java:128)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n\" }]"},"task":{"uid":"05eae3b3-1359-4cb9-bb66-f1be11c9a94a","name":"JobProcessing","event":"END","status":"NOK","output":{},"input":{"job_order_id_string":"JobOrder.102796.xml","ipf_release_string":"06.19","filename_strings":["S3B_TM_0_NAT____20230427T121743_20230427T135718_20230427T144300_5975_078_380______LN3_D_AL_002.SEN3","S3B_SL_1_VSC_AX_20230427T151543_20500101T000000_20230427T221531___________________LN3_D_NN____.SEN3","S3B_SL_1_O_S3AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_PCP_AX_20180425T000000_20991231T235959_20211011T120000___________________MPC_O_AL_004.SEN3","S3B_SL_1_NAS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3__AX___MA1_AX_20230427T150000_20230428T030000_20230428T054206___________________ECW_O_SN_001.SEN3","S3__AX___CLM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3","S3__AX___LWM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_N_S9AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_IRE_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_NBS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_TM_0_NAT____20230427T121743_20230427T135725_20230427T144345_5981_078_380______LN3_D_AL_002.SEN3","S3__AX___MFA_AX_20230427T210000_20230428T090000_20230427T175740___________________ECW_O_NT_001.SEN3","S3B_SL_1_ADJ_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3B_SL_0_SLT____20230427T151549_20230427T153225_20230603T012827_0995_078_381______LN3_O_NR_002.SEN3","S3B_SL_1_N_S8AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3__AX___OOM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3","S3B_AX___OSF_AX_20180425T191855_99991231T235959_20221110T110324___________________EUM_O_AL_001.SEN3","S3B_SL_1_GEC_AX_20190101T000000_20991231T235959_20191010T120000___________________MPC_O_AL_004.SEN3","S3B_SL_1_CLP_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_ESSTAX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3__AX___MA1_AX_20230427T030000_20230427T150000_20230427T174655___________________ECW_O_SN_001.SEN3","S3B_SL_1_OAS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_N_S7AX_20180425T000000_20991231T235959_20190912T120000___________________MPC_O_AL_003.SEN3","S3__AX___MFA_AX_20230426T150000_20230427T030000_20230426T175739___________________ECW_O_NT_001.SEN3","S3B_SL_1_VIC_AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3","S3B_SL_1_N_F1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3","S3B_SL_1_OBS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_O_S1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3__AX___MA1_AX_20230426T150000_20230427T030000_20230427T054056___________________ECW_O_SN_001.SEN3","S3__AX___DEM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_OAS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","JobOrder.102796.xml","S3B_TM_0_NAT____20230427T135726_20230427T153700_20230427T163220_5973_078_381______LN3_D_AL_002.SEN3","S3__AX___MA1_AX_20230427T090000_20230427T210000_20230427T174654___________________ECW_O_SN_001.SEN3","S3B_SL_1_OBS5AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_TM_0_NAT____20230427T135726_20230427T153643_20230427T163249_5956_078_381______LN3_D_AL_002.SEN3","S3B_SL_1_O_S8AX_20180425T000000_20991231T235959_20180712T120000___________________MPC_O_AL_002.SEN3","S3__AX___MFA_AX_20230427T090000_20230427T210000_20230427T060246___________________ECW_O_NT_001.SEN3","S3B_SL_1_O_F1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3","S3B_SL_1_CDP_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_ANC_AX_20180425T000000_20991231T235959_20190912T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_NAS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_OAS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_NBS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_N_F2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_002.SEN3","S3__AX___MFA_AX_20230427T030000_20230427T150000_20230427T055200___________________ECW_O_NT_001.SEN3","S3B_SL_1_OBS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_N_S1AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_GEO_AX_20180425T000000_20991231T235959_20190912T120000___________________MPC_O_AL_003.SEN3","S3__AX___MA1_AX_20230427T210000_20230428T090000_20230428T055253___________________ECW_O_SN_001.SEN3","S3B_SL_1_N_S3AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_N_S2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3__AX___MA1_AX_20230426T210000_20230427T090000_20230427T055159___________________ECW_O_SN_001.SEN3","S3__AX___MFA_AX_20230427T150000_20230428T030000_20230427T175742___________________ECW_O_NT_001.SEN3","S3__AX___TRM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_RTT_AX_20180425T000000_20991231T235959_20211201T120000___________________MPC_O_AL_002.SEN3","S3B_SL_1_O_F2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_0_SLT____20230427T135725_20230427T151549_20230603T002633_4703_078_381______LN3_O_NR_002.SEN3","S3B_SL_1_CLO_AX_20180425T000000_20991231T235959_20180409T120000___________________MPC_O_AL_001.SEN3","S3__AX___MFA_AX_20230426T210000_20230427T090000_20230426T175738___________________ECW_O_NT_001.SEN3","S3B_SL_1_O_S9AX_20180425T000000_20991231T235959_20180712T120000___________________MPC_O_AL_002.SEN3","S3B_AX___FRO_AX_20230427T000000_20230507T000000_20230430T065356___________________EUM_O_AL_001.SEN3","S3B_SL_1_NBS4AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3__SL_2_LSTBAX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3","S3B_SL_1_NAS6AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3","S3B_SL_1_LCC_AX_20180425T000000_20991231T235959_20201015T120000___________________MPC_O_AL_002.SEN3","S3B_SL_1_O_S7AX_20180425T000000_20991231T235959_20200410T120000___________________MPC_O_AL_004.SEN3","S3B_SL_1_O_S2AX_20180425T000000_20991231T235959_20181002T120000___________________MPC_O_AL_003.SEN3"]},"quality":{},"error_code":1,"duration_in_seconds":35.318,"missing_output":[{"product_metadata_custom_object":{"processing_level_integer":1,"orbit_number_integer":null,"product_type_string":"SL_1_RBT___","platform_serial_identifier_string":"B","ending_date_time_date":"2023-04-27T15:32:24.668418Z","instrument_short_name_string":"SL","platform_short_name_string":"SENTINEL-3","beginning_date_time_date":"2023-04-27T15:15:48.897411Z"},"end_to_end_product_boolean":true,"estimated_count_integer":1}]}}

[pcuq-ads]

For SR1-NTC, here are traces NOK. None of them are stored on the database.

https://app.zenhub.com/files/398313496/df45cac5-c0d4-4ad8-b3b5-3a02d98f2600/download

[nleconte-csgroup]

A fix is available on the develop branch : https://github.com/COPRS/monitoring/commit/84be84a5e651ca79d09c23ccdea8cd22b16f8c1f

This issue will be fixed in version 1.6.0-rc1.

[nleconte-csgroup]

The following trace has been ingested :

{"@timestamp":1688214010.276091,"time":"2023-07-01T12:20:10.2760914Z","stream":"stdout","_p":"F","log":"{\"header\":{\"type\":\"REPORT\",\"timestamp\":\"2023-07-01T12:20:10.272000Z\",\"level\":\"ERROR\",\"mission\":\"S3\",\"workflow\":\"NOMINAL\",\"rs_chain_name\":\"S3-SL1-NTC\",\"rs_chain_version\":\"1.13.1-rc1\"},\"message\":{\"content\":\"[code 53] [msg esa.s1pdgs.cpoc.common.errors.obs.ObsParallelAccessException: {'bucket': \\\"ops-rs-l0\\\", 'key': \\\"S3A_TM_0_NAT____20230427T125626_20230427T143603_20230427T151800_5977_098_138______LN3_D_AL_002.SEN3\\\", 'msg': \\\"Download in /data/localWD/102564 fails: Error: Number of retries has exceeded while performing download objects ops-rs-l0/S3A_TM_0_NAT____20230427T125626_20230427T143603_20230427T151800_5977_098_138______LN3_D_AL_002.SEN3 to /data/localWD/102564 after 11 attempts: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 0000018911650BBF981E14E2D0553F48; S3 Extended Request ID: 56vjXmzjtnnWCFVnI+iFO4VnVg3K2DBFGT4P6DnIuSe576zy4HVmnDibmj05GDy5), S3 Extended Request ID: 56vjXmzjtnnWCFVnI+iFO4VnVg3K2DBFGT4P6DnIuSe576zy4HVmnDibmj05GDy5\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\\n\\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\\n\\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\\n\\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)\\n\\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)\\n\\tat com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1409)\\n\\tat com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1270)\\n\\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.getExpectedFiles(S3ObsServices.java:286)\\n\\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.lambda$downloadObjectsWithPrefix$3(S3ObsServices.java:218)\\n\\tat esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)\\n\\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsServices.downloadObjectsWithPrefix(S3ObsServices.java:212)\\n\\tat esa.s1pdgs.cpoc.obs_sdk.s3.S3ObsClient.downloadObject(S3ObsClient.java:200)\\n\\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.downloadObjects(AbstractObsClient.java:109)\\n\\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.download(AbstractObsClient.java:244)\\n\\tat esa.s1pdgs.cpoc.obs_sdk.AbstractObsClient.lambda$downloadCall$0(AbstractObsClient.java:128)\\n\\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\\n\\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\\n\\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\\n\\tat java.base/java.lang.Thread.run(Thread.java:829)\\n\\\" }]\"},\"task\":{\"uid\":\"ed06259e-3083-43b2-b1e9-c77caab149ef\",\"name\":\"JobProcessing\",\"event\":\"END\",\"status\":\"NOK\",\"output\":{},\"input\":{\"job_order_id_string\":\"JobOrder.102564.xml\",\"ipf_release_string\":\"06.19\",\"filename_strings\":[\"S3A_SL_1_VIC_AX_20160216T000000_20991231T235959_20161012T120000___________________MPC_O_AL_004.SEN3\",\"S3__AX___MFA_AX_20230427T090000_20230427T210000_20230427T060246___________________ECW_O_NT_001.SEN3\",\"S3A_SL_1_O_S8AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_GEO_AX_20160216T000000_20991231T235959_20190912T120000___________________MPC_O_AL_008.SEN3\",\"S3A_SL_1_CLP_AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_001.SEN3\",\"S3A_SL_1_NBS4AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_010.SEN3\",\"S3__AX___MA1_AX_20230427T150000_20230428T030000_20230428T054206___________________ECW_O_SN_001.SEN3\",\"S3__AX___MFA_AX_20230426T210000_20230427T090000_20230426T175738___________________ECW_O_NT_001.SEN3\",\"S3__AX___OOM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3\",\"S3__AX___MFA_AX_20230427T150000_20230428T030000_20230427T175742___________________ECW_O_NT_001.SEN3\",\"S3A_SL_1_O_S3AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_011.SEN3\",\"S3A_SL_1_CLO_AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_N_S8AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_OBS5AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_012.SEN3\",\"S3A_SL_1_OAS4AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_012.SEN3\",\"S3__AX___LWM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3\",\"S3__AX___TRM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3\",\"S3A_SL_1_IRE_AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_001.SEN3\",\"S3__AX___CLM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3\",\"S3A_TM_0_NAT____20230427T125626_20230427T143602_20230427T151840_5975_098_138______LN3_D_AL_002.SEN3\",\"S3A_SL_1_OAS6AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_012.SEN3\",\"S3A_SL_1_OBS4AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_012.SEN3\",\"S3A_SL_1_N_F1AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"JobOrder.102564.xml\",\"S3A_SL_1_OBS6AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_012.SEN3\",\"S3A_SL_1_RTT_AX_20160216T000000_20991231T235959_20211201T120000___________________MPC_O_AL_002.SEN3\",\"S3A_SL_0_SLT____20230427T141400_20230427T143125_20230602T182744_1044_098_138______LN3_O_NR_002.SEN3\",\"S3A_SL_1_O_F2AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_NAS4AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_010.SEN3\",\"S3A_SL_1_ANC_AX_20160216T000000_20991231T235959_20190912T120000___________________MPC_O_AL_010.SEN3\",\"S3A_SL_1_LCC_AX_20160216T000000_20991231T235959_20201015T120000___________________MPC_O_AL_002.SEN3\",\"S3A_SL_1_O_F1AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_GEC_AX_20190101T000000_20991231T235959_20191010T120000___________________MPC_O_AL_009.SEN3\",\"S3A_SL_1_N_F2AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3__AX___MA1_AX_20230427T030000_20230427T150000_20230427T174655___________________ECW_O_SN_001.SEN3\",\"S3A_AX___OSF_AX_20160216T192404_99991231T235959_20220330T090651___________________EUM_O_AL_001.SEN3\",\"S3__AX___MFA_AX_20230426T090000_20230426T210000_20230426T060243___________________ECW_O_NT_001.SEN3\",\"S3A_SL_1_O_S2AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_011.SEN3\",\"S3__AX___MFA_AX_20230427T030000_20230427T150000_20230427T055200___________________ECW_O_NT_001.SEN3\",\"S3A_SL_1_O_S7AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_TM_0_NAT____20230427T125626_20230427T143603_20230427T151800_5977_098_138______LN3_D_AL_002.SEN3\",\"S3A_AX___FRO_AX_20230427T000000_20230507T000000_20230430T065556___________________EUM_O_AL_001.SEN3\",\"S3A_SL_1_N_S3AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_009.SEN3\",\"S3__AX___MA1_AX_20230426T210000_20230427T090000_20230427T055159___________________ECW_O_SN_001.SEN3\",\"S3A_SL_1_PCP_AX_20160216T000000_20991231T235959_20211011T120000___________________MPC_O_AL_013.SEN3\",\"S3__AX___MA1_AX_20230426T090000_20230426T210000_20230426T174653___________________ECW_O_SN_001.SEN3\",\"S3__AX___DEM_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3\",\"S3A_SL_1_CDP_AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_001.SEN3\",\"S3A_SL_1_OAS5AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_012.SEN3\",\"S3__AX___MA1_AX_20230426T150000_20230427T030000_20230427T054056___________________ECW_O_SN_001.SEN3\",\"S3__SL_2_LSTBAX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3\",\"S3A_SL_1_N_S9AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_N_S1AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_009.SEN3\",\"S3A_SL_1_N_S7AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_NBS5AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_010.SEN3\",\"S3A_SL_1_N_S2AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_009.SEN3\",\"S3A_SL_1_NBS6AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_010.SEN3\",\"S3A_SL_0_SLT____20230427T125625_20230427T141400_20230602T185317_4654_098_138______LN3_O_NR_002.SEN3\",\"S3A_SL_1_O_S9AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3\",\"S3A_SL_1_ESSTAX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_001.SEN3\",\"S3__AX___MFA_AX_20230426T150000_20230427T030000_20230426T175739___________________ECW_O_NT_001.SEN3\",\"S3A_SL_1_VSC_AX_20230427T141357_20500101T000000_20230427T202321___________________LN3_D_NN____.SEN3\",\"S3A_SL_1_O_S1AX_20160418T094050_20991231T235959_20180202T120000___________________MPC_O_AL_011.SEN3\",\"S3A_SL_1_NAS6AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_010.SEN3\",\"S3A_SL_1_NAS5AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_010.SEN3\",\"S3__AX___MA1_AX_20230427T090000_20230427T210000_20230427T174654___________________ECW_O_SN_001.SEN3\",\"S3A_SL_1_ADJ_AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_001.SEN3\"]},\"quality\":{},\"error_code\":1,\"duration_in_seconds\":34.864,\"missing_output\":[{\"product_metadata_custom_object\":{\"processing_level_integer\":1,\"orbit_number_integer\":null,\"product_type_string\":\"SL_1_RBT___\",\"platform_serial_identifier_string\":\"A\",\"ending_date_time_date\":\"2023-04-27T14:31:24.526327Z\",\"instrument_short_name_string\":\"SL\",\"platform_short_name_string\":\"SENTINEL-3\",\"beginning_date_time_date\":\"2023-04-27T14:13:59.663856Z\"},\"end_to_end_product_boolean\":true,\"estimated_count_integer\":1}]}}","kubernetes":{"pod_name":"s3-sl1-ntc-part1-execution-worker-v2-9f8d6d5b6-6l9sx","namespace_name":"processing","pod_id":"ae15a12e-b831-4259-a096-fc4a34f09e69","labels":{"linkerd.io/control-plane-ns":"networking","linkerd.io/proxy-deployment":"s3-sl1-ntc-part1-execution-worker-v2","linkerd.io/workload-ns":"processing","pod-template-hash":"9f8d6d5b6","role":"spring-app","spring-app-id":"s3-sl1-ntc-part1-execution-worker-v2","spring-application-name":"execution-worker","spring-deployment-id":"s3-sl1-ntc-part1-execution-worker-v2","spring-group-id":"s3-sl1-ntc-part1"},"host":"cluster-ops-node-139","container_name":"s3-sl1-ntc-part1-execution-worker-v2","docker_id":"13dfa28a090ea6d81becb948136c278e5a25c64b0febcd321ccd334a79314220","container_hash":"artifactory.coprs.esa-copernicus.eu/rs-docker-private/rs-ipf-s3-sl1@sha256:2e41ce794be958922bbf2025e9fddce17e46f40b4471a80737dc04876927c1e8","container_image":"artifactory.coprs.esa-copernicus.eu/rs-docker-private/rs-ipf-s3-sl1:1.13.1-rc1"}}

And now the NOK processing is in the database :

monitoring=# select * from processing where rs_chain_name='S3-SL1-NTC';
  id  | mission | rs_chain_name | rs_chain_version | workflow | level | status |     processing_date     | end_sensing_date | t0_pdgs_date | duplicate 
------+---------+---------------+------------------+----------+-------+--------+-------------------------+------------------+--------------+-----------
 2168 | S3      | S3-SL1-NTC    | 1.13.1-rc1       | NOMINAL  | ERROR | NOK    | 2023-07-01 12:20:10.272 |                  |              | f
(1 row)

If you want this fix earlier in OPS, you can use the tag 1.5.0-develop for the filter component.

[LAQU156]

IVV_CCB_2023_w27 : Moved into Accepted CS for correction on moniroting V1.6.0, Priority major.

[LAQU156]

CS_CCB_2023_w27 : Moved into "Sprint Backlog" for the correction on Monitoring V1.6.0

[nleconte-csgroup]

Delivered in monitoring 1.6.0-rc1 (https://github.com/COPRS/monitoring/releases)

[pcuq-ads]

SYS_CCB_w29 : This issue is fixed with release 1.6.

Test with version 1.5.0-developed (=1.6).

We can see OL1-RAC-NRT processing NOK captured by the monitoring.