[BUG] S3-SL2-NRT, no missing output section on the trace for a processing NOK.

pcuq-ads commented 1 year ago

Environment:

Platform: OPS Orange Cloud
Configuration: OPS
Infrastructure v1.5.0
RS core ingestion, distribution, data lifecycle, compression 1.13.1-rc1
RS core metadata 1.13.1-rc1
RS core Monitoring 1.5.0-rc1
RS add-on S3-SL2-NRT 1.13.1-rc1

Traceability:

Test(s):
Requirements:

Current Behavior: When the processing S3-SL2-NRT processing ends with NOK status, the section missing_output is empty.

Expected Behavior: Each NOK job processing shall have a trace with missing_output section filled.

Steps To Reproduce: Check S3-SL2-NRT processing traces.

Test execution artefacts (i.e. logs, screenshots…) Here are the trace NOK S3-SL2-NRT https://app.zenhub.com/files/398313496/58a9fab4-98a7-478e-b598-e124323dc872/download

Whenever possible, first analysis of the root cause Hypothesis :

Bug Generic Definition of Ready (DoR)

[X] The affect version in which the bug has been found is mentioned
[X] The context and environment of the bug is detailed
[X] The description of the bug is clear and unambiguous
[X] The procedure (steps) to reproduce the bug is clearly detailed
[ ] The tested User Story / features is linked to the bug if available
[X] Logs are attached if available
[ ] A data set attached if available

Bug Generic Definition of Done (DoD)

[ ] the modification implemented (the solution to fix the bug) is described in the bug.
[ ] Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
[ ] Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
[ ] Code is compliant with coding rules (SONAR Report as evidence)
[ ] Acceptance criteria of the related User story are checked and Passed

w-jka commented 1 year ago

@pcuq-ads The ticket description speaks of NRT but the traces you provided are from a NTC chain. The config and implementation of the NRT chain looks good, however for the NTC chain the following parameter is wrongly set to false:

# Output Product Estimation 
app.execution-worker.process.productTypeEstimationEnabled=false

Please set this value to true to activate the content of the missing_output section.

LAQU156 commented 1 year ago

IVV_CCB_2023_w26 : Moved into "Accepted OPS" for checking. Can be moved into "Accepted Werum" after control. Priority major, to be fixed phase 1

pcuq-ads commented 1 year ago

Dear @w-jka

The problem I see is on S3-SL2-NRT but for version 1.13.1 (sorry for the wrong traces on the description). Here is the problem:

{
  "header": {
    "type": "REPORT",
    "timestamp": "2023-06-19T17:20:06.775000Z",
    "level": "ERROR",
    "mission": "S3",
    "workflow": "NOMINAL",
    "rs_chain_name": "S3-SL2-NRT",
    "rs_chain_version": "1.13.1-rc1"
  },
  "message": {
    "content": "[code 290] [exitCode 255] [msg Task /usr/local/components/S3IPF_SL2_06.21/bin/SL2.bin failed]"
  },
  "task": {
    "uid": "19c47c90-de0d-418a-a65c-5ec52e5e9d31",
    "name": "JobProcessing",
    "event": "END",
    "status": "NOK",
    "output": {},
    "input": {
      "job_order_id_string": "JobOrder.99851.xml",
      "ipf_release_string": "06.21",
      "filename_strings": [
        "S3__AX___MF1_AX_20230618T000000_20230618T120000_20230617T061202___________________ECW_O_NR_001.SEN3",
        "S3A_SL_1_IRE_AX_20160216T000000_20991231T235959_20180202T120000___________________MPC_O_AL_001.SEN3",
        "S3__SL_2_LSTVAX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_1_LCC_AX_20160216T000000_20991231T235959_20201015T120000___________________MPC_O_AL_002.SEN3",
        "S3A_AX___FRO_AX_20230616T000000_20230626T000000_20230619T065330___________________EUM_O_AL_001.SEN3",
        "S3A_SL_2_N3_CAX_20160216T000000_20991231T235959_20170116T120000___________________MPC_O_AL_003.SEN3",
        "S3__AX___MF1_AX_20230617T060000_20230617T180000_20230616T180656___________________ECW_O_NR_001.SEN3",
        "S3A_SL_2_D2_CAX_20160216T000000_20991231T235959_20170116T120000___________________MPC_O_AL_003.SEN3",
        "S3__AX___MF1_AX_20230617T180000_20230618T060000_20230617T175613___________________ECW_O_NR_001.SEN3",
        "S3A_SL_2_S7N_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_2_S7O_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3__SL_2_LSTBAX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_2_LSTEAX_20160216T000000_20991231T235959_20210413T120000___________________MPC_O_AL_003.SEN3",
        "S3__AX___MF1_AX_20230617T120000_20230618T000000_20230617T060115___________________ECW_O_NR_001.SEN3",
        "S3A_SL_2_SDI2AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3__SL_2_SSTAAX_20220212T120000_99991231T235959_20220213T064539___________________MET_O_NN_001.SEN3",
        "S3A_SL_1_N_S9AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3",
        "S3A_SL_1_RTT_AX_20160216T000000_20991231T235959_20211201T120000___________________MPC_O_AL_002.SEN3",
        "S3__AX___MF1_AX_20230617T180000_20230618T060000_20230617T060041___________________ECW_O_NR_001.SEN3",
        "S3A_SL_2_SDI3AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3__SL_2_LSTWAX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_2_LSTCAX_20160216T000000_20991231T235959_20190215T120000___________________MPC_O_AL_003.SEN3",
        "S3A_SL_2_SSESAX_20000101T000000_20991231T235959_20160721T120000___________________MPC_O_AL_002.SEN3",
        "S3__AX___MF1_AX_20230617T060000_20230617T180000_20230617T060117___________________ECW_O_NR_001.SEN3",
        "S3__SL_2_SST_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_2_N3RCAX_20160216T000000_20991231T235959_20170116T120000___________________MPC_O_AL_003.SEN3",
        "S3A_SL_1_N_S8AX_20160216T000000_20991231T235959_20170324T120000___________________MPC_O_AL_006.SEN3",
        "S3A_SL_2_S8N_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3__AX___MF1_AX_20230617T120000_20230618T000000_20230616T180654___________________ECW_O_NR_001.SEN3",
        "S3__SL_2_DIMSAX_20230618T000000_20230618T235959_20230619T131149___________________LND_O_AL_001.SEN3",
        "S3A_SL_2_S9O_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_2_D3_CAX_20160216T000000_20991231T235959_20170116T120000___________________MPC_O_AL_003.SEN3",
        "S3A_AX___OSF_AX_20160216T192404_99991231T235959_20220330T090651___________________EUM_O_AL_001.SEN3",
        "S3A_SL_1_RBT____20230617T221351_20230617T221851_20230619T150112_0299_100_100______LN3_D_NR_002.SEN3",
        "S3A_SL_2_F1N_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_2_N2_CAX_20160216T000000_20991231T235959_20170116T120000___________________MPC_O_AL_003.SEN3",
        "S3A_SL_2_S8O_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3__SL_2_IMSCAX_20160216T000000_20991231T235959_20210413T120000___________________MPC_O_AL_001.SEN3",
        "S3A_SL_2_PCP_AX_20160216T000000_20991231T235959_20211201T120000___________________MPC_O_AL_007.SEN3",
        "S3A_SL_2_S9N_AX_20000101T000000_20991231T235959_20151214T120000___________________MPC_O_AL_001.SEN3",
        "S3__AX___MF1_AX_20230618T000000_20230618T120000_20230617T175612___________________ECW_O_NR_001.SEN3",
        "JobOrder.99851.xml"
      ]
    },
    "quality": {},
    "error_code": 1,
    "duration_in_seconds": 445.631,
    "missing_output": []
  }
}

The fix has been applied on version 1.13.2 . I can not conclude yet because I do not have processing NOK with version 1.13.2.

I propose to wait two weeks (15/7) before closing the issue.

pcuq-ads commented 1 year ago

SYS_CCB_w27 : No more observation since 21 days. The ticket can be closed.

COPRS / rs-issues

[BUG] S3-SL2-NRT, no missing output section on the trace for a processing NOK. #1011