search

COPRS / rs-issues

This repository contains all the issues of the COPRS project (Scrum tickets, ivv bugs, epics ...)
2 stars 2 forks source link

[BUG] [METADATA EXTRACTION] Metadata extraction do not provide information for some S3 products. #1017

Closed pcuq-ads closed 1 year ago

pcuq-ads commented 1 year ago

Environment:

Traceability:

Current Behavior: Trace from Metadata Extraction chain does not provide metadata for some S3 products.

Expected Behavior: Metadata Extraction chain shall provide metadata for all products.

Steps To Reproduce: Check trace for following products:

Test execution artefacts (i.e. logs, screenshots…) Here is a trace from metadata extraction for product S3B_SR_0_SRA__20230624T212407_20230624T221337_20230625T002506_2970_081_057____LN3_O_NR_002.SEN3 We can see that the field product_metadata_custom_object is empty

{
    "header": {
      "type": "REPORT",
      "timestamp": "2023-06-25T00:26:09.481000Z",
      "level": "INFO",
      "mission": "S3",
      "workflow": "NOMINAL",
      "rs_chain_name": "METADATA",
      "rs_chain_version": "1.13.2-rc1"
    },
    "message": {
      "content": "End metadata extraction"
    },
    "task": {
      "uid": "bd8ab986-a50c-4623-be57-aedeef3bf7f8",
      "name": "MetadataExtraction",
      "event": "END",
      "status": "OK",
      "output": {
        "mission_identifier_string": "S3",
        "type_string": "S3_PUG",
        "product_sensing_start_date": "2023-06-24T21:24:06.929216Z",
        "product_sensing_stop_date": "2023-06-24T22:13:37.322709Z",
        "product_metadata_custom_object": {},
        "end_to_end_product_boolean": true
      },
      "input": {
        "filename_string": "S3B_SR_0_SRA____20230624T212407_20230624T221337_20230625T002506_2970_081_057______LN3_O_NR_002.SEN3"
      },
      "quality": {},
      "error_code": 0,
      "duration_in_seconds": 0.727,
      "missing_output": []
    }

Whenever possible, first analysis of the root cause Hypothesis : XXX

Bug Generic Definition of Ready (DoR)

Bug Generic Definition of Done (DoD)

nleconte-csgroup commented 1 year ago

At first glance, product is in the table product :

monitoring=# select * from product where filename = 'S3B_SR_0_SRA____20230624T212407_20230624T221337_20230625T002506_2970_081_057______LN3_O_NR_002.SEN3';
-[ RECORD 1 ]--------------+----------------------------------------------------------------------------------------------------
id                         | 588477
filename                   | S3B_SR_0_SRA____20230624T212407_20230624T221337_20230625T002506_2970_081_057______LN3_O_NR_002.SEN3
custom                     | {}
timeliness_name            | 
timeliness_value_seconds   | 0
end_to_end_product         | t
t0_pdgs_date               | 2023-06-24 22:22:20
prip_storage_end_date      | 2023-06-25 00:28:08.763
generation_begin_date      | 2023-06-25 00:23:37.695
generation_end_date        | 2023-06-25 00:26:08.746
catalog_storage_begin_date | 2023-06-25 00:26:08.754
catalog_storage_end_date   | 2023-06-25 00:26:09.481
prip_storage_begin_date    | 2023-06-25 00:28:08.583
quality_check_begin_date   | 
quality_check_end_date     | 
first_download_date        | 
eviction_date              |
pcuq-ads commented 1 year ago

The issue is about field "custom". It is empty on the extract provided.

nleconte-csgroup commented 1 year ago

The field custom is filled with the information in the metadata extraction's trace : https://github.com/COPRS/monitoring/blob/76fb79d0b9744d647636c31adab8a05f4905127d/rs-cores/MONITORING/Executables/additional_resources/ConfigMap-ingestor.yaml#L131

And the field is empty in the trace : image

The issue seems to be on the TRACE produced by the metadata extraction component instead of the monitoring.

pcuq-ads commented 1 year ago

On the example provided with the anomaly, you cans see that there are some metadata associated to the product : "S3B_SR_0_SRA__20230624T212407_20230624T221337_20230625T002506_2970_081_057____LN3_O_NR_002" inside the trace image.png

and we can not see the metadata inside the product table.

Some metadata are well retrieved but not all of them. This is the issue.

nleconte-csgroup commented 1 year ago

@pcuq-ads

The monitoring component is extracting everything that is configured and available in the TRACE : image

As you can see, the following fields are either missing or empty :

pcuq-ads commented 1 year ago

Ok. You are right. I thought at first that the full block was part of metadata.

pcuq-ads commented 1 year ago

I propose to update the title of this issue and assign the issue to WERUM.

LAQU156 commented 1 year ago

IVV_CCB_2023_w27 : Moved into "Accepted Werum" for correction, Priority major, to be fixed phase 1

LAQU156 commented 1 year ago

Werum_CCB_2023_w27 : Moved into "Product Backlog" for correction (1.14.0-rc2)

vgava-ads commented 1 year ago

Delivered in Production Common v1.14.0 (refer to https://github.com/COPRS/production-common/releases)

pcuq-ads commented 1 year ago

SYS_CCB_w29 : Test to be performed after deployment of version 1.14 for both RS CORE Metadata Extraction and Metatada Search Controler. Easy check after 1 week of production.

pcuq-ads commented 1 year ago

SYS_CCB_w30 : This issue is fixed with release Production Common V1.14.0. image.png