Create S3_SY2_VGS_NTC processor as RS add-on

Woljtek commented 1 year ago

User story description: As a COPRS integrator, I need to implement S3 SYN2_VGS_NTC processor in SCDF workflow, so I can deploy S3_SYN_VGS_NTC RS-add-on.

The name of the RS-Add-on has to be and has to be include in processing-sentinel-3 repository. This US might be block by: https://pforge-exchange2.astrium.eads.net/jira/browse/DIAPA-645 "Sentinel-3 images are deprecated and unusable as is"

The processor S3_SYN2_VGK produces 1 internal product and 4 end user products:

SY_1_MISR__ (internal)
SY_2_VGP___
SY_2_VGK_ (also used as input to produce SY_2VG1 & SY_2_V10___)
SY_2_VG1___
SY_2_V10___

Definition of Ready

[ ] Get grant access to RS Sentinel Data Processors Publishing service (Function 1)
[ ] RS add-on S3_SL1 is delivered
[ ] RS add-on S3_OL1 is delivered
[ ] S3IPF.ICD.005 Interface Control Document - SYNERGY Level 1 & 2 is avaiblale in F1(V1.5 or newer)
[ ] SYNERGY Level 1 & 2 CFI is available in F1 (SY2_VGP.bin - 06.11)
[ ] SYNERGY VGT-S IPF is available in F1

Acceptance criteria:

[ ] Processing chain delivered as ‘RS add-on’ compliant with processing COPRS-ICD-ADST-001133963 - ICD RS addon.
[ ] Messages format are compliant to COPRS-ICD-ADST-001144532 - ICD Processing Message Format.
[ ] Traces are compliant to COPRS-ICD-ADST-001048444 - ICD log
[ ] All properties needed to specialize the ‘RS add-on’ are defined on ‘stream-parameters.properties’ template file.
[ ] The routing of message at the end of processing is configurable on ‘stream-parameters.properties’ file.
[ ] When a processing failed, automatic retry is performed a number of times (the number of time is configurable on ‘stream-parameters.properties’ file).
[ ] Remove the priority filter.
[ ] Example of configuration for NTC is provided in documentation.
[ ] A configurable dead letter queue is managed.
[ ] The SYN L1 output, SY_1_MISR__, is an internal product
[ ] A trace for SYN L2 1 km VEGETATION-Like product (~VGT-P) - TOA Reflectance (SY_2_VGP__) product is provided as test result
[ ] A trace for SYN L2 Surface reflectance over Land– used as input for VG-S product(SY_2_VGK__) product is provided as test result
[ ] A trace for SYN L2 (L3) 1 km VEGETATION-Like product (~VGT-S10) 10 day synthesis surface reflectance and NDVI (SY_2_VG10__) product is provided as test result
[ ] A trace for SYN L2 (L3) 1 km VEGETATION-Like product (~VGT-S1) 1 day synthesis surface reflectance and NDVI (SY_2_V1__) product is provided as test result

Definition of Done:

[ ] All acceptance criteria are fulfilled.
[ ] 'RS add-on' is delivered in RS Sentinel Data Processors Publishing service (Function 1).
[ ] Containers from the 'RS add-on' are generated from Github and pushed to Artifactory (Function 2).
[ ] All templates of RS-Add-on properties files are available on Github repository (Function 2).
[ ] All custom properties from ‘stream-parameters.properties’ are documented on Github (Function 2).

Woljtek commented 1 year ago

Hi @w-fsi This US includes both STC et NTC, do you want to split up into two separated US?

Create S3_SY2_VGS_STC processor as RS add-on
Create S3_SY2_VGS_NTC processor as RS add-on

(Same question for #581 & #585)

w-fsi commented 1 year ago

@Woljtek : Yes, I would vote for it. Especially as the Synergy are even more different to each other than in other workflows.

Woljtek commented 1 year ago

Updated to handle only NTC timeliness.

Woljtek commented 1 year ago

Please find the image here: https://artifactory.coprs.esa-copernicus.eu:443/artifactory/cfi/processors/sentinel3/s3-sy2-vgs/06.11

w-jka commented 1 year ago

During this story the following actions were performed:

Implementation of a new S3SynergyTypeAdapter to handle the specifics to create SY2 VG1 and SY2 V10 jobs
Extension of the PDUTypeAdapter to handle TILE-PDUs which are only used by VG1 and V10 products
Extension of cronbased-trigger application in order to be able to handle offsets of the ingestion time. This is necessary in order to trigger the SYNERGY vegetation workflows after all necessary SYNERGY products are in the system
documentation of the new TypeAdapter as well as configuration changes for the performed extensions
creation of the RS-Addon "s3-sy2-vgs-ntc"
creation of the RS-Addon "s3-pug-ntc"

w-jka commented 1 year ago

END trace for a production of VG1 product:

{
    "header": {
        "type": "REPORT",
        "timestamp": "2022-12-12T15:14:30.949000Z",
        "level": "INFO",
        "mission": "S3",
        "workflow": "NOMINAL",
        "rs_chain_name": "S3-SY2-VGS-NTC",
        "rs_chain_version": "develop"
    },
    "message": {
        "content": "End job processing"
    },
    "task": {
        "uid": "9115e4e2-6754-4ff0-9f42-058a774fabf7",
        "name": "JobProcessing",
        "event": "END",
        "status": "OK",
        "output": {
            "debug": false,
            "t0_pdgs_date": "2022-12-12T14:54:00.000000Z",
            "filename_string": "S3A_SY_2_VG1____20040802T000000_20040802T235900_20221212T151428_DDDD_001_429_FFFF_WER_D_NT_NNN.SEN3"
        },
        "input": {
            "job_order_id_string": "JobOrder.22155.xml",
            "ipf_release_string": "06.11",
            "filename_strings": [
                "S3A_SY_2_VGK____20040802T005415_20040802T014015_20221207T160324_DDDD_001_430_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T023515_20040802T032115_20221207T160324_DDDD_001_431_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T041615_20040802T050215_20221207T160324_DDDD_001_432_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T055715_20040802T064315_20221207T160324_DDDD_001_433_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T073815_20040802T082415_20221207T160324_DDDD_001_434_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T091915_20040802T100515_20221207T160324_DDDD_001_435_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T110015_20040802T114615_20221207T160324_DDDD_001_436_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T124115_20040802T132715_20221207T160324_DDDD_001_437_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T142215_20040802T150815_20221207T160324_DDDD_001_438_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T160315_20040802T164915_20221207T160324_DDDD_001_439_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T174415_20040802T183015_20221207T160324_DDDD_001_440_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T192515_20040802T201115_20221207T160324_DDDD_001_441_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T210615_20040802T215215_20221207T160324_DDDD_001_442_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VGK____20040802T224715_20040802T233315_20221207T160324_DDDD_001_443_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_AX___FRO_AX_20040702T223000_20991231T235959_20171130T082116___________________WER_D_AL____.SEN3",
                "S3A_AX___OSF_AX_20040702T203000_20991231T235959_20171130T082119___________________WER_D_AL____.SEN3",
                "S3A_SY_2_PCPSAX_20040702T223000_20991231T235959_20171130T082119___________________WER_D_AL____.SEN3",
                "JobOrder.22155.xml"
            ]
        },
        "quality": {},
        "error_code": 0,
        "duration_in_seconds": 7.12,
        "missing_output": []
    }
}

w-jka commented 1 year ago

END trace for a V10 product:

{
    "header": {
        "type": "REPORT",
        "timestamp": "2022-12-12T15:32:47.839000Z",
        "level": "INFO",
        "mission": "S3",
        "workflow": "NOMINAL",
        "rs_chain_name": "S3-SY2-VGS-NTC",
        "rs_chain_version": "develop"
    },
    "message": {
        "content": "End job processing"
    },
    "task": {
        "uid": "fdbd6de2-204f-483c-b2ff-238b9d2d219c",
        "name": "JobProcessing",
        "event": "END",
        "status": "OK",
        "output": {
            "debug": false,
            "t0_pdgs_date": "2022-12-12T14:54:00.000000Z",
            "filename_string": "S3A_SY_2_V10____20040711T000000_20040720T235900_20221212T153245_DDDD_001_116_FFFF_WER_D_NT_NNN.SEN3"
        },
        "input": {
            "job_order_id_string": "JobOrder.22175.xml",
            "ipf_release_string": "06.11",
            "filename_strings": [
                "S3A_SY_2_VG1____20040711T000000_20040711T235900_20221212T151146_DDDD_001_116_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040712T000000_20040712T235900_20221212T151153_DDDD_001_130_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040713T000000_20040713T235900_20221212T151201_DDDD_001_144_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040714T000000_20040714T235900_20221212T151208_DDDD_001_158_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040715T000000_20040715T235900_20221212T151215_DDDD_001_173_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040716T000000_20040716T235900_20221212T151222_DDDD_001_187_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040717T000000_20040717T235900_20221212T151230_DDDD_001_201_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040718T000000_20040718T235900_20221212T151237_DDDD_001_215_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040719T000000_20040719T235900_20221212T151245_DDDD_001_230_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_SY_2_VG1____20040720T000000_20040720T235900_20221212T151252_DDDD_001_244_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_AX___FRO_AX_20040702T223000_20991231T235959_20171130T082116___________________WER_D_AL____.SEN3",
                "S3A_AX___OSF_AX_20040702T203000_20991231T235959_20171130T082119___________________WER_D_AL____.SEN3",
                "S3A_SY_2_PCPSAX_20040702T223000_20991231T235959_20171130T082119___________________WER_D_AL____.SEN3",
                "JobOrder.22175.xml"
            ]
        },
        "quality": {},
        "error_code": 0,
        "duration_in_seconds": 7.328,
        "missing_output": []
    }
}

w-jka commented 1 year ago

END trace for PUG ntc production for an V10 product:

{
    "header": {
        "type": "REPORT",
        "timestamp": "2022-12-12T15:34:13.766000Z",
        "level": "INFO",
        "mission": "S3",
        "workflow": "NOMINAL",
        "rs_chain_name": "S3-PUG-NTC",
        "rs_chain_version": "develop"
    },
    "message": {
        "content": "End job processing"
    },
    "task": {
        "uid": "97dd61fa-da92-4f03-8abc-febc319987f9",
        "name": "JobProcessing",
        "event": "END",
        "status": "OK",
        "output": {
            "debug": false,
            "t0_pdgs_date": "2022-12-12T14:54:00.000000Z",
            "filename_strings": [
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_AFRICA____________WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_NORTH_AMERICA_____WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_SOUTH_AMERICA_____WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_CENTRAL_AMERICA___WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_NORTH_ASIA________WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_WEST_ASIA_________WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_SOUTH_EAST_ASIA___WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_ASIAN_ISLANDS_____WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_AUSTRALASIA_______WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153353_EUROPE____________WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_AFRICA____________WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_NORTH_AMERICA_____WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_SOUTH_AMERICA_____WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_CENTRAL_AMERICA___WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_NORTH_ASIA________WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_WEST_ASIA_________WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_SOUTH_EAST_ASIA___WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_ASIAN_ISLANDS_____WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_AUSTRALASIA_______WER_D_NT_NNN.SEN3",
                "S3A_SY_2_V10_BW_20040721T000000_20040731T235900_20221212T153353_EUROPE____________WER_D_NT_NNN.SEN3"
            ]
        },
        "input": {
            "job_order_id_string": "JobOrder.22178.xml",
            "ipf_release_string": "03.45",
            "filename_strings": [
                "S3A_SY_2_V10____20040721T000000_20040731T235900_20221212T153345_DDDD_001_258_FFFF_WER_D_NT_NNN.SEN3",
                "S3A_AX___FRO_AX_20040702T223000_20991231T235959_20171130T082116___________________WER_D_AL____.SEN3",
                "LOG",
                "S3A_SY_2_CVSBAX_20040702T223000_20991231T235959_20171130T082118___________________WER_D_AL____.SEN3",
                "S3A_SY_2_PVSBAX_20040702T223000_20991231T235959_20171130T082118___________________WER_D_AL____.SEN3",
                "JobOrder.22178.xml"
            ]
        },
        "quality": {},
        "error_code": 0,
        "duration_in_seconds": 25.358,
        "missing_output": []
    }
}

w-fsi commented 1 year ago

Werum_CCB_2023_w30: Done.

COPRS / rs-issues

Create S3_SY2_VGS_NTC processor as RS add-on #583