search

stcorp / procsim

Tool to simulate satellite data processor tasks
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

Times must be set before creating MPH #7

Closed lmar76 closed 3 years ago

lmar76 commented 3 years ago

I tried to generate S1_RAWP_0M ptoducts, using the following scenario:

{
    "mission": "biomass",
    "scenarios": [
        {
            "name": "TOM_TDS - SM - DTID 8282 - L0PFS3 inputs",
            "file_name": "N/A",
            "processor_name": "procsim",
            "processor_version": "1.2.0",
            "task_name": "N/A",
            "task_version": "N/A",
            "log_level": "debug",
            "anx": [
                "2017-01-01T06:00:01.272000Z",
                "2017-01-01T07:38:12.217000Z"
            ],
            "data_takes": [
                {
                    "data_take_id": "8282",
                    "start": "2017-01-01T06:01:31.394000Z",
                    "stop": "2017-01-01T06:03:44.504000Z",
                    "swath": "S1",
                    "operational_mode": "SM"
                }
            ],
            "mission_phase": "Tomographic",
            "global_coverage_id": "01",
            "major_cycle_id": "01",
            "repeat_cycle_id": "01",
            "track_nr": "1",                      
            "begin_position": "2017-01-01T06:01:31.394Z",
            "end_position": "2017-01-01T06:03:44.504Z",
            "baseline": 1,
            "outputs": [
              {
                "type": "S1_RAWP_0M"
              }
            ]      
        }
    ]
}

I have executed procsim:

$ /opt/pnode/bin/procsim -s "TOM_TDS - SM - DTID 8282 - L0PFS3 inputs" /opt/pnode/etc/l0-inputs-generation.json
2021-08-31T17:38:50.326690 N/A    [0000012816]: [I] Procsim v1.2.0 processor stub simulator
2021-08-31T17:38:50.326727 N/A    [0000012816]: [I] Simulate scenario TOM_TDS - SM - DTID 8282 - L0PFS3 inputs
2021-08-31T17:38:50.334920 N/A    [0000012816]: [D] Set baseline to 1
2021-08-31T17:38:50.336864 N/A    [0000012816]: [D] Set begin_position to 2017-01-01 06:01:31.394000
2021-08-31T17:38:50.336911 N/A    [0000012816]: [D] Set end_position to 2017-01-01 06:03:44.504000
2021-08-31T17:38:50.336925 N/A    [0000012816]: [D] Set mission_phase to Tomographic
2021-08-31T17:38:50.336935 N/A    [0000012816]: [D] Overwrite global_coverage_id from NA to 01
2021-08-31T17:38:50.336944 N/A    [0000012816]: [D] Overwrite major_cycle_id from 1 to 01
2021-08-31T17:38:50.336951 N/A    [0000012816]: [D] Overwrite repeat_cycle_id from __ to 01
2021-08-31T17:38:50.336959 N/A    [0000012816]: [D] Overwrite track_nr from 0 to 1
2021-08-31T17:38:50.336984 N/A    [0000012816]: [D] Set creation_date to 2017-01-01 08:03:45
2021-08-31T17:38:50.337058 N/A    [0000012816]: [D] Output directory is /opt/pnode/data/WD-TOM_TDS-SM-DTID8202-L0PFS3
2021-08-31T17:38:50.337106 N/A    [0000012816]: [D] Set swath to S1
2021-08-31T17:38:50.337118 N/A    [0000012816]: [D] Set operational_mode to SM
2021-08-31T17:38:50.337127 N/A    [0000012816]: [D] Set data_take_id to 8282
2021-08-31T17:38:50.337139 N/A    [0000012816]: [D] Datatake 8282 from 2017-01-01 06:01:31.394000 to 2017-01-01 06:03:44.504000
2021-08-31T17:38:50.337154 N/A    [0000012816]: [D] Incomplete slice, start and end unaligned
2021-08-31T17:38:50.337208 N/A    [0000012816]: [I] Create BIO_S1_RAWP_0M_20170101T060131_20170101T060344_T_G01_M01_C01_T001_F____01_8VGME9
2021-08-31T17:38:50.337361 N/A    [0000012816]: [E] Times must be set before creating MPH
2021-08-31T17:38:50.337374 N/A    [0000012816]: [I] Terminate with code 1

The execution terminates with the following error messages:

2021-08-31T17:38:50.337361 N/A    [0000012816]: [E] Times must be set before creating MPH
2021-08-31T17:38:50.337374 N/A    [0000012816]: [I] Terminate with code 1
pe1obw commented 3 years ago

The problem is that the Validity start/stop times are not known. I reworked the error message to be more precise here. ‘Normally’ the validity times are copied from some input product, but I understand you want to generate Sx_RAWP product from scratch. However, the SxRAWP product generator is not responsible for slicing, so it needs the validity times.

I see a few solutions for this:

  1. We add parameters to set/override the validity times as well. Seems the most straightforward solution, but requires you to add these to the scenario as well.

  2. We could calculate validity times, if not available. This could be done using the slicing mechanism as currently used in the RAWSxxx10 generator. However, I'm not sure it makes sense that the S1 product generator performs slicing?

  3. We can first generate a (dummy) RAWSxxx_10 product and use that as input for this generator. The RAWSxxx generator has the slicer logic on board and requires only begin/end_positions. In the current version, it is not possible to directly generate RAWSxxx products, due to missing begin/end_position parameters for these products. However, these can be added simply and I'll do that anyway. However, another issue is that the ‘metadata_source’ currently must be one of the input files, in the job order, and I assume you don't want to create a job order! This could be solved by just trying all matching files in the output directory, if no matching input file is specified, or if no job order exists.

lmar76 commented 3 years ago

Why the validity start/stop times are not known? The scenario includes the "begin_position" and "end_position" which should specify the validity start/stop times...

pe1obw commented 3 years ago

Perhaps there is a misunderstanding about these times? The current implementation is as follows:

The scenario parameters begin_position and end_position are used to set the Main Product Header fields ‘bio:EarthObservation/om:phenomenonTime/gml:TimePeriod/gml:beginPosition’ and ‘bio:EarthObservation/om:phenomenonTime/gml:TimePeriod/gml:endPosition’, respective. They also map to the ‘UTC start date and time’ and ‘UTC stop date and time’ parts of the product name. The ‘validity times’ in Main Product Header fields bio:EarthObservation/om:validTime/gml:TimePeriod/gml:beginPosition and bio:EarthObservation/om:validTime/gml:TimePeriod/gml:endPosition are equal to the begin/end position for most product types.

However, for RAWS_ and L0 products, the validity times represent the slice start/end time, and for L1 products they represent the frame start/end time. As such, they should be either calculated by the slicer/framer, or read from the Main Product Header from another product, to be specified using the metadata_source parameter.

lmar76 commented 3 years ago

My apologies, I have misunderstood your explanation.

In my opinion:

Can you identify any issue on this approach? Do you agree?

Regarding the ability to read metadata from MPH of input product(s), this could cause some issues if, for a given product type, there are more input products (e.g. level 0 step 2). In this case, it seems that Procsim reads the MPH of only one input product and this could cause issues. Maybe, in this case we should disable this procedure and use only the information provided via begin_position/end_position in the scenario file or Job Order TOI. Do you agree?

I thought that it is always (i.e. for each file type) possible to generate products "from scratch", i.e. without providing a JobOrder and input files, but providing the needed parameters in the scenario file. If not, which are the products that can be generated in this way?

pe1obw commented 3 years ago

Ah, you're completely right. The idea was, and is, to generate products from scratch, without (the need of) reading any input product.

So, I fully agree, and see no issues. I modified the code to copy the phenomenon times to the validity times, if not known. These times are then input to the product generator.

To test this, I added a test script (test/biomass/test_generate_all.sh) to generate every Biomass product type, without any input file. They are all generated, although the L1 SCS/DGM generator is a bit critical, as it requires the slice length to be 5 times the frame length (also taking the start/end durations into account).