Possibility to use 2 data sets using Sen2Three

japoveda commented 6 years ago

We need 2 data sets as input for change detection, using the Sen2Three as prior step there is not way to get two datasets. We tried two options:

Option 1: Run twice the same service in on Workflow It seems that the system does not like to run twice the same service in one workflow. We are having this error: https://portal.eo4a.science/workspace/execute/782

Option 2: Select 2 tile in Copernicus services If we select 2 tiles in Copernicus service and run Sen2Three, only one of the tiles is analysed by Sen2Three and the second one is omitted.

derekocallaghan commented 6 years ago

Hi Alex,

Option 1: the issue here is that the workflow only supports a single starting task node, in this case (https://portal.eo4a.science/workflows/782), you have two. One possibility is to chain the two separate branches into a single branch with four task nodes. However, in this case, I would have a single Copernicus task that has two product ids, followed by a single Sen2Three task.
Option 2: Is this the same issue as https://github.com/parameterspace-ie/eo4a-service-trials/issues/5? If not, can you specify the workflow job URL, and workflow URL.

Derek

derekocallaghan commented 6 years ago

In addition to my questions in the last comment, I see that although there are two Copernicus services in the workflow definition at https://portal.eo4a.science/workflows/782, both of them are configured to retrieve the same product id S2A_MSIL2A_20170504T115401_N0205_R023_T29UNA_20170504T115402:86ac371c-cb17-4047-8106-5f15f6298054. If you click on JSON request representation (or edit the workflow), you'll see the text below:

{
   6c46132fa9fd4504855beb1e03465ab2: {
      wps_id: "resource-management.data:scihub",
      wps_server_name: "EO4Atlantic Pathfinder",
      uuid: "6c46132fa9fd4504855beb1e03465ab2",
      product_ids: "S2A_MSIL2A_20170504T115401_N0205_R023_T29UNA_20170504T115402:86ac371c-cb17-4047-8106-5f15f6298054"
   },
   0d1a34b2148444759762a43f9839d524: {
      wps_id: "correction.sen2three",
      wps_server_name: "EO4Atlantic Pathfinder",
      uuid: "0d1a34b2148444759762a43f9839d524",
      s2_product_dir: "{reference|6c46132fa9fd4504855beb1e03465ab2|output_dir}",
      resolution: ""
   },
   f0d1d7b2f02c4f46be2cf85e5c46b91a: {
      wps_id: "resource-management.data:scihub",
      wps_server_name: "EO4Atlantic Pathfinder",
      uuid: "f0d1d7b2f02c4f46be2cf85e5c46b91a",
      product_ids: "S2A_MSIL2A_20170504T115401_N0205_R023_T29UNA_20170504T115402:86ac371c-cb17-4047-8106-5f15f6298054"
   },
   350b670199e74a48bee9e17fd2f91653: {
      wps_id: "correction.sen2three",
      wps_server_name: "EO4Atlantic Pathfinder",
      uuid: "350b670199e74a48bee9e17fd2f91653",
      s2_product_dir: "{reference|f0d1d7b2f02c4f46be2cf85e5c46b91a|output_dir}",
      resolution: ""
   }
}

japoveda commented 6 years ago

Thanks Derek,

Cloud you check the workflow and job bellow? Even though we are selecting 2 product IDs in the Copernicus service and both of them are downloaded in the output directory. Sen2Three only process one of them

Workflow https://portal.eo4a.science/workflows/754

Job https://portal.eo4a.science/workspace/exec/856

derekocallaghan commented 6 years ago

Hi Alex,

If you go to the job at https://portal.eo4a.science/workspace/exec/856, then click the "Download service task data from the following environments:" EO4Atlantic Pathfinder button, and navigate to correction.sen2three/output/sen2three/, there is a single Sen2Three output product: S2A_MSIL03_20170717T113321_N0205_R080_T29UNU_20160101T000000.zip. Both input products were for T29UNU so producing a single "merged" product is the expected Sen2Three behavior.

Viewing the correction.sen2three/service.log file also confirms that Sen2Three has processed both products:

...
unzip /opt/eo4a/data/workflow/resource-management.data:scihub/output/scihub/S2A_MSIL2A_20170717T113321_N0205_R080_T29UNU_20170717T113326.zip
...
unzip /opt/eo4a/data/workflow/resource-management.data:scihub/output/scihub/S2A_MSIL2A_20171005T113321_N0205_R080_T29UNU_20171005T113316.zip
...
Procedure L3_Tables: start import, elapsed time[s]: 4.116
Progress[%]: 0.34 : Procedure L3_Tables: Level L3 band B02 imported, elapsed time[s]: 4.081
Progress[%]: 0.68 : Procedure L3_Tables: Level L3 band B03 imported, elapsed time[s]: 4.046
Progress[%]: 1.02 : Procedure L3_Tables: Level L3 band B04 imported, elapsed time[s]: 4.145
Progress[%]: 1.37 : Procedure L3_Tables: Level L3 band B05 imported, elapsed time[s]: 4.126
Progress[%]: 1.71 : Procedure L3_Tables: Level L3 band B06 imported, elapsed time[s]: 4.406
Progress[%]: 2.08 : Procedure L3_Tables: Level L3 band B07 imported, elapsed time[s]: 4.504
Progress[%]: 2.45 : Procedure L3_Tables: Level L3 band B08 imported, elapsed time[s]: 5.925
Progress[%]: 2.95 : Procedure L3_Tables: Level L3 band B8A imported, elapsed time[s]: 4.464
Progress[%]: 3.32 : Procedure L3_Tables: Level L3 band B11 imported, elapsed time[s]: 4.112
Progress[%]: 3.66 : Procedure L3_Tables: Level L3 band B12 imported, elapsed time[s]: 3.992
Progress[%]: 3.99 : Procedure L3_Tables: Level L3 band AOT imported, elapsed time[s]: 1.950
Progress[%]: 4.16 : Procedure L3_Tables: Level L3 band SCL imported, elapsed time[s]: 1.748
Progress[%]: 4.30 : Procedure L3_Tables: start import, elapsed time[s]: 0.012
Progress[%]: 4.30 : Procedure L3_Tables: Level L2A band B02 imported, elapsed time[s]: 4.087
Progress[%]: 4.64 : Procedure L3_Tables: Level L2A band B03 imported, elapsed time[s]: 4.155
Progress[%]: 4.99 : Procedure L3_Tables: Level L2A band B04 imported, elapsed time[s]: 4.115
Progress[%]: 5.33 : Procedure L3_Tables: Level L2A band B05 imported, elapsed time[s]: 4.146
Progress[%]: 5.68 : Procedure L3_Tables: Level L2A band B06 imported, elapsed time[s]: 4.325
Progress[%]: 6.04 : Procedure L3_Tables: Level L2A band B07 imported, elapsed time[s]: 4.367
Progress[%]: 6.40 : Procedure L3_Tables: Level L2A band B08 imported, elapsed time[s]: 5.937
Progress[%]: 6.90 : Procedure L3_Tables: Level L2A band B8A imported, elapsed time[s]: 4.438
Progress[%]: 7.27 : Procedure L3_Tables: Level L2A band B11 imported, elapsed time[s]: 4.081
Progress[%]: 7.61 : Procedure L3_Tables: Level L2A band B12 imported, elapsed time[s]: 3.993
Progress[%]: 7.94 : Procedure L3_Tables: Level L2A band AOT imported, elapsed time[s]: 1.919
Progress[%]: 8.10 : Procedure L3_Tables: Level L2A band SCL imported, elapsed time[s]: 1.936
Progress[%]: 8.26 : Procedure L3_Process: processing with resolution 20 m, elapsed time[s]: 0.007
Progress[%]: 8.26 : Procedure L3_Process: start of Pre Processing, elapsed time[s]: 0.000
Progress[%]: 8.26 : Procedure L3_Process: start of Spatio Temporal Processing, elapsed time[s]: 0.001
Progress[%]: 8.26 : Procedure L3_Synthesis: pre processing, elapsed time[s]: 0.000
Progress[%]: 8.26 : Procedure L3_Process: nr processed tiles: 1, elapsed time[s]: 0.000
Progress[%]: 8.26 : Procedure L3_Process: start of Post Processing, elapsed time[s]: 41.652
Progress[%]: 11.73 : Procedure L3_Tables: start export, elapsed time[s]: 0.001
Progress[%]: 11.73 : Procedure L3_Tables: band B02 exported, elapsed time[s]: 6.219
Progress[%]: 12.25 : Procedure L3_Tables: band B03 exported, elapsed time[s]: 6.303
Progress[%]: 12.78 : Procedure L3_Tables: band B04 exported, elapsed time[s]: 6.455
Progress[%]: 13.31 : Procedure L3_Tables: band B05 exported, elapsed time[s]: 6.440
Progress[%]: 13.85 : Procedure L3_Tables: band B06 exported, elapsed time[s]: 6.778
Progress[%]: 14.42 : Procedure L3_Tables: band B07 exported, elapsed time[s]: 6.901
Progress[%]: 14.99 : Procedure L3_Tables: band B08 exported, elapsed time[s]: 7.139
Progress[%]: 15.59 : Procedure L3_Tables: band B8A exported, elapsed time[s]: 6.932
Progress[%]: 16.16 : Procedure L3_Tables: band B11 exported, elapsed time[s]: 6.492
Progress[%]: 16.70 : Procedure L3_Tables: band B12 exported, elapsed time[s]: 6.382
Progress[%]: 17.24 : Procedure L3_Tables: band SCL exported, elapsed time[s]: 3.431
Progress[%]: 17.52 : Procedure L3_Tables: band MSC exported, elapsed time[s]: 3.285
Progress[%]: 17.80 : Procedure L3_Tables: band TCI exported, elapsed time[s]: 12.335
Progress[%]: 18.82 : Procedure L3_Tables: stop export, elapsed time[s]: 0.008
Progress[%]: 18.82 : Procedure L3_Synthesis: post processing, elapsed time[s]: 0.013
...

japoveda commented 6 years ago

Hi Derek,

This is the problem, to perform change detection we need 2 different datasets (same area different dates). We would like to use Sen2Three outputs because the cloud issue, so we need that the system can run twice Sen2Three with two different Copernicus task outputs (new and old) and use those 2 outputs as input of our service. Or an alternative solution.

Copernicus Old image > Sen2Three Old image output -> ........................................................................................................................... -> Sentinel change detection Copernicus New image > Sen2Three New image output ->

derekocallaghan commented 6 years ago

Hi Alex,

Sen2Three requires at least two level 2 Sentinel-2 products for a particular region (e.g. T29UNA) in order to generate a single level 3 product for the region (see http://step.esa.int/main/third-party-plugins-2/sen2three/ for more details and user manual). If only a single product is passed, it will just output it unchanged.

In your case, you are passing a single product for both time 1 and time 2 into corresponding subsequent Sen2Three tasks, so the output you're getting is expected.

At a minimum, you would need to do something like:

Task 1: Copernicus - retrieve two or more products for time 1 (e.g. in the same week)
Task 2: Sen2Three - use Task 1 outputs as input - this will generate a single output for ~time 1
Task 3: Copernicus - retrieve two or more products for time 2 (e.g. in the same week)
Task 4: Sen2Three - use Task 3 outputs as input- this will generate a single output for ~time 2

Derek

japoveda commented 6 years ago

Thanks Derek,

This is what I meant in my first post. We implemented this workflow, but the system only supports a single starting task node. As I understand from you answer, Task 1 and Task 3 ar two nodes.

Alex

derekocallaghan commented 6 years ago

Hi Alex,

Yes, I was thinking that Tasks 1-4 would be 4 workflow nodes.

We will investigate permitting >1 starting nodes, which will enable to you have two disconnected Task 1 ->Task 2 and Task-3 ->Task 4 branches as you require here. In the meantime, I'd suggest you'd just have a single Copernicus task that retrieves two Level 2A Sentinel-2 products (i.e. Sen2Cor-corrected) for time 1 and 2 respectively, or if you want Sen2Three products for local development, to have two separate workflows (Task 1-> Task2) and (Task 3 -> Task 4), which will enable you to download any Sen2Three outputs.

Derek

derekocallaghan commented 6 years ago

Hi Alex,

This will be resolved with the following workflow:

Task 1 - Copernicus - will retrieve all products for time 1 and time 2 (at least 2 products per time).
Task 2 - Sen2Three - it will be possible to specify groups of comma-separated product identifiers using a new optional parameter set. I.e. the first product id group would be for time 1 ("S2AMSIL2A...<product 1>",...,S2AMSIL2A..."), and the second product id group would be for time 2 ("S2AMSIL2A...<product 1>",...,S2AMSIL2A...").

Although both product groups will contain identifiers from the same region, Sen2Three will be executed independently for each group, resulting in two output products, one each for time 1 and time 2.

This will be available in version 0.2.8, which will be released in early January. In the meantime, I'd suggest you use L2A products for service development, or create L3A products with separate workflows, one workflow each for time 1 and time 2, and download the resulting output products.

Derek

derekocallaghan commented 6 years ago

Hi Alex,

The Sen2Three service has been updated to support the workflow described in my last post, please see the corresponding User Guide page for more details: https://docs.eo4a.science/services/atmosphere.html#sen2three

Derek

derekocallaghan commented 6 years ago

Closing as it has been confirmed that the Sen2Three service modifications were deployed and successfully tested.

parameterspace-ie / eo4a-service-trials

Possibility to use 2 data sets using Sen2Three #6