merge_cubes and divide in overlap resolver

[clausmichele]

I was trying to perform the n to 1 division needed for Eurac's use case without success. The process graph is the following and it's also stored in batch jobs with name merge_cubes_division_n_to_1

Did I misunderstood how to use merge_cubes? Or is this type of merge not yet implemented?

Cheers, Michele

{
  "process_graph": {
    "1": {
      "process_id": "merge_cubes",
      "arguments": {
        "cube1": {
          "from_node": "filtertemporal1"
        },
        "cube2": {
          "from_node": "reducedimension1"
        },
        "overlap_resolver": {
          "process_graph": {
            "1": {
              "process_id": "divide",
              "arguments": {
                "x": {
                  "from_parameter": "x"
                },
                "y": {
                  "from_parameter": "y"
                }
              },
              "result": true
            }
          },
          "parameters": [
            {
              "name": "x",
              "description": "",
              "schema": {}
            },
            {
              "name": "y",
              "description": "",
              "schema": {}
            }
          ]
        }
      }
    },
    "2": {
      "process_id": "reduce_dimension",
      "arguments": {
        "data": {
          "from_node": "1"
        },
        "reducer": {
          "process_graph": {
            "1": {
              "process_id": "min",
              "arguments": {
                "data": {
                  "from_parameter": "data"
                }
              },
              "result": true
            }
          },
          "parameters": [
            {
              "name": "data",
              "description": "",
              "schema": {}
            }
          ]
        },
        "dimension": "t"
      }
    },
    "loadcollection1": {
      "process_id": "load_collection",
      "arguments": {
        "bands": [
          "B4"
        ],
        "id": "COPERNICUS/S2",
        "spatial_extent": {
          "west": 11.04177474975586,
          "south": 46.03653930484754,
          "east": 11.206226348876955,
          "north": 46.12417679163178
        },
        "temporal_extent": [
          "2017-03-01",
          "2017-06-01"
        ]
      }
    },
    "filtertemporal1": {
      "process_id": "filter_temporal",
      "arguments": {
        "data": {
          "from_node": "loadcollection1"
        },
        "extent": [
          "2017-03-01",
          "2017-04-01"
        ]
      }
    },
    "reducedimension1": {
      "process_id": "reduce_dimension",
      "arguments": {
        "data": {
          "from_node": "filtertemporal1"
        },
        "dimension": "t",
        "reducer": {
          "process_graph": {
            "mean1": {
              "process_id": "mean",
              "arguments": {
                "data": {
                  "from_parameter": "data"
                }
              },
              "result": true
            }
          },
          "parameters": [
            {
              "name": "data",
              "description": "",
              "schema": {}
            }
          ]
        }
      }
    },
    "renamelabels1": {
      "process_id": "rename_labels",
      "arguments": {
        "data": {
          "from_node": "2"
        },
        "dimension": "bands",
        "target": [
          "R"
        ]
      }
    },
    "apply1": {
      "process_id": "apply",
      "arguments": {
        "data": {
          "from_node": "renamelabels1"
        },
        "process": {
          "process_graph": {
            "linearscalerange1": {
              "process_id": "linear_scale_range",
              "arguments": {
                "inputMax": -5,
                "inputMin": -20,
                "outputMax": 255,
                "outputMin": 0,
                "x": {
                  "from_parameter": "x"
                }
              },
              "result": true
            }
          },
          "parameters": [
            {
              "name": "x",
              "description": "",
              "schema": {}
            }
          ]
        }
      }
    },
    "saveresult1": {
      "process_id": "save_result",
      "arguments": {
        "data": {
          "from_node": "apply1"
        },
        "format": "PNG",
        "options": {
          "red": "R",
          "green": "G",
          "blue": "B"
        }
      },
      "result": true
    }
  }
}

[m-mohr]

@claxn @bgoesswe One of you has implemented the merge_cubes operation, right? Can you check this, please? Let me know if there's something I can help with.

[bgoesswe]

Yes, I'll have a look

[clausmichele]

Update:

1. Using 2 load_collection processes with merge_cubes with overlap resolver produces a result. Instead, using the same data as input twice produces an error. I put here the two process graphs.
2. The label setting in array_element process inside merge_cube is ignored. In fact, in the working graph attached here there are two labels which are wrong but it works anyway. Setting no label at all produces the same result.
3. What is the meaning of X and Y in merge_cubes? Are they cube1 and cube2? Because I can use only x, only y, x and y, y and x but the result is always the same. I guess that the merge_cubes process doesn't do what is expected.

openEO_merge_cubes_division_n_to_1_works.txt

openEO_merge_cubes_division_n_to_1_notworking.txt

[m-mohr]

@bgoesswe Any news regarding merge_cubes?

3. What is the meaning of X and Y in merge_cubes? Are they cube1 and cube2? Because I can use only x, only y, x and y, y and x but the result is always the same. I guess that the merge_cubes process doesn't do what is expected.

Likely a GEE issue. See also https://github.com/Open-EO/openeo-processes/issues/184.

[bgoesswe]

I am sorry for the inconvenience, vacation times and deliverables kept me from continuing this, I will try to fix this within this week.

Using 2 load_collection processes with merge_cubes with overlap resolver produces a result. Instead, using the same data as input twice produces an error. I put here the two process graphs.

Thanks for the test inputs, this is of course wrong behavior to be investigated.

The label setting in array_element process inside merge_cube is ignored. In fact, in the working graph attached here there are two labels which are wrong but it works anyway. Setting no label at all produces the same result.

Since, the overlap resolver is at the current productive instance not implemented and executed at all, there will be no errors from the overlap resolver graph at the moment, so it is basically ignored right now.

[m-mohr]

Since, the overlap resolver is at the current productive instance not implemented and executed at all

Thanks for the hint. I wasn't aware that the deployed GEE instance is out-dated. I just updated the deployed version.

[claxn]

I had a discussion with @bgoesswe, who tries to currently implement a solution for the overlap_resolver. @clausmichele: We came to the conclusion that the overlap_resolver as described by the process specs is too complicated to be generally solved in GEE. There might be a workaround, but I think we would still need some time to work this out. I had a look at the EURAC use case, and it seems like, that you can circumvent the usage of merge_cubes. If you call divide, I assume your input data set is given in the linear domain, not in dB. If you would call anomaly(apply(apply(dc, log10), multiply(x, 10))), i.e., converting linear units to dB and then doing a subtraction with respect to the mean value (process anomaly) (and after this operation converting dBs back to linear units), then you would have your desired result, I guess. I will further discuss the overlap_resolver with @bgoesswe and @m-mohr , but this has a low priority at the moment, since we have to focus on D26.

[m-mohr]

The overlap_resolver has been removed temporarily.

[m-mohr]

Closing this issue in favor of #25, which is basically tackling the same issue.