Simple rendering of user defined processes?

[jdries]

With UDP's, our users can now create and share higher level processes. The editor also has a nice feature that loads such a UDP directly from json: https://editor.openeo.org/?server=https://openeo.vito.be&discover=1&process=https://openeo-dev.vito.be/openeo/1.0/processes/vito/MSI

This however shows the full process graph, with all it's underlying complexity. This is needed for a user that wants to edit that graph, but not if you just want to run this process that somebody shared with you:

For the use case where users want to simply start a (batch) job based on a process that was shared with them, it would be nice to drop in to a view like this: Where a user can set the parameters, and then off course has the option to start a new job.

Would that be feasible within the web editor, based on a URL parameter if needed?

[m-mohr]

The editor also has a nice feature that loads such a UDP directly from json: https://editor.openeo.org/?server=https://openeo.vito.be&discover=1&process=https://openeo-dev.vito.be/openeo/1.0/processes/vito/MSI

Hmm, interesting, there is either an issue with the process or the Web Editor. Some connections are missing, I have to look into the issue there (but my JSON Editor struggles with the massive kernels right now).

This however shows the full process graph, with all it's underlying complexity. This is needed for a user that wants to edit that graph, but not if you just want to run this process that somebody shared with you:

I still think this could be valuable as it could indeed be interesting what the actual process does. What we could probably look into: Show the whole process as a single "block" with parameters you can edit. That is already possible with UDPs that are actually stored, but there are several issues to overcome here so not an easy task.

For the use case where users want to simply start a (batch) job based on a process that was shared with them, it would be nice to drop in to a view like this: Where a user can set the parameters, and then off course has the option to start a new job.

Would that be feasible within the web editor, based on a URL parameter if needed?

Interesting idea. Easily doable if I could simply store the loaded process as UDP, but I feel like that is too intrusive. The other option is more difficult to implement as I have to mimic a "UDP" on the client-side, which is not supported yet and would need quite a bit of new code.

Another issue with this process is that you can't directly make use of it without adding a save_result, right?

(And lastly, you can't directly run it with &discover=1 as you are not logged in yet.)

I'll try to look into this for 0.9 over the summer.

[m-mohr]

For debugging: Process graph without kernels -> MSI.txt

@jdries The date parameter is ill-defined, must match the type of filter_temporal: https://github.com/Open-EO/openeo-processes/blob/master/filter_temporal.json#L21-L61

Here's how the parameters should probably look like (and which you'd probably want to have for a better UI in the Web Editor anyway):

  "parameters": [
    {
      "default": "SENTINEL2_L2A_SENTINELHUB",
      "description": "The collecion_id for calculation of MSI",
      "name": "collection_id",
      "optional": true,
      "schema": {
        "type": "string",
        "subtype": "collection-id"
      }
    },
    {
      "default": "B08",
      "description": "Band name of Sentinel-2 band 08",
      "name": "band08",
      "optional": true,
      "schema": {
        "type": "string",
        "subtype": "band-name"
      }
    },
    {
      "default": "B11",
      "description": "Band name of Sentinel-2 band 11",
      "name": "band11",
      "optional": true,
      "schema": {
        "type": "string",
        "subtype": "band-name"
      }
    },
    {
      "default": "SCL",
      "description": "Band name of the scene classification band",
      "name": "band_classification",
      "optional": true,
      "schema": {
        "type": "string",
        "subtype": "band-name"
      }
    },
    {
      "description": "An temporal extent for which to compute the Sentinel-2 MSI.",
      "name": "date",
      "schema": {
        "type": "array",
        "subtype": "temporal-interval",
        "minItems": 2,
        "maxItems": 2,
        "items": {
          "anyOf": [
            {
              "type": "string",
              "format": "date-time",
              "subtype": "date-time"
            },
            {
              "type": "string",
              "format": "date",
              "subtype": "date"
            },
            {
              "type": "string",
              "subtype": "year",
              "minLength": 4,
              "maxLength": 4,
              "pattern": "^\\d{4}$"
            },
            {
              "type": "null"
            }
          ]
        },
        "examples": [
          [
            "2015-01-01T00:00:00Z",
            "2016-01-01T00:00:00Z"
          ],
          [
            "2015-01-01",
            "2016-01-01"
          ]
        ]
      }
    },
    {
      "description": "A GeoJSON polygon",
      "name": "polygon",
      "schema": {
        "type": "object",
        "subtype": "geojson",
        "properties": {
          "type": {
            "const": "Polygon"
          }
        }
      }
    }
  ],

[jdries]

Thanks, I was already working on some of those bad schemas, but your example certainly helps. This is another example of a process without huge kernels: https://editor.openeo.org/?server=https://openeo.vito.be&discover=1&process=https://openeo.vito.be/openeo/1.0/processes/vito/BIOPAR

[m-mohr]

This is another example of a process without huge kernels: https://editor.openeo.org/?server=https://openeo.vito.be&discover=1&process=https://openeo.vito.be/openeo/1.0/processes/vito/BIOPAR

Thanks. This process also has invalid schemas, by the way. The Web Editor 0.8.0-rc.4 will handle those issues a bit better now. (Why do you do the explicit filter_temporal instead of passing the parameter into load_collection?)

I'm still thinking how this could be implemented in a good way, but have not found a good answer yet. Could we discuss this at some point so that I can better understand the requirements and expectations? @jdries

[m-mohr]

Some additional thoughts from this morning (mostly as notes for myself):

I could probably import a UDP imported via URL into the model builder like a UDP from /process_graphs, I just need to define a new "block type", which then expands again on export. A user could send use the (i) button to see the PG (but not edit it). The parameters could be set via the usual edit button and I could also show this by default. Probably a new query parameter would be good for this kind of behavior (e.g. &udp=... instead of &process=...).

The issue is now that the API doesn't allow to run UDP with parameters directly as we would need to set the parameters somehow, but this is only supported by back-ends if it has been stored as UDP and then users can set parameters in a new process which uses the UDP. There's no way to store a UDP (with parameters) as batch job and pass the parameters in the same call, this is an API issue.

Also, it is not a trivial task to simply replace all the { from_parameter: ... } nodes in the process directly due to scoping and not fully defined processes. e.g. the udp could have a data parameter, but a child (like reduce_dimension) could also have a data parameter. While for reduce_dimension, there's usually a well-defined process definition, you run into issues once you have parameters in use that are coming from (not completely defined) UDP, i.e. where no parameter definition is given. Then it's not clear how to resolve, but that may actually also be an issue during execution. So UDPs used in UDPs must be well-defined. It could be a good idea to require the parameters property in the API for UDPs.

Storing a UDP directly on Web Editor load could be problematic as it would need to wait for login, but this may never happen or only very late. Usually a user is not logged in by default right now, see https://editor.openeo.cloud

Simply replacing the udp process graph with a single block may be confusing as it's not clear whether save_result is already included or not.

[m-mohr]

We now have a prototype for this in the branch ns-support that follows the discussions from the recent telco. The changes will likely be released as part of v0.9.0.

[m-mohr]

Here's an example of the implementation:

https://editor.openeo.org/?server=https://openeo.vito.be&namespaces=vito&process=MSI@vito&edit-node=1

Closing this issue. Please open new issues for bugs and related feature requests.