version for process graphs

[jdries]

As part of reproducibility & traceability, users are asking me how to record the version of the process graph itself in the output metadata. We already record backend version and the process graph itself, but for production projects, our users also attach versions to the workflow itself. I see two locations where this can be relevant:

• user defined processes
• batch jobs

My initial proposal would be to have a top-level property 'version' next to the others like title and description.

[m-mohr]

Just to be clear, what version do you mean (and what does it include)?

1. The revision of the process graph with its nodes and parameters itself (how is it determined? Is each change of an attribute in the Web Editor for example a change? is it only set each time it's sent to the server?)
2. Version of the processes and/or API?
3. The number of the run of the batch job? (e.g. if you run, update, run?)
4. Something else?

[jdries]

It's closest to option 1, but basically would be defined by the user who creates the job. The main problem is that small changes to the parameter in a process graph are not necessarily version changes. For instance, a graph with the same version can be applied to different spatial extents. This interpretation is something that can only be done correctly by the author of the process graph.

[m-mohr]

Aha, I see.

Some thoughts:

• In general, another place where this could potentially be relevant is secondary web services, I think.
• If this has more job focus, would it make sense to add something like a stac property to the job submission, which shall be merged with the resulting batch job metadata? That could be generally nice that you can customize the STAC output a bit. Then you could set the STAC version property. That's of course limited to batch jobs.
• When we are talking about "process graphs" here, I think what we really mean is the "process" (i.e. including its metadata), not just the graph with its nodes.
• We also have the general question, whether adding a version to a process specification (i.e. also pre-defined processes) would be useful: https://github.com/Open-EO/openeo-processes/issues/12 (and then the question is: aligned with the released version number or just an incrementing number on any "relevant" change)? Adding a version flag to a process that can be filled arbitrarily shouldn't be a big deal, I guess...

What do you think?

Related issue: https://github.com/Open-EO/openeo-api/issues/472

[soxofaan]

as already mentioned, I had same kind of question in #472 about adding extra PG metadata (e.g. client software version)

[m-mohr]

Closing in favor of #472