Provenance prototype

[agstephens]

@cehbrecht, @ellesmith88, let's discuss the provenance plan here. I will state a set of requirements and some suggestions, then we can discuss until we agree on a plan :-)

Terminology

• PC: provenance container - the object that collects up the provenance and can be serialised into various output formats.
• DSID: Dataset Identifier

Requirements

These are the requirements for our first provenance prototype:

• Do not put information into the PC if that information can be found elsewhere.
  • e.g. If a DSID can be looked up elsewhere in order to get the file list, then do not include the file list in the PC.
• Do include all the input DSIDs in the PC
• Do include information about the service software:
  • name of the software
  • version of the software
  • URL to the software
  • we might choose to point to the releases on the https://roocs.github.io/releases/ site
• Write the PC as a JSON file that is returned with the data files (within the MetaLink document)
• If fixes are applied, then they need to be recorded in the PC

[agstephens]

Suggestions for implementation

These are my initial suggestions, feel free to contest/improve them:

• Use the prov library as demonstrated in Carsten's examples
• The PC should contain a workflow record that includes a process chain of the operations defined in the call to rook
• Work out how our code keeps track of changes and adds them to the PC:
  • could use a global object - gets modified by some functions/methods
  • could use decorators - e.g. if a function is decorated with @report_prov then there is an extra argument available inside the function to report on its behaviour.
• We should only report a minimum number of operations/stages in the PC
• The following software should be recorded:
  • rook + version
  • roocs release (as in the combined releases, see: https://roocs.github.io/releases/ )
• We need to treat the daops fixes in a special way:
  • The fix lives in an ElasticSearch index which is date-stamped:
  • General search: https://elasticsearch.ceda.ac.uk/roocs-fix/_search
  • Points to a fixed version: https://elasticsearch.ceda.ac.uk/roocs-fix-2020-10-12/_search
  • There is an issue that the fix points to an implementation in the code, but it might refer to a different version of the (e.g. daops) code than we are running on the production service.
  • How should we resolve this?
    • What if we change the implementation (e.g. in daops)?
    • Would we have to re-run all tests to check fixes still work, on all datasets, then update all fixes in a new version of the fixes index???

[agstephens]

@cehbrecht @ellesmith88 over to you - what are your thoughts?

[ellesmith88]

I don't know if I fully understand the daops fix issue, but if we were to change the implementation of a fix then this would go out into the next version of daops. We could then create a new dated index to go along with this version and then only swap the alias over once the new version is in production.

This would potentially involve updating the fix records if the operands needed were changed and testing that they work, which could be tricky, but if there were a procedure for doing this it might be ok. To create the new index I think all the fixes have to be reindexed anyway (?) so this would probably have to happen anyway with a new release.

[agstephens]

Does this mean that we have to keep the entire history of all indexes in ElasticSearch? Or should we serialise them into GitHub? Maybe there aren't many...?

[cehbrecht]

* If _fixes_ are applied, then they need to be recorded in the PC

Would we like to record all applied fixes in the prov document? We could also point to a version of the used fixes. This could maybe even just a release documentation for the fixes.

I think the aim of the prov document is not to make the results reproducible but to give indicators why the results might have changed with a newer version.

[cehbrecht]

@agstephens I could start with a prototype in rook. I don't think it belongs to rook (maybe to daops, or daprov). But for prototyping it is easier to start here. We can use the prototype for further discussion ... to have something visible which then gets also returned with a wps request.

[agstephens]

+1

[agstephens]

Prototype works :-)