Problem: child workflow tasks aren't visible in the UI

Is your feature request related to a problem? Please describe.

Users can develop a child workflow to run a custom workflow on their SIPs. For example, a client may want to run a validation check to ensure that the SIP complies with their local SIP specification, or ensure that all of the file formats in the SIP are permitted. At present, none of this information appears in the Enduro user interface - meaning that the user has no visibility into whether the workflow was actually run. If a SIP fails one of the checks, it is not easy for operators to determine why the workflow halts, unless they go digging in the Temporal logs for more information.

Describe the solution you'd like

The goal is to make custom task information and outcomes visible in the Enduro UI. We have a client with this need (SFA) so we can use their child workflow as an example implementation, but this will be a constant need for any Enduro client/user with custom child workflow activities - meaning that ideally, we define some reusable abstractions in how we do this so it is easy to replicate for future custom tasks.

For SFA, a breakdown of the suggested task names, success messages (to be added to the Notes col), and failure outputs, can be found here in the SFA Miro board

NOTE: there are two additional proposed pre-ingest tasks there (in the blue rows) that do not currently exist in PoC#1, and are not part of SFA pre-ingest validation. These will be described further in another card, but this card should be used for any work to add them to the Enduro UI.

There is also one task in yellow, that actually happens at the end of ingest / during post-ingest - the creation and delivery of the metadata bundle for delivery to the AIS. The linked table above also includes suggested UI info for that custom activity as well.

Finally, since we will want to reuse this pattern, we should add some developer notes - perhaps to the pre-processing base repo's README, or even an internal doc for now - on how other developers can reuse the pattern for future custom workflow activities.

Describe alternatives you've considered

None

Additional context

We have a related open issue, #934, about the Notes column, and adding more feedback and information to the Enduro UI. There is still some discussion about how to implement these in a way that is user-friendly and accessible.

I'm implementing this by adding Enduro API endpoints to add preservation actions and preservation tasks, then I'll POST to those endpoints from pre-processing to add the pre-processing tasks to the Enduro database.

I'll do this work as a set of smaller pull requests:

[ ] Add a "Create preservation action" API endpoint to Enduro (#952)
[ ] Add a "Create preservation task" API endpoint to Enduro
[ ] Add an Enduro API client with config to preprocessing-sfa
[ ] Add the preprocessing-sfa API client calls to add preservation tasks to Enduro
[ ] Copy the Enduro API client implementation from preprocessing-sfa to pre-processing-base

Thanks for outlining the work needed @djjuhasz. Why did you decide to go this way instead of using the workflow result?

I have some concerns about this approach:

As we talked today, the idea was to show the preprocessing preservation tasks as part of the create AIP preservation action:
- It looks like you are planning to create a new preservation action to capture all preprocessing tasks, but I'm not sure how that will display in the UI.
- For a new action, we'll need to pass the parent workflow id (maybe that could be obtained through Temporal). Using the existing preservation action will require to do pass its DB id.
I'm not sure if exposing this operations in the API is a good idea, they look too internal for me and I struggle to find another use case for them.
Another task not included in that list will be to make this requests work against an Enduro instance with authentication enabled. We worked around this issue by having a second instance (only accessible inside the cluster) with authentication disabled, but that won't be possible/desirable in other deployments.

My main concern about using the workflow result would be its payload limits, but I don't think we want to add that many tasks to the UI (same as with AM tasks) and we could try to control the notes size. This approach will take a lot less time, extending the workflow result struct, reusing the existing local activity to create those tasks in the DB from the child workflow results, having access to the running preservation action id, and specially not having to worry about API interactions and authentication.

We can discuss it further tomorrow, but I think it's not too late to change the approach it this makes sense to you.

@jraddaoui thanks for your questions, I've responded below.

Thanks for outlining the work needed @djjuhasz. Why did you decide to go this way instead of using the workflow result?

Conceptually I have been thinking of preprocessing and processing (the Enduro workflow) as two separate applications, and therefore we should be respectful of the application boundaries by using well defined, public interfaces between the two applications. From your comments here and in our meeting yesterday, it sounds to me like you are thinking of the processing - preprocessing child workflow system as a single application, using Go function signatures to define the interface between the two components and relying on the Temporal workflow gRPC communication layer to pass data back and forth.

It's definitely simpler and more efficient to pass the preservation task data back in the preprocessing child workflow results, taking advantage of the Temporal gRPC communications layer, then via the Enduro HTTP API. I do think we are going to have to be careful of the limitations on passing data between Temporal workflows, and to be clear that the processing and preprocessing workflow executions don't share memory.

I have some concerns about this approach:

* As we talked today, the idea was to show the preprocessing preservation tasks as part of the create AIP preservation action:

  * It looks like you are planning to create a new preservation action to capture all preprocessing tasks, but I'm not sure how that will display in the UI.
  * For a new action, we'll need to pass the parent workflow id (maybe that could be obtained through Temporal). Using the existing preservation action will require to do pass its DB id.

Yes, I just assumed that we'd want a separate preservation action for the preprocessing, since the data is coming from a different system than the Archivematica job data. As we agreed in the meeting yesterday I will go forward with adding the preprocessing preservation tasks to the existing "Create AIP" preservation action. Good point about having to pass the parent workflow ID or preservation action database key to preprocessing, I didn't think of that.

* I'm not sure if exposing this operations in the API is a good idea, they look too internal for me and I struggle to find another use case for them.

We could use the new API endpoints to send processing data to Enduro from any systems that act on the preservation objects. I don't think we should assume that the child workflow will be the only way that the package is modified before or after AIP creation processing. For instance if we chose to use a completely independent application for extracting the sidecar metadata from an SFA AIP, we could use the API to display the actions of that application in the Enduro dashboard. Of course this is all theoretical right now, and I don't think it's a strong argument for adding the API endpoints I proposed at this time.

* Another task not included in that list will be to make this requests work against an Enduro instance with authentication enabled. We worked around this issue by having a second instance (only accessible inside the cluster) with authentication disabled, but that won't be possible/desirable in other deployments.

Yes, but need to solve the problem of API authentication with the current Enduro implementation anyway, these changes aren't introducing a new problem. :wink:

My main concern about using the workflow result would be its payload limits, but I don't think we want to add that many tasks to the UI (same as with AM tasks) and we could try to control the notes size. This approach will take a lot less time, extending the workflow result struct, reusing the existing local activity to create those tasks in the DB from the child workflow results, having access to the running preservation action id, and specially not having to worry about API interactions and authentication.

I agree that using the Temporal workflow interface for data sharing is easier as is adding the preprocessing data to the Enduro database inside the Enduro processing workflow rather than via a REST API. I think you are right that the cardinality and size the child workflow data are small enough that we won't run into the Temporal payload limits. I do think we'll need to be careful about adding more data to the preprocessing workflow results in the future though so we don't run into the limits.

We can discuss it further tomorrow, but I think it's not too late to change the approach it this makes sense to you.

The simplicity and shorter development time of your approach makes sense to me.

Conceptually I have been thinking of preprocessing and processing (the Enduro workflow) as two separate applications, and therefore we should be respectful of the application boundaries by using well defined, public interfaces between the two applications. From your comments here and in our meeting yesterday, it sounds to me like you are thinking of the processing - preprocessing child workflow system as a single application, using Go function signatures to define the interface between the two components and relying on the Temporal workflow gRPC communication layer to pass data back and forth.

It's definitely simpler and more efficient to pass the preservation task data back in the preprocessing child workflow results, taking advantage of the Temporal gRPC communications layer, then via the Enduro HTTP API. I do think we are going to have to be careful of the limitations on passing data between Temporal workflows, and to be clear that the processing and preprocessing workflow executions don't share memory.

Yes, that was my thinking and how it was outlined in the document where we explored options for preprocessing. We even considered a replaceable package that could share clients and so on.

We could use the new API endpoints to send processing data to Enduro from any systems that act on the preservation objects. I don't think we should assume that the child workflow will be the only way that the package is modified before or after AIP creation processing. For instance if we chose to use a completely independent application for extracting the sidecar metadata from an SFA AIP, we could use the API to display the actions of that application in the Enduro dashboard. Of course this is all theoretical right now, and I don't think it's a strong argument for adding the API endpoints I proposed at this time.

In this case I'd consider it as an external application and I doubt we'd want to show those preservation actions in our UI. I'd expect those in PREMIS events inside the package, which is also another option to pass data between workflows or in an initial SIP.

Yes, but need to solve the problem of API authentication with the current Enduro implementation anyway, these changes aren't introducing a new problem. 😉

Not a new problem, but it will require us to address it to deliver this feature.

We should also add Identify SIP structure, Restructure SIP, and Bag SIP to the list of child workflow activities visible in the UI, per @djjuhasz's suggestion. I've added the task name, success message, and failure message to the table

LGTM! I can see the child workflow tasks, start and end times, and notes.

artefactual-sdps / enduro

Problem: child workflow tasks aren't visible in the UI #949