Problem: Processing configuration components are hard-coded making sustainable and flexible customization of Archivematica difficult, e.g. hard-coded decision-point options in the processing configuration window

[finwo]

Problem: The labels used for the microservices providing choices are hardcoded in archivematica/src/dashboard/src/components/administration/forms.py and archivematica/src/MCPServer/lib/server/processing_config.py. More specifically: Defining a custom label for a choice is not possible, preventing a "microservice" to be preconfigurable in the administration page in the dashboard.

Expected behaviour:

Specifying a label in the config of a link in the workflow.json allows you to override the label of an existing link on the edit page for your workflow under administration.

Current behaviour:

The edit page takes longer than 30 seconds (resulting in a timeout in our systems, so whether or not it actually responds is unclear).

Steps to reproduce:

Add a label key (string or with language keys doesn't matter) under the config of a link in the workflow.json.

Your environment (version of Archivematica, operating system, other relevant details):

Version of Archivematica: 1.9, 1.10, 1.11

Operating system: Dockerized, ubuntu 16.04/18.04

Goal of the fix:

In our environment, we use custom microservices. The ones which provide choices are not configurable in the administration page, even though the code generating that page suggests the choices being shown are read (almost) directly from the workflow.json file. We'd like to be able to have our microservices configurable, so we don't have to manually create a processingMCP.xml or replace the whole workflow.json for each transfer using different settings.

[sromkey]

@ross-spencer do you mind investigating?

[finwo]

@Frottello Are you watching this issue yet?

[FransPicturae]

@finwo yes i'm subscribed. So I keep a watch.

[ross-spencer]

Okay, I think I've a grasp of this one.

First, I think workflow.json is largely working as expected and that workflow choice labels can be modified. That is, however, only for existing jobs, and largely, only those which can be changed in the processing configuration.

See as an example "Examine contents" choices when I change that in workflow.json and then restart the MCP Server.

Second though, we have a problem with how much a developer can be expected to customize the workflow with ease and how much Archivematica can be expected to know about that customization.

Before heading down that path, it's probably important to note that workflow.json is still a work in progress and at the time it was accepted into the Archivematica code-base it was done so primarily to support translation efforts, but there are still improvements to be made. See for example some of the links associated with https://github.com/archivematica/Issues/issues/1207.

If we customize workflow.json to add new chain-links, and make those chain-links of the type MicroServiceChainChoice we can expect to see those choices in the workflow as follows:

NB. the diff for this change is as follows:

diff --git src/MCPServer/lib/assets/workflow.json src/MCPServer/lib/assets/workflow.json
index 273f88f4..5ab15b99 100644
--- src/MCPServer/lib/assets/workflow.json
+++ src/MCPServer/lib/assets/workflow.json
@@ -1,5 +1,17 @@
 {
     "chains": {
+        "1115dd57-1111-4137-a7a2-9aa3286abfd6": {
+            "description": {
+                "en": "Custom option 1"
+            },
+            "link_id": "5555dd57-5555-4137-a7a2-9aa3286abfd6"
+        },
+        "2225dd57-2222-4137-a7a2-9aa3286abfd6": {
+            "description": {
+                "en": "Custom option 2"
+            },
+            "link_id": "6665dd57-6666-4137-a7a2-9aa3286abfd6"
+        },
         "01d80b27-4ad1-4bd1-8f8d-f819f18bf685": {
             "description": {
                 "en": "Yes",
@@ -8440,6 +8452,79 @@
                 "sv": "Undersök innehåll"
             }
         },
+        "333ba875-3333-4d2c-85a7-a4c622ac1014": {
+            "config": {
+                "@manager": "linkTaskManagerChoice",
+                "@model": "MicroServiceChainChoice",
+                "chain_choices": [
+                    "1115dd57-1111-4137-a7a2-9aa3286abfd6",
+                    "2225dd57-2222-4137-a7a2-9aa3286abfd6"
+                ]
+            },
+            "description": {
+                "en": "Custom microservice?"
+            },
+            "exit_codes": {},
+            "fallback_job_status": "Failed",
+            "fallback_link_id": null,
+            "group": {
+                "en": "Custom microservice group"
+            }
+        },
+        "5555dd57-5555-4137-a7a2-9aa3286abfd6": {
+            "config": {
+                "@manager": "linkTaskManagerDirectories",
+                "@model": "StandardTaskConfig",
+                "arguments": "\"reject\" \"%SIPUUID%\"",
+                "execute": "failedSIPCleanup_v1.0",
+                "filter_file_end": null,
+                "filter_file_start": null,
+                "filter_subdir": null,
+                "stderr_file": null,
+                "stdout_file": null
+            },
+            "description": {
+                "en": "Reject custom microservice one"
+            },
+            "exit_codes": {
+                "0": {
+                    "job_status": "Completed successfully",
+                    "link_id": "3467d003-1603-49e3-b085-e58aa693afed"
+                }
+            },
+            "fallback_job_status": "Failed",
+            "fallback_link_id": "3467d003-1603-49e3-b085-e58aa693afed",
+            "group": {
+                "en": "Reject SIP"
+            }
+        },
+        "6665dd57-6666-4137-a7a2-9aa3286abfd6": {
+            "config": {
+                "@manager": "linkTaskManagerDirectories",
+                "@model": "StandardTaskConfig",
+                "arguments": "\"reject\" \"%SIPUUID%\"",
+                "execute": "failedSIPCleanup_v1.0",
+                "filter_file_end": null,
+                "filter_file_start": null,
+                "filter_subdir": null,
+                "stderr_file": null,
+                "stdout_file": null
+            },
+            "description": {
+                "en": "Reject custom microservice two"
+            },
+            "exit_codes": {
+                "0": {
+                    "job_status": "Completed successfully",
+                    "link_id": "3467d003-1603-49e3-b085-e58aa693afed"
+                }
+            },
+            "fallback_job_status": "Failed",
+            "fallback_link_id": "3467d003-1603-49e3-b085-e58aa693afed",
+            "group": {
+                "en": "Reject SIP"
+            }
+        },
         "ad011cc2-b0eb-4f51-96bb-400149a2ea11": {
             "config": {
                 "@manager": "linkTaskManagerDirectories",
@@ -9909,7 +9994,7 @@
             "exit_codes": {
                 "0": {
                     "job_status": "Completed successfully",
-                    "link_id": "ef6332ee-a890-4e1b-88de-986efc4269fb"
+                    "link_id": "333ba875-3333-4d2c-85a7-a4c622ac1014"
                 }
             },
             "fallback_job_status": "Failed",

NB. the two links both point to "reject transfer".

As we can see the changes are reflected in the Workflow, but the difficulty for the developer is that they do not appear in the processing configuration options - the new changes are not configurable in the UI.

NB. As noted above, it is possible to manually write a processing config file which means decision making during a transfer can still be automated, which is good to know as a workaround, but I do acknowledge that this is less than satisfactory for developers customizing Archivematica with components not likely to make it into the mainline branches.

As Robin alludes to, some of the chain choices are picked up by "type" in processing_config.py in the MCP Server. And we also require type to be configured there.

The labels for the processing config (the left hand column) are picked up in the dashboard code. And here.

NB. In the code we acknowledges a TODO.

This is also a lot of effort (both to discover how to do, and to implement) and it would benefit developers if some of this complexity was removed (it not entirely). Being able to control the view from workflow.json might be one approach and does seem conceivable as a starting position and I'd suggest the intuitive way of doing it.

There is a tangentially related issue https://github.com/archivematica/Issues/issues/1109 whereby the translation artifacts aren't built during deployment and so it also looks like the processing configuration labels are largely "English" only. 1109 also points out that there is some work to be done to work with non-ASCII strings correctly too.

[ross-spencer]

@finwo @FransPicturae I've gone through this today to get a better feel for it. Hopefully I've captured the essence here accurately. Let me know if not. I've also drafted a new title for the issue to try and capture it a bit more precisely/attach a bit more detail. Hopefully that looks okay too? Let us know if not.

[finwo]

@ross-spencer Sorry for the late response. The problem is not the select working during a run but the option not being able to be selected from the administration/edit page.

When we try adding labels to it, copied from other selects, the edit page becomes unusable. Because of this, we can not pre-configure a default option in archivematica for a custom step in the workflow, even though all the data is there in the same way your microservices have the data present.