Add support for Workflow Manager items

[chris-fox]

Workflow Manager is now available in Online an Enterprise. Workflow Manager has a REST api to query and create the required items. The root url is a little different for Online and Enterprise so we will want to account for this in creation and deployment: https://developers.arcgis.com/workflow-manager/

I have looked through the api doc and create some test items and this is how I would suggest we approach creating and deploying these items:

Creation

• There is not much to a Workflow item data, we can ignore this. The item data will handled automatically when we use the api to create a new workflow during deployment (see below).
• For the Workflow item template we should get all the item properties, resources and thumbnail
• The key initial aspects that define the Worlflow item are the Diagrams and the Job Templates. We can use the following apis to get the Diagrams and Job Templates
  • getDiagrams Will return an array of diagrams infos
  • getDiagram Will return the full json for a single diagram based on a given diagramId
  • getJobTemplates Will return an array of job template infos
  • getJobTemplate Will return the full json for a single diagram based on a given jobTemplateId
• Using the above we can get all the diagrams and templates and I would propose we store these in the template properties, for example "properties" : { "diagrams": [], "jobTemplates": [] }
• We don't need to look for any specific dependencies with this item type, but at the end of packaging we should use the post processing step to search for variables in the diagrams and jobTemplates. For example there will likely be itemId reference for other templates or portalUrl references in the diagrams and jobTemplates.

Deployment

• We can use the createWorkflowItem api to create a new workflow item in the users content at the root folder. This handles creating the workflow item, supporting feature layers/view and the associated group.
• New workflows always have an "Introduction to Workflow Manager" Diagram and Job Template as a helpful starting point for new users. As part of the solution deployment we should delete these in the new workflow prior to adding the diagrams and job templates defined in the solution item. We can use the deleteDiagram and deleteJobTemplate apis to do this.
• We can use the createDiagram and createJonTemplate apis to add the diagrams and jobTemplates defined in the template properties. We need to create diagrams before the jobTemplates as the jobTemplate will reference a diagram.
• We can then update the item details for the workflow from the template and apply the thumbnail
• We should also move the workflow and associated feature layers/views into the same folder as the solution.
• At the end of the deployment we should run the post-process to do a final replacement of variables in the diagrams and jobTemplates in case there was a cyclical dependency. You can use the updateDiagram and updateJobTemplate apis if an update is required during this post-process.

Deleting

• We can use deleteWorkflowItem to handle deleting the workflow and the supporting feature layers/views and group.

I have created some sample items that can be used in testing. The group below contains a workflow, survey123 form and web application. The survey123 form and web application are both referenced in the workflow so it is a good test that they are templatized properly and swizzled on deployment.

6d45ec5c65464b20aaa5ac9ea4b55579

[chris-fox]

@MikeTschudi, Looking through the api, I think there might be an easier approach to this. There is an exportWorkflowConfiguration and importWorkflowConfiguration.

The export creates a .wmc file and will contain all the diagrams, jobTemplates and several other properties configured in the workflow. This could be a much simpler approach than having to handle every property individually. What is not clear to me is what kind of file is the .wmc file. Is it a json file? We need to be able to templatize properties in the diagrams so if it is not a file type that we can look for itemId references than this approach may not work for us.

There is no UI to get a wmc file in Workflow Manager and I have tried to use the python api to get it but I am having trouble authenticating. If you could get a sample .wmc file form the example workflow referenced in the comment above I would like to take a look to see if it might be a viable approach.

[MikeTschudi]

Sample .wmc file; it's a zip with a renamed extension. Because GitHub doesn't support ".wmc", the file is attached with the ".zip" extension. workflow_configuration_1691603968445.zip

[MikeTschudi]

Within the .wmc file, there appears to be just three files that need to be templatized:

• diagrams.json
• jobExtPropertyTableDefinitions.json
• roleGroups.json

[chris-fox]

@MikeTschudi, would it hurt to just templatize all the files. If there ends up being nothing in the file to templatize than it shouldn't hurt anything, but I guess my concern is I don't know enough yet to say which files might have other item references. Also if things change over time we are more likely to just handle the additional cases we are not aware of yet.

[MikeTschudi]

@chris-fox, it's a good idea--can't guarantee contents of file.

[chris-fox]

@MikeTschudi, I have been testing this today and have found workflow manager items are failing to deploy. I have been testing with this Workflow item:

https://localdeployment.maps.arcgis.com/home/item.html?id=661e0e38d4ad4a8f97731b22cb6c95dd

It creates successfully, but when I deploy it fails. Looking at the network traffic I am finding it is failing on the import command with the following error:

{
    "error": {
        "message": "Feature layer call failed with 400: Invalid URL",
        "type": "WorkflowDataStoreException"
    }
}

I believe the issue is earlier in the logic we are using addItem to add the new workflow item. However for Workflows there also need to be the supporting feature services and views for the import to work.

I believe instead need to call the createWorkflowItem api to create a new workflow item in the users content at the root folder. This handles creating the workflow item, supporting feature layers/view and the associated group.

After this we could do things like update the item details of the workflow and thumbnail and move the items to the solution contents folder. Then we could use the import api to import the configuration.

[chris-fox]

@MikeTschudi, I tested this out and found I was able to successfully package and deploy a workflow item. In the resulting workflow I was able to create new jobs and run through them. I also had a survey with webhook configured to create jobs in the workforce project and this deployed successfully as well and was able to create the job correctly.

The issue I am seeing is references to items contained within the workflow are not being swizzled. For example in am embedding a web app (that is also in the group), but the portalBaseUrl and item id variables are not being applied. Below is a screenshot from the json for the "diagrams.json" property under the workflow template:

In addition, I am also referencing the survey in the workflow and it is not swapped with the variable for in the "jobExtPropertyTableDefinitions.json" property:

I would expect these items be discovered in the group and for them to be swapped out for variables in the workflow json property files so that they can be swizzled on deployment. We will also want to look for any feature service url references included in the solution. I would then expect these to be added as dependencies of the workflow template to ensure the items are created first and the variables can be resolved.

I am using this group to test:

https://localdeployment.maps.arcgis.com/home/group.html?id=d6888336943548e58bf7f40e7cf824e9#overview

[MikeTschudi]

@chris-fox, all of the swapping occurs during deployment: for each id recognized, it is replaced if it is found in the solution. This can be changed to templatizing.

I'll look into discovering the feature services that are missing from the group but should be added.

[chris-fox]

@MikeTschudi, thanks I would prefer to go the templatizing route as it is consistent with other item types and makes it easier for us to manually modify the definition in the template if we want to add or remove replacements.

With regard to the feature services, I don’t think you need to auto discover feature services that were not shared with the group.

I was just saying if there was a feature service included in the group or a dependency of another item like a web map and that same feature service url was referenced in any of the json files of the workflow item that we should also templatize those.

[MikeTschudi]

Sounds good. It appears to be much easier than I expected: I more-or-less moved the code from deploy to create.

Thanks regarding discovery. By having the service in the group, it gets an entry in the template dictionary. Thus, when I discover what looks like an AGO id in the workflow config and it's in the dictionary, I can templatize it and add it to the workflow's dependencies.

[chris-fox]

@MikeTschudi, I was testing this morning with group https://localdeployment.maps.arcgis.com/home/group.html?id=d6888336943548e58bf7f40e7cf824e9#overview and it is working almost perfect. The only issue I noticed is the portal base url in an app url reference is not being replaced with the {{portalBaseUrl}} variable:

I would expect it look like below similar to how we are doing this in the corresponding Web Mapping Application item url in the group:

There is also another reference to 'https://localdeployment.maps.arcgis.com' in one of the json files for a path to the workflow itself:

We should find any references to the portal base url and replace with the {{portalBaseUrl}} variable. If I manually edit the solution item to make these replacements and deploy everything works perfectly.

[chris-fox]

We discovered during testing that we were not providing the correct root url for enterprise. At this point in the release we have decided we will not be able to support workflow manager for Enterprise 11.3. Things are working well for AGOL, I will update the milestone to the AGOL release and we can work on addressing this for Enterprise so it is available for 11.4.

https://developers.arcgis.com/workflow-manager/api-reference/rest/

[chris-fox]

See the comment below to dynamically get the root url for the workflow manager api for both ArcGIS Online (prod, qa and dev) and ArcGIS Enterprise

https://devtopia.esri.com/WebGIS/solution-deployment-apps/issues/64#issuecomment-4705767

[chris-fox]

@MikeTschudi, I attempted to package this group this morning with the latest create solution demo app from the develop branch but it failed with a console error on the workflow item. This was working previously and wondering if there was a regression with the workflow root url updates.

https://localdeployment.maps.arcgis.com/home/group.html?id=d6888336943548e58bf7f40e7cf824e9

[chris-fox]

Verified that I was able to create a solution from a workflow and deploy it in production. Everything was swizzled properly and behaved as expected. I also verified that the root url was properly updated for devext for both creation and deployment in solution.js.

Everything is looking good. I have asked some team members to test with their solutions as well.