search

finos / waltz

Enterprise Information Service
https://waltz.finos.org
Apache License 2.0
182 stars 129 forks source link

Survey Workflow: design #5757

Open jessica-woodland-scott-db opened 2 years ago

jessica-woodland-scott-db commented 2 years ago

Users would like the ability to use a workflow system to drive the issuance of surveys with a view to updating one or more assessments in waltz.

For example, an architectural review process: This workflow is used to track architectural governance compliance of change initiatives.

2 assessments will be used:

An external batch job determines which change initiatives are in scope and sets the outcome assessment to 'In Scope'. A workflow job will wait for assessments in this state and issue a "scoping" survey and update the workflow status assessment to be 'Pending Completion'. etc.... Once the survey is approved the initial assessment is updated to reflect the end state.

Workflow Support

Over the past year we have been using Waltz to support custom workflows. These workflows are orchestrated using bespoke jobs and are not easily reusable. We believe a small set of reusable, composable functionality could simplify the creation future workflows.

An Example workflow

Examples of these bespoke workflow processes have been in areas such as Records Management. The Records Management process coordinates multiple surveys with the goal of providing assessment and rating values against the subject application.

Scoping Survey

The process starts with a job issuing a Scoping survey to determine what, if any, record classes a subject application manages. The scoping survey is issued to the application owner and is reviewed by a representative for the Records Management team. Once approved the record classes are used to determine the next step:

for each declared record class a provisional rating is stored as a Records Management Taxonomy Rating for that application

Detail Survey

A secondary job periodically checks the Records Management Taxonomy Rating looking for provisional ratings. For each rating found a new Detailed Survey is issued against the application with a similar set of owners and approvers. As each of the Detail Surveys are approved then the taxonomy ratings are either removed, or the rating moves from provisional to Confirmed.

Observations

The two jobs which issue the Scoping and Detail surveys as intentionally decoupled. They use a combination of Assessments and Measurable (Taxonomy) Ratings to have small, controlled amount of shared state.

Due to this separation, Records Management Administrators can override values in the ratings or assessments and the workflow will act correctly.

For example, an administrator could manually store some provisional record ratings against an application and the corresponding detail surveys would be issued without the need for a scoping survey. Similarly, if an administrator want to manually set some confirmed ratings against an application any outstanding detail survey will be withdrawn.

Towards configurable workflows

If we take the above example and also consider some other use-case we can start cataloguing a set of functionality needed to deliver a flexible workflow solution.

jessica-woodland-scott-db commented 2 years ago

There are the following tables in Waltz which could be modified to accommodate these changes:

Most likely by creating a new table to track the state, and renaming entity_workflow_state to entity_workflow_instance

jessica-woodland-scott-db commented 2 years ago

Proposed tables:

workflow_definition

Attribute Type Description
id Long Sequential identifier
name String name
description String description
external_id String external id
created_at Timestamp workflow first created
created_by String user to create workflow
owning_role String which users can modify this workflow
status String ACTIVE / DRAFT / REMOVED

workflow_state

Attribute Type Description
id Long identifier
workflow_defn_id Long FK to workflow_definition
name String name
description String description
external_id String Description
state_kind String PARKED (Did not fulfil any starting predicate) / IN_PROGRESS /COMPLETED

workflow_instance

Attribute Type Description
id Long Sequential identifier
workflow_definition_id Long FK to workflow definition
state_id Long state of this instance
parent_entity_id Long Entity this workflow is acting on
parent_entity_kind String Entity this workflow is acting on
external_id String external id
created_at Timestamp workflow first created
created_by String user to create workflow
last_updated_at Timestamp when was this instance last updated
last_updated_by String user to last update this instance
jessica-woodland-scott-db commented 2 years ago

Example Rules

Records Mgmt

Initialisation

switch 
  case:
    - Pred: App in development && has no survey issued && is not euda
    - State: INITIAL_SURVEY_ISSUED/IN_PROGRESS
    - Side-effect: issueSurvey("INIT")
   default:
     - State: NOT_REQUIRED/PARKED

Transitions

    HasRecords
        - InitialState: INITIAL_SURVEY_ISSUED
        - Pred: hasApprovedSurvey("INIT") && surveyAnswer("INIT", "Q1") == "HAS_RECORDS"
        - State: DETAIL_SUREVY_ISSUED/IN_PROGRESS
        - Side-effect: issueSurveyToAllEntitiesListedInSurvey("DETAIL", "INITIAL", "Q3") and updateAssessment("HAS_RECORDS", "YES")

    DoesNotHaveRecords
        - InitialState: INITIAL_SURVEY_ISSUED
        - Pred: hasApprovedSurvey("INIT") && surveyAnswer("INIT", "Q1") == "DOES_NOT_HAVE_RECORDS"
        - State: NO_RECORDS/COMPLETED
        - Side-effect: updateAssessment("HAS_RECORDS", "NO")

    DetailSurveyComplete
        - InitialState: DETAIL_SUREVY_ISSUED
        - Pred: hasApprovedSurvey("DETAIL")
        - State: DETAIL_SURVEY_COMPLETED/COMPLETED
        - Side-effect: updateRatingsFromSurveyQuestion("RECORDS", "12", "YES")

Arch Gov

Initialisation

switch 
    case:
        - Pred: CI is not retired and hasAssessment("ARCH_SCOPE") == 'IN_SCOPE' (from clarity checks)
        - State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS
        - Side-effect: issueSurvey("DISCOVERY")
    default:
        - State: NOT_REQUIRED/PARKED

Transitions

NoArchChange
    - InitialState: DISCOVERY_SURVEY_ISSUED
    - Pred: hasApprovedSurvey("DISCOVERY") && surveyAnswer("DISCOVERY", "Q1") == "No - No Architectural Change"
    - State: DISCOVER_SURVEY_NO_ARCH_CHANGE/COMPLETED
    - Side-effect: updateAssessment("ARCH_SCOPE", "DESCOPED")

ArchChange
    - InitialState: DISCOVERY_SURVEY_ISSUED
    - Pred: hasApprovedSurvey("DISCOVERY") && surveyAnswer("DISCOVERY", "Q1") == "Yes"
    - State: ARCH_REVIEW_SURVEY_ISSUED/IN_PROGRESS
    - Side-effect: issueSurvey("ARCH_REVIEW")

DiscoveryOverdue
    - InitialState: DISCOVERY_SURVEY_ISSUED
    - Pred: !hasApprovedSurvey("DISCOVERY") && monthsSinceLastUpdate > 6
    - State: DISCOVERY_SURVEY_OVERDUE/IN_PROGRESS
    - Side-effect: issueReminder()

 ArchChangeFromOverdue
    - InitialState: DISCOVERY_SURVEY_OVERDUE
    - Pred: hasApprovedSurvey("DISCOVERY")
    - State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS        

ArchReviewCompleted
    - InitialState: ARCH_REVIEW_SURVEY_ISSUED
    - Pred: hasApprovedSurvey("ARCH_REVIEW")
    - State: ARCH_REVIEW_COMPLETED/COMPLETED
    - Side-effect: updateAssessment("ARCH_SCOPE", getValueFromSurveyQuestion("ARCH_REVIEW", "Q4"))
davidwatkins73 commented 2 years ago

Arch Gov (using context variables)

Context

name lookupFn example
subject this is implicitly available {extId: 'INV123', name: "Move to cloud", status: "ACTIVE" }
scoping_assessment assessment("ARCH_IN_SCOPE") IN_SCOPE
approval_survey survey_info("DISCOVERY") { status: 'APPROVED', lastUpdate: '2021-09-15', ... }
review_survey survey_info("ARCH_REVIEW") { status: 'APPROVED', lastUpdate: '2021-12-25', ... }
scope_answer survey_answer("DISCOVERY", "Q1") 'Yes'
survey_answers survey_answers("DISCOVERY") {Q1: 'Yes'', Q2: ['foo', 'baa'], ,,, }
name lookupRef example
subject this is implicitly available {extId: 'INV123', name: "Move to cloud", status: "ACTIVE" }
scoping_assessment {kind: "assessment", ext: "ARCH_IN_SCOPE"} IN_SCOPE

survey_answers.Q1.value === 'Yes' or scope_answer.value === 'Yes'

The above shows examples of direct vars (i.e. scope_answer) and aggregate vars (survey_answers).
Not sure which will be better yet, suspect direct vars will be clearer but the extra setup may offest this advantage. There is also a tension around performance, aggregate vars are great if lots of parts are used, but are inefficient if only one part is needed.
In opposition direct vars are efficient if only one or two are needed, if lots are needed from the same parent entity then aggregates would probably be better. This may be mitigated by grouping similar lookupFns (like the approach taken in Report Grids)

Using the direct approach, the arch gov definition would look like:

Initialisation

switch 
    case:
        - Pred: subject.status == 'ACTIVE' && scoping_assessment == 'IN_SCOPE' (from clarity checks)
        - State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS
        - Side-effect: issueSurvey("DISCOVERY")
    default:
        - State: NOT_REQUIRED/PARKED

Transitions

NoArchChange
    - InitialState: DISCOVERY_SURVEY_ISSUED
    - Pred: approval_survey.status == 'APPROVED' && scope_answer == "No - No Architectural Change"
    - State: DISCOVER_SURVEY_NO_ARCH_CHANGE/COMPLETED
    - Side-effect: updateAssessment("ARCH_SCOPE", "DESCOPED")

ArchChange
    - InitialState: DISCOVERY_SURVEY_ISSUED
    - Pred: approval_survey.status == 'APPROVED' && scope_answer == "Yes"
    - State: ARCH_REVIEW_SURVEY_ISSUED/IN_PROGRESS
    - Side-effect: issueSurvey("ARCH_REVIEW")

DiscoveryOverdue
    - InitialState: DISCOVERY_SURVEY_ISSUED
    - Pred: !approval_survey.status == 'APPROVED' && approval_survey.issuedOn > fromNow(6, "months")
    - State: DISCOVERY_SURVEY_OVERDUE/IN_PROGRESS
    - Side-effect: issueReminder()

 ArchChangeFromOverdue
    - InitialState: DISCOVERY_SURVEY_OVERDUE
    - Pred: approval_survey.status == 'APPROVED'
    - State: DISCOVERY_SURVEY_ISSUED/IN_PROGRESS        

ArchReviewCompleted
    - InitialState: ARCH_REVIEW_SURVEY_ISSUED
    - Pred: review_survey.status == 'APPROVED'
    - State: ARCH_REVIEW_COMPLETED/COMPLETED
    - Side-effect: updateAssessment("ARCH_SCOPE", getValueFromSurveyQuestion("ARCH_REVIEW", "Q4"))