Data approval workflow (MVP)

mtwestra commented 9 years ago

There is a need to implement an approval workflow in FLOW. This means that: 1) on the survey level, there should be a checkbox 'Data needs approval' 2) When this is checked, raw data that comes in will be in an 'unapproved' state. Users with the right permissions can approve them individually, or in bulk.

mtwestra commented 9 years ago

Perhaps this should be a setting on the survey?

Needs to have a related permission as well

muloem commented 9 years ago

Interestingly enough this feature used to be present and we deprecated it. I think users felt it got in the way but perhaps @iperdomo has better context as to why.

mtwestra commented 8 years ago

Proposal for technical design: https://github.com/akvo/akvo-product-design/blob/17c5f9d207f34ab034902ce087dec5c5b624b4dd/FLOW/Features/82-approval-workflow/TechnicalDesign.md

janagombitova commented 8 years ago

Here you can track the current discussion on the implementation from the users' point of view as well as user stories and background information:

User stories and scenarios document https://docs.google.com/document/d/1BKJCBYmlgrgVcJ4W7FzLLQQCxPJVm-A0NofiAW4pZVY/edit

Feature definition document https://docs.google.com/document/d/1ladrDGFMizo1bXS9r2W9iv6og-mCUb8uBIppHgu6jeA/edit

mtwestra commented 8 years ago

More inspiration: How sharepoint handles 'approval workflow': https://support.office.com/en-us/article/All-about-Approval-workflows-078c5a89-821f-44a9-9530-40bb34f9f742

http://www.cardinalpointsolutions.com/PDFs/Taking%20the%20Work%20out%20of%20Workflow%20with%20PeopleSoft%20Approval%20Framework.pdf

https://docs.oracle.com/cd/E13228_01/fscm9pbr0/eng/psbooks/scin/chapter.htm?File=scin/htm/scin03.htm

All of these use the concept of actions that the approver takes: "approve, deny, hold and push pack", or " approve or reject the item, to request a change in the item, to reassign the task, or to cancel or delete the task". From the mockups by Kiarii, this might be something useful

mtwestra commented 8 years ago

Following the mockups by Kiarii, here is a proposal for the technical implementation.

Mockups:

Data structure

1) introduce a 'workflowGroup' kind, with properties:

defaults: id, lastUpdatedDate, createdDate
displayName: text field

2) introduce a 'workflowItem' kind, with properties

defaults: id, lastUpdatedDate, createdDate
workflowGroupId: foreign key to workflow group

3) introduce a 'workflowItemHistory' kind, with properties

defaults: id, lastUpdatedDate, createdDate
workflowGroup id: foreign key to workflow group
workflowItemId: foreign key to workflow item
timestamp: unix timestamp
comment: text field

4) introduce a 'workflowPermission' kind, with properties:

defaults: id, lastUpdatedDate, createdDate
userId: foreign key to user
surveyGroupId: foreign key to surveyGroup
workflowItemId: foreign key to workflowItem

5) add a 'workflowEnabled' flag to the surveyGroup kind 6) add a 'workflowGroup' field to the surveyGroup kind 7) add a 'workflowItemId' field to the datapoint kind (surveyedLocale kind): foreign key to the workflowItem 8) add a 'workflowStatusId' field to the datapoint kind: Enum with types "APPROVED, REJECTED, PENDING"

Dashboard

In the survey advanced settings section, add a checkbox 'enable workflow features', and a dropdown allowing the choice of which workflow to use
When a workflow is chosen, display the workflow steps, with an action 'users responsible'
When this action on a particular workflow step is clicked, the (admin) user can add users to a workflow step
In the datapoint list, add a column 'workflow status', displaying the currently active status, as shown in the mockup
When the 'workflow status' item in the header is clicked, a popup is shown where filters on the column can be managed, as shown in the popup
When a workflow status item in one of the data points in the list is clicked, a popup is shown with the workflow history, and the ability to select a new status and add a comment, as shown in the mockup

Backend

In addition to adding the new kinds described above:

add a permission to 'manage workflows', which is on by default for the admin role and off for the others
In the report exports, export the workflow status as a single column, combining the workflow item and the status, so 'Financial Audit - APPROVED'.
In the report imports, ignore this column

janagombitova commented 8 years ago

Functional design document

https://github.com/akvo/akvo-product-design/blob/f8fa0af5be2e1337b74f1cb00ffc926751fa3554/FLOW/Features/82-approval-workflow/FunctionalDesign%20DRAFT.md

Mockups (most current version):

ordered workflow - https://drive.google.com/open?id=0B53AXQ5Y0VCMUGE1d3RfRlJfbXM
unordered workflow - https://drive.google.com/open?id=0B53AXQ5Y0VCMSlZreXh3RS1CZjA
search and filters in the new Data tab which replaces the current Inspect data tab and Monitoring tab - https://drive.google.com/open?id=0B53AXQ5Y0VCMU2JjRTB6NHB2cFk

janagombitova commented 8 years ago

Review of initial implementation and proposed changes

This feature is defined and part of our contractual obligations to SNV together with webforms. The initial planning was to deliver the features in the beginning of 2016 and now the deadline is moved to June 2016. Together with @muloem @mtwestra and @LarsSjogreen we looked at the planning and available resources and agreed that we will not be able to deliver both Webforms and Workflow at the same time. As the workflow has higher priority for SNV programmes (and the functionality around webforms can still be done with the app) we have decided to change the planning and focus first on workflow.

Secondly, we have decided to review the design created by @Kiarii and define a MVP which will not involve multiple UI changes to the dashboard, reuse as many existing components as possible and implement the functionality in the current dashboard structures. We will implement the designed mockups in later iterations.

Changes to original feature

approval workflow will be implemented first as a feature flag and will be enabled on the biogas and cookstoves dashboards only
it will only be possible to use this feature with monitored surveys as the use cases provided by SNV are mostly based on using monitoring surveys (approving the data while collecting different sets of information is other forms linking to the same entity)
when adding a user to a workflow step, the manager can only add one user to one step and not multiple users as proposed previously
approval workflow feature will exist in the current Monitoring tab
we will not merge the Inspect data tab and Monitoring tab, nor create a Library tab with this MVP
it will not be possible to add a code to a workflow step
created workflows will not be shown in a list with further details, but in a dropdown (as cascades are selected in the Cascade resources tab)
no 'view workflow' page, the user will get directly to the edit workflow page
it will only be possible to apply to workflow steps on a data point in the list of points per survey, not when viewing the forms submitted for a particular point
no filtering will be added to the data tabs (following release after the mvp)
no special report template will be created to shown the changes in the workflow status for data point (following implementation step)

Implementation steps on the MVP:

[x] add new 'Workflow' tab under Data tab, next to Cascade resources
[x] create, edit, save, delete and view a workflow (as proposed in the mockups) reusing the way we create option questions or cascade questions
[x] In the survey advanced settings section, add a checkbox 'enable workflow features', and a dropdown allowing the choice of which workflow to use
[x] When a workflow is chosen for survey, display the workflow steps, with an action 'users responsible
[x] in current Monitoring tab add new column Workflow status (validation that this column will only be shown for surveys that have a workflow added to their definition) located between Last update and Action
[x] add in page pop up which enables to view workflow steps, to change and to save workflow step in the table (taking into consideration that the user has to have the permission to do so)
[x] save the time stamp and user name of who made the change of workflow status and show it in the workflow change and preview in page pop up
[x] add a permission to 'manage workflows, which is on my default for the admin role and off for other
[x] In the report exports, export the workflow status as a single column, combining the workflow item and the status, so 'Financial Audit - APPROVED'."
[x] In the report imports, ignore this column

Existing documentation on the "original" feature

User stories and scenarios https://docs.google.com/document/d/1BKJCBYmlgrgVcJ4W7FzLLQQCxPJVm-A0NofiAW4pZVY/edit
Feature definition discussion https://docs.google.com/document/d/1ladrDGFMizo1bXS9r2W9iv6og-mCUb8uBIppHgu6jeA/edit
Mockups feedback https://docs.google.com/document/d/1aIxMnMP4SFePy4uvkUtZp6yfjKSoRFMur-KgiD9wNPc/edit
Functional design document https://github.com/akvo/akvo-product-design/blob/f8fa0af5be2e1337b74f1cb00ffc926751fa3554/FLOW/Features/82-approval-workflow/FunctionalDesign%20DRAFT.md
Mockups (most current version):
- ordered workflow - https://drive.google.com/open?id=0B53AXQ5Y0VCMUGE1d3RfRlJfbXM
- unordered workflow - https://drive.google.com/open?id=0B53AXQ5Y0VCMSlZreXh3RS1CZjA
- search and filters in the new Data tab which replaces the current Inspect data tab and Monitoring tab - https://drive.google.com/open?id=0B53AXQ5Y0VCMU2JjRTB6NHB2cFk

janagombitova commented 7 years ago

The agreed upon requirements for this functionality have been finished with 1.9.17 release. The implementation is currently released to two partner instances and is being tested. Around July/August we will review the implementation to learn:

how is it used in their projects?
what is the benefit of having this functionality
what is missing
to be defined

After that we will review and decide on the next steps that will be covered in a separate issue.

akvo / akvo-product-design