Core questions/question groups to be used across multiple different survey forms (IDH)

[janagombitova]

Context

More and more we see that organisations are putting effort into understanding how their different projects contribute to their overall goals and impact. This means that organisations have indicators that they monitor across all or multiple projects (horizontal indicators) while projects still track their individual (vertical) ones. This means that some indicators overlap across projects, they must be the same (consistent) to allow for aggregation and disaggregation on the organisational level.

Indicator data often comes from primary data collected in the field, that is then cleaned and aggregated. This data is collected using surveys. Thus indicators are translated into questions (either an indicator matches one questions or an indicator is calculated from multiple question data). For the organisational (horizontal) indicators these questions need to be asked in the same/identical way, with the same question definitions (from question text, to limitations, to order) in order to be consistent and thus to be aggregated on the organisational level.

Why do we have this issue? What are we trying to solve? 

The goal of this change is to:

• make designing a survey form easier - create more reusable components (as we have with cascades, for example)
• ensure data comes in in a consistent way to decrease the time needed when cleaning and analysing the data

How will this benefit the users?

• Decrease time spent on survey creation - making one hurdle smaller to use Flumen and data in projects across an organisation
• Provide an incentive to look at organisational goals and how to use data to track them

How will this benefit Akvo? 

• An improvement to the survey editor will show we are still actively contributing to making it easy to work with data, making Flumen a more attractive tool to use
• More partners will be included to use standard question groups across their forms asking for our advice on the indicator set up and standardization

Metrics We guess that if we provide a way to reuse question groups across different surveys and survey forms these changes will happen by 1 April 2021:

• Because we make it easier to create survey forms:
  • Copying surveys will decrease - do we track this? per instance/ on all?
    • Considering it will be used by 1 Flow instance only but this instance creates most of the forms by copying surveys, we guess a 75% decrease on the instance
    • 5% decrease on all instances
  • Usage of the solution will increase
    • from 0 question groups to 20 per instance
    • from 0 users to 2 per instance
  • We will receive internal requests to use for other partners - from 1 (IDH) to 10 partner instances
• Because we will provide an incentive to look at organisational goals besides project goals:
  • Increase in new visualisations by 5%
  • Increase in views of visualisations and dashboards by 10%
  • Increase in the use of Lumen dashboard filters by 5%
• Because more partners will be included to use standard question groups across their forms:
  • Increase in views on Akvopedia survey library by 15%

Opportunity 

IDH has requested to be able to:

• Create organisational core questions that are used in a unified way across all their project survey forms
• The core questions are created by a limited amount of people
• If a change is made to the core question, it needs to be reflected across every form using it (think of how cascades work - if you change them, the survey form that uses it needs to be re-published)

User story map

https://app.mural.co/t/akvo3857/m/akvo3857/1602234982535/09eb4d0ac66d733f8124fd7492c3303644a45bc3

Current status quo

Currently, when creating survey forms in Flow, you have two options:

• You create a survey form from scratch
• You copy an existing survey form and delete what you do not need and edit what you need to change - this is what our Akvo IDH team does today
• We considered allowing to copy just question groups across survey forms - but never implemented it: https://github.com/akvo/akvo-flow/issues/1293

How are the core questions created today? Carmen has created a master survey based on the core questions defined by Ethel. They also specified the question order to make the survey follow a logical sequence and skip logic is also implemented. Every time a new set of core questions needs to be used by a project (think of a new project starting with a new commodity - some questions though similar need to be adjusted to match that new commodity) Carmen adds them to them the master survey.

How are the core questions used today? The project manager copies the master survey. She deleted the questions she does not need. She adds some project-specific questions to the form. She translates the form. The form is reviewed and then finalised for data collection.

The idea 

The idea is to provide a way to define question groups or questions outside of forms that can be re-used across multiple forms. If a change is made to a core question then this is reflected in each form and the form needs to be republished. Only a few people can have access to these core questions to add and edit them. Anyone who can create and edit a survey form can use all the available core questions per their instance. A core question should not be editable once brought into a form.

Next steps 

Take this as an idea validation - we create it for one partner where our Akvo staff is the user. This way we can quickly get a MVS out (Minimal Viable Solution - what are the most needed components to solve the problem at hand, to allow the user do go through the entire workflow, but with least implementation effort possible), validate and pivot before we decide to go/no-go with the solution to all our users.

As this should be taken as an experiment, we should see how not to implement it inside of Flumen, but outside making it easy to ditch if needed and only enable for specific instances.

[janagombitova]

Today we spoke about the request, what problems are we trying to solve that the current copying of a master survey does not solve and how to keep it to only IDH and Akvo staff use so we can truly treat it as an experiment. Here are the notes:

• problems we are trying to solve: 
  • core questions/groups cannot be edited
  • core questions need to be reusable across any number of forms
  • updating of core questions needs to be reflected across all the forms that use them
• reusable components are what we need
• MVS: https://app.mural.co/t/akvo3857/m/akvo3857/1602234982535/09eb4d0ac66d733f8124fd7492c3303644a45bc3?wid=0-1605536546909
• by tomorrow 16:00 we will create a PoC of the riskiest part - updating the reusable component and making sure the changes are reflected in all the forms. If this works we will review how much effort we need for the rest, if less than 2 week, we go for it.
• we will review with Carmen how the template questions are customised to make sure we understand the need.

[janagombitova]

This comment is to keep track of the decisions, changes and learning points we made in the 1st week of implementing the MVS of this opportunity.

V1 Minimal Viable Solution

The v1 of the Minimal Viable Solution (MVS) is defined in the comment above. The riskiest assumption: updating of core questions needs to be reflected across all the forms that use them —> so we will first see how much effort it is to build this part before we start building the solution.

Pivot from V1

After discussing with Carmen our idea of the solution and the riskiest part that we will try to build first to learn how much time the entire solution needs, we learned:

• There are 2 user problems:
  • Data consistency problem - The problem we are addressing - across all IDH projects data needs to be collected in the same way to allow it to be compared on an overall IDH level and across different projects. This means some questions must be asked and defined the same way across all survey forms, other questions must be customised but still follow the IDH template
  • Creating a survey is a hassle and takes a lot of time - this problem we are not directly addressing these changes as different solutions might be needed
• Our assumption that core questions/groups cannot be edited —> partially true —> Some question groups in the Master survey are strictly defined and nothing can be changed there. But there are some question groups where some questions cannot be changed but some questions have to be customised for each project.
• Assumption core questions need to be reusable across any number of forms —> true
• Assumption - and most risky part - updating of core questions needs to be reflected across all the forms that use them —> Not true:
  • It rarely happens that a question group that is defined in the Master survey is changed. Before these questions are digitised they are reviewed multiple times so once they are put in Flow, we can strongly assume they are there for good and they will not change.
  • If it does happen, then for the survey forms that already have data to the form and the data collection is no longer ongoing, the survey form cannot change. When analysing the data you always look at the survey form definition to map the data to how it was collected. If the form version changes from the version that was used for data collection, you cannot understand the data structure.
  • If it does happen and survey forms are just starting with data collection, then these changes need to be reflected in the form, but data cleaning might also need to happen so it needs to be handled by a human.
  • If it does happen and we are about to start a new project, then a new copy of the master survey is made

With this new information, we realised our riskiest assumption is not even true and not needed in the solution. Thus we pivoted our idea.

V2 Minimal Viable Solution

• Carmen defines the template survey
• In Flow we hardcode which survey this is using the survey ID
• When starting a new IDH project Mercy and Mulo select which question groups from the template survey they need, change the questions that need to be changed, create the new survey.
• In the new survey the question groups (and thus the questions) cannot be edited
• As we want to minimise the number of changes that we put into Flow (as we are treating this as an experiment to be used by 1 partner only), we will build the solution outside of Flow

The riskiest assumption:

• questions and question groups can be set as immutable
• building this outside of Flow will take less effort and allow us more flexibility

The 1st thing before we start will be to check the 2 assumptions above.

Pivot from V2

• We checked if questions and question groups can be set as immutable —> We already have a flag we can use on questions, we need to add the same to the questions groups - minimal effort
• We checked building this outside of Flow will take less effort and allow us more flexibility —> in this case user authentication is the riskiest part as Flow’s auth is based on sessions
• We then looked at how we can build V2 and realised that if we could
 * 1) build it separately from Flow but deployed within Flow
  • 2) within Flow
  • 3) outside of Flow
• When estimating what needs to be done for all three options we realised that regardless of the choice, we need to make question groups and questions created by copying a template immutable. This is a change that overlaps all three options. Furthermore, this change addressed the core data consistency problem.

So we decided to slice out the work even further a redefined our MVS to v3, where v3 only focuses on handling the data consistency problem and not making survey design simpler.

V3 Minimal Viable Solution

• Hard code in the config file of the IDH instance which survey is considered a template
• If this template is copied none of the questions groups can be edited, only deleted
• Questions within question groups cannot be touched at all
• New question groups with questions can be added anywhere in the form

All work on V3 is linked to this issue/epic.

I will keep the epic closed, but use it to track the decision we make around this opportunity and the next steps taken, and link the smaller issues here. Why? I am testing how the epic feature works across Github and Zenhub.