Visualization of frequencies for multiple option questions

[muloem]

Context

We aim to make working across Flow and Lumen as smooth as possible what means Lumen should be able to handle Flow responses i.e. correctly display, transform and visualize them in a manner that is simple and as straightforward as possible for our users. In this issue, we look at a particular type of response; those from multiple option questions in Flow.

This is a building block for the implementation of multiple option question support in Lumen. For wider context see the original issue https://github.com/akvo/akvo-lumen/issues/2225

Problem or idea

Currently, in Lumen, multiple option questions are imported into a single column with the selected options separated by a pipe: Example orange|red|yellow. If Salim attempts to create a visualization for this column, say a bar chart, Lumen currently treats the entire column as a single response and so the number of possible combinations of responses results in categories that are not meaningful. See the example chart below.

Solution or next step

As a first step, we would like to introduce support for visualizing frequencies of selected responses to multiple options questions. To keep things simple, this visualization should be based on the single column corresponding to MOQ responses, without the need to perform any special splitting or transformation of data. The categories on the X-axis should correspond to the options available for the multiple option question. e.g. see the chart below based on the same data as the previous chart above.

[Kiarii]

This means that for the first iteration, we will support the analysis of MOQ data. On the grid, the current representation of MOQ data will be maintained: A row’s value contains only selected option names/labels.

On the analysis side, Lumen should able to perform metric calculation of respective options and the resulting bar chart (we will start with this for the first iteration) is auto-populated with the options and respective metric.

the Google Sheets example below shows what we are aiming for for the first iteration

Other considerations;

• Depending on the ease of implementation, we will support sub-bucketing
• We will disable metric col support at least for the first iteration,
• Create an onboarding screen to notify the user of the ability to analyze MOQ data out of the box

In later iterations, based on user feedback and usage stats, we might consider

• Providing the ability to hide some options; for now we think users can use the "Number of buckets to show" option to minimize the number of options in view, in cases where free text is used
• Auto-suggesting cluster & edit, in cases of free text entry
• (Auto) splitting MOQ col into children cols.. which could be visualized using the metric col option at least for the bar charts

[tangrammer]

current research:

• flow survey: uat1.akvoflow.org > _Lumen tests > Example dataset (for testing and demos)
• MOQs: flow column

  {:groupId "550809118", :name "What vegetables to you grow? ", 
  :groupName "Agri", :createdAt "2019-08-05T12:30:56.369Z", 
  :type "OPTION", :variableName nil, 
  :id "536739116", :order 3, :modifiedAt "2019-08-06T08:50:14.771Z"}

• lumen db column type

  {
  "key": false,
  "sort": null,
  "type": "text",
  "title": "Do you grow your own fruit and/or vegetables for your own consumption?",
  "hidden": false,
  "groupId": "550809118",
  "direction": null,
  "groupName": "Agri",
  "columnName": "c544429116",
  "multipleId": null,
  "multipleType": null
  }

So, currently lumen receives an OPTION question that is stored as TEXT type in lumen. Then to offer this functionality (once the changes were merged) we need a new imported dataset and/or update a current one

[janagombitova]

Making bar charts work like a charm! Well done!

I did notice one thing we need to see how to handle. Currently, you can only use bar charts to create visualisations from the option data type - that is ok for data where there is more than one option in the cell. But now all option questions are marked as option datatype. So also a question like 'Gender' where the answer is either 'male' or 'female'. This column is now also set as option, but I cannot create a pie chart as the column type is not supported. Either we need to:
1) Enable support for option type for other visualisations where text is also allowed 2) Or we need to enable switching the column type from option to text and other way around.

I would vote for the 1st option.

[tangrammer]

yep! we need to update the visualizations conditions menu configs to adapt to new OPTION type

[janagombitova]

@tangrammer GREAT!

[janagombitova]

@tangrammer to add to the comment above, we also should allow transformations to work with the Option data type if they work with text data

[janagombitova]

Transforming OPTION columns works well.

I can also make visualisations with this column type, but if my OPTION is a multiple-answer-option-question I expect all the visualisations to handle the categories as the bar chart does. So I expect to have the categories grouped to show the frequency of each different value.

But for pie, polar, donut, bubble, map and scatter we currently these take each answer combination as a unique value, as before.

Here is an example of a map: https://lumen.akvotest.org/visualisation/5f3b92d5-a36f-4447-9048-1942cb794dce Here is an example of a bar chart, same dataset, same column as in the map but the values grouped the way expected: https://lumen.akvotest.org/visualisation/5f3b9433-551b-4010-b122-1386a5c1d91d

[janagombitova]

@tangrammer, @marvinkome and @kardan I found one more bug:

When defining any visualisation you can filter the data. Filtering on a TEXT or NUM column works perfectly. However, when I select an OPTION column to filter the data I get a blank page and the error below. I would expect that the OPTION data type works the same as TEXT in this case too.

[janagombitova]

I tested the OPTION functionality and the core of it works really well (besides maps as those are still WIP):

• Transformations work well
• Filtering on the dataset in the dataset works well - I found one issue, but it is not a commonly done action, so we can resolve it after the release - https://github.com/akvo/akvo-lumen/issues/2826
• Visualisations work well - scatter too, just the colours of the legend do not match the points, but I assume that is because of this issue: https://github.com/akvo/akvo-lumen/issues/625#issuecomment-675426318
• sharing and exporting works well too
• adding to dashboards and filtering the visuals on another column too

To conclude, besides the issue mentioned below and maps we are good to go 👍

~## issue found that needs attention~

~Stacking bars with Option data works well, but you cannot switch from split to stack/stack percentage ~

[tangrammer]

@janagombitova "Stacking bars with Option data works well, but you cannot switch from split to stack/stack percentage" seems like a different issue, doesn't it?

[janagombitova]

@tangrammer I wonder if it is related to these changes as if I have a bar chart that does not use an OPTION column, stacking and switching from stacked to percentage stack or split works fine. But we can handle it separately if that suits you better

[janagombitova]

Support documentation all set: https://lumensupport.akvo.org/article/show/114426-coming-up-next-how-can-i-analyse-data-from-multiple-answer-questions

Measuring success

Our goal is to make working across Flow and Lumen as smooth as possible. With this issue, we assume that the use of the derived js transformation will decrease as users will no longer need to handle MAOQs with js code but will use the OPTION data type. To measure the success of this change we can track the use of Derived column js overtime and assume to see a decrease in its use.

Baseline:

We assume the frequency of use of derive column will drop and decrease from the current 10% of unique transformation events to 5% by the end of 2020.

[janagombitova]

After giving all the issues one more go I decided that we handle the stacking problem as a separate issue, it does not block the release.

So this is good to go! YaaaY!

[janagombitova]

Testing on dark-demo I did not find any other issues. We are good to go and will tackle any bugs users find once reported.

[janagombitova]

Actually, the maps don't group the Options on dark-demo as they did before. Example here: https://dark-demo.akvolumen.org/visualisation/5f3e4384-9579-469b-ad5c-5bdd5d0cdf2b

[janagombitova]

I have been testing all different combinations and besides the few separate issues we handled today, all works fine. So I am going to close this issue.

On the other hand, we just spoke with Juan that the maps, if working with multiple answer option question work well but are misleading. We will create a separate issue to look into alternative solutions.