Releasing new dataset grid Data groups & support for RQGs

[janagombitova]

Context

Under the RQGs feature flag, we have a UI change - Data groups + walkthrough that changes how the data set looks in the UI. It breaks the dataset into Data groups and the user sees only the columns for the selected data group. There is a new side panel that holds the data groups. This change only applies to Flow datasets.

We aim to release small changes frequently so users do not even have the chance to notice that something has changed. But this change is noticeable especially because the dataset will look differently depending on its source. Secondly, it will allow to work with repeated groups what has been missing in Lumen. But due to the nature of RQG data, we will impose restrictions on transformations and visualisations.

Opportunity

The idea is to release the change to all users to be able to opt-in. Then after a period of time (one or two weeks for example), set the change as default and allow them to opt-out. Again after a decided period of time, the new UI will be the default and the old one removed.

This idea also comes from this document describing how we release. The entire workflow could be:

• have the changes behind the feature flag, including the initial opt-in
• enable the changes on demo for internal testing
• remove feature flag and release to all tenants

Next steps

As this way of releasing will require that we handle backwards compatibility well, we need to understand what change is needed and how much effort it is to see if RQGs is a feasible case to do so, or not before we make the final decision. So, the next step is to discuss the technical feasibility with the team.

[janagombitova]

Before we proceed with this issue, I want us to first discuss this in the team to see if this is the good case to try out this way of releasing and if it is feasible. If we decide to proceed we can reopen the issue.

Currently our release of this feature follows to steps we also did recently with OPTION and legend order:

• having all changes (+ walkthrough) on a feature flag
• enable the feature flag on demo (and perhaps a few more internally used instances) for internal testing
• after successful internal testing, remove the flag and push the change (+ walkthrough) to all users

[tangrammer]

WIP TODO

• opt-in opt-off

• foreign references??

• define which kind of transformations are possible between different data groups

• define which kind of settings could be available in the different vizs

• keep transformations index

[janagombitova]

@tangrammer some notes from the decision we made on 24 June: https://github.com/akvo/akvo-lumen/issues/2127#issuecomment-648723557. I am not saying we should not revisit them, just wanted to point out that we already defined where we start with the implementation, to help with the decisions related to how to release this monster

define which kind of transformations are possible between different data group

We decided that transformation between "regular" data groups should be possible, as they are today. But for transformations that use more than one column and one of the columns is from a repeated group, the user can only select the 2nd column from that same repeated group. We even have some work done to support this behind the feature flag: https://github.com/akvo/akvo-lumen/issues/2741

define which kind of settings could be available in the different vizs

Here we decided: To keep things small and to ensure users do not make visualisations that do not work, we can start by not allowing to visualise data outside of the RQG - you cannot select a column from a different data group if you selected a column from a RQG data group. We can then start enabling comparing RQG data with other columns from other groups, one by one after the main release.

[tangrammer]

column (and derivation column) names

• column names integrity is preserved by flow
• derived columns indexing need to be by dataset (instead of by dataset_version) to avoid conflicts

[tangrammer]

backward compatibility transformation log

• decide in which dataset_version(s) are going to be stored
  • can we delay this change and choose the easy impl so far? (storing all txs in main namespace)
  • if different ds then keep order based on index or time

[janagombitova]

We decided to first release the user visible new UI grid with data groups. This will be released with the walkthrough and an option to opt-out for the user. The opt-out should be removed after a set period of time. Then we will release the RQG changes to all users once ready as we did previous releases (test with feature flag, remove flag, push to all). So from this point of view this issue can be closed.

[janagombitova]

@tangrammer if you want to kee the issue, please explain and reopen ;)