Add codes to cascade questions

[ichinaski]

Overview

Cascade resources do store codes already in GAE. However this properties are not reflected in the serialised version of the resource (SQLite database), therefore the app cannot submit the code as the value.

Actions:

• All the required data is already present in the datastore, thus is up to FLOW services to add the codes in the SQLite database. In this step no action is needed neither in GAE nor Dashboard.
• The most critical part is to determine the new response serialization format, since response will potentially include both the code and the value (for each cascade level). This decision needs to be aligned with the app implementation.
• Store new values. The current cascade responses, if I'm not mistaken, are fully stored as a single value in the datastore. Depending on the previously mentioned format, some further action will be needed at this stage.
• Data reports. Same as above: the required actions will be dependent on the serialization format we agree on.

Format options:

• Although most response types are held in a single property (string serialized value), this code is an interesting attribute, since it will be shared by other response types as well (option codes to begin with, but could be extended as product needs evolve). Taking this into account, adding a new property to the Response objects could be very useful, as the codes storage will be homogeneous among all response types. However, the cascade responses contain the whole path value, thus multiple codes will be needed. This slightly complicates this approach. Option question can contain multiple values (and therefore codes) too, when the multiple responses flag is enabled. Should we take this approach further, it'd make sense to use code arrays, instead of single codes, per response.
• Packed code-values pairs. As opposed to adding new properties, we can extend the current paradigm of packing values, either with JSON-like structures, or with pipe separated strings. Even though the JSON version is more flexible, it will require more time to visualize data properly, as every single view (dashboard, reports, etc) will need to accordingly parse and display a human-readable version of the response.

[janagombitova]

Duplicate of #1209

[ichinaski]

Sample response representing the first approach (code property):

// Cascade response

{
    "questionId": 0001,
    "answerType": "CASCADE",
    "value": "United Kingdom|England|Bristol",
    "code": "acb|def|ghi"
}

// Option question (single response)

{
    "questionId": 0002,
    "answerType": "OPTION",
    "value": "United Kingdom",
    "code": "acb"
}

// Option question (multiple responses)

{
    "questionId": 0003,
    "answerType": "OPTION",
    "value": "United Kingdom|Netherlands|Germany",
    "code": "acb|xyz|jkl"
}

[ichinaski]

The decision we have taken is to use JSON-encoded values for complex responses (and eventually, everywhere): https://github.com/akvo/akvo-product-design/blob/master/FLOW/arch/ADR-004.md

[janagombitova]

Related issue regarding how we visualize codes in Reports: https://github.com/akvo/akvo-flow/issues/1230

[janagombitova]

Suggestion on how to visualize codes + cascades in raw data report

Suggestion for exporting cascades with codes in raw data report. Each cascade level will sit in its own cell/ column and the code:value formatting will copy the decision made with Codes for Option questions. If we put the entire cascade path with the codes in one cell, it may become too condensed and unreadable for the users.

[iperdomo]

@janagombitova remember that RAW DATA serves two purposes, there is the need to use it for data cleaning, so having everything packed in one cell is machine readable

[mtwestra]

@janagombitova, @iperdomo , I think we need to have two versions: one raw data report, which has everything in one column, and one 'export' version, which separates everything in columns.

I would make the column headers specific, so location_state_code, location_state, location_region_code, location_region, location_district_code, location_district, etc. Same for codes with options: someoption_code, someoption

[janagombitova]

@mtwestra @iperdomo @jonase @ichinaski @muloem

Based on your comments I suggest we can implement the reports work in two batches:

Implementation 1 - raw data report - everything in one cell, with reusing the proposed structure for options code1:value1|code2:value2|code3:value3 in the code+cascade as well: statecode:state|regioncode:region|districtcode:distrtict

Implementation 2 - Create either a report setting or a new type of report (based on UX) which will separate items into different columns. However, there are again multiple options how to approach this:

I personally prefer: Code in one cell, Value in one cell + multi select (vertical), because if we divide items in separate columns, we should divide everything and the vertical view is consistent with our decision on repeated question groups.

[rumca]

I think the test plans for this are captured in issue number 317 on the mobile repo, but just to confirm that I see what I think is the correct new format in reports as below:

(showing one old response and one new one)

I'm going to close this issue and if we need to make future enhancements to the reports we can do so against a new issue :+1:

[jonase]

TestPlan

Test Plan for cascade questions with codes support in the raw data reports

• Create two different cascade resources. One with codes and one without codes
• Create a form with different question types including two cascade questions which uses the previously created resources
• Submit a few responses with the device
• Go to "Reports > Export Reports" and select the folder/survey/form
• Generate two different reports:
  • One report with the "Export this report for use with an external system" setting enabled (found under "Show advanced settings"
  • One report without the above setting enabled (which we later can use for data cleaning)

The report with "Export this report for use with an external system"

• The cascade question response values should be split into several columns (one per cascade level)
• If the cascade has codes:

+-------------+-------------+-------------+
| code1:text1 | code2:text2 | code3:text3 |
+-------------+-------------+-------------+

• If the cascade does not have codes:

+-------+-------+-------+
| text1 | text2 | text3 |
+-------+-------+-------+

The report without "Export this report for use with an external system"

• The cascade question response values should be in a single column
• If the cascade has codes:

+-----------------------------------------+
| code1:text1 | code2:text2 | code3:text3 |
+-----------------------------------------+

• If the cascade does not have codes:

+-----------------------+
| text1 | text2 | text3 |
+-----------------------+

Data cleaning

• For the data cleanable report, change some of the cascade responses (both types: with/without codes)
• Upload the the report with the "cleaned" data
• Check the data on the dashboards "Inspect Data" tab. The changes you made should be visible there.
• Check the datastore to verify that the uploaded cascade responses (in the QuestionAnswerStore) are encoded as JSON strings

[rumca]

@jonase interestingly enough it's not actually possible to create a cascade without code properties set as far as I am aware. If there isn't one explicitly set, then it's set to the same value as the name. At least that's the case for all the cascades I've created manually and via the importer.

Otherwise this test plan passes as described :+1:

[janagombitova]

@jonas, is this expected (Ruarcc's message) and is this visible to the users? I though we said, that if the user does not create a code for any value, then he does not see the code (anywhere - app, dashboard, reports)

[jonase]

@janagombitova @rumca I might have confused cascade & option codes here. Currently the implementation assumes:

• Old responses does not include codes, and when we reimport we will use the new format but with missing code. So if the original response was Finland|Vasa we will upload it as [{"name": "Finland"}, {"name": "Vasa"}]. (i.e., no code).
• New responses can have codes or not (and this is where I might be confused). With codes could look like [{"name": "Finland", "code": "FI"}, {"name": "Vasa", "code": "VA"}]. This will render in the excel cell as FI:Finland|VA:Vasa. Without code it would simply render as Finland|Vasa. If the code == name in the response we will render as Finland:Finland|Vasa:Vasa which I suppose is not what we want. @ichinaski in the new format, if the code is missing (can it even be missing?) will the response be [{"name": "Finland"}, {"name": "Vasa"}] or [{"name": "Finland", "code": "FI"}, {"name": "Vasa", "code": "VA"}]?

[janagombitova]

@jonas, so the user does not see a code, if he did not add one. If so, then good. I thought @rumca implied that codes are always visible (if created by user, or not - copy of the value name)

[ichinaski]

in the new format, if the code is missing (can it even be missing?) will the response be [{"name": "Finland"}, {"name": "Vasa"}] or [{"name": "Finland", "code": "FI"}, {"name": "Vasa", "code": "VA"}]?

The former. If the option has no codes, the app will not assign one by default. Note however, that the current value/text implementation (code/text as of now), does assign a default value (code), which results in the form XML being fully populated, that is never having missing values (codes). I believe this is something we'll have to adjust in the backend implementation for cases wherein the user provides no codes.

[rumca]

In my experience it's not possible to not see a code, if you don't explicitly add one then the name is used as the code and that's what is populated in the report.

[janagombitova]

Then yes, this needs to be changed

[rumca]

code no longer displays if it's the same as the name :+1: