Adding label-value to questions options

[mtwestra]

Overview

The aim is to add a value to question options, in the same way as HTML options have a label and a value. The main purpose for this is to make it easier to import FLOW data into existing databases. The label would be used for display, the value will be used for in API's or optionally in data exports.

In many cases, existing databases use coding schemes to codify answers to questions. For example, 'yes' might be 1, and 'no' might be 0. Another example could be 'banana'=01, 'orange'=02, 'grapefruit'=03.

Links to documents

https://github.com/akvo/akvo-product-design/blob/master/FLOW/Features/34-LabelValue/FunctionalDesign.md https://github.com/akvo/akvo-product-design/blob/master/FLOW/Features/34-LabelValue/VisualDesign.md https://github.com/akvo/akvo-product-design/blob/master/FLOW/Features/34-LabelValue/TechnicalDesign.md

[mtwestra]

Proposed user interface: similar to Trello: 1) you can add individual items as options 2) you can also add a list of items, which are then turned into a list of individual items 3) you should be able to change the order of the items 4) when a list item is created, you should be able to add a value to it on the right hand side 5) there should be an 'x' behind each item, so they can be deleted.

[mtwestra]

Changing dependent question identification

At the moment, we have this situation: 1) an option question has different options, where each option is an object in the datastore, with an id, a display text and a code 2) when the list of options in the option question is changed, the changed list is compared to the old list by display text. If the value is not found, a new option is created. This has the problem that when an option display text is only slightly changed, for example from 'yes' to 'Yes', a new option is created, breaking the reference link to this option in dependent questions and translations. 3) Dependent questions refer to options in other questions by display text 4) translations of options refer to the options by id 5) In the survey xml, the options are serialised with their display text and code. 6) when the device serializes data, it serializes only the code.

As we are moving towards having codes for options, we need to change the UI for option creation towards having a list of separate items, each with a code and a text. In this case, it is easier to keep the identity of the item, when then text changes for example. Therefore, the proposal is to move to the following situation in refering to / dealing with options

1) in the UI, we create a list of options, each with a code and a text. New options are added by an 'add' button. Existing options can be edited by changing the values in the textfields. This means that an option keeps its identity, unless it is deleted explicitly. For example, if a user creates an option with display text 'yes', and later changes this to 'Yes', the option id will be the same, but the display text will be different. 2) in dependent questions, we will add a field 'dependentQuestionOptionId', which is a list of the ids of the question options which should trigger this question. In that way, we refer to options by id instead of by display text. 3) In the survey xml, we serialize the ids of the options in addition to the code and display text. In addition, for the dependent question, we serialise the list of option ids which should cause this question to display 4) In the answer value, the device should serialize both the code, the display text, the translated display text if it was displayed, and the language code. 5) In reports, people can choose to either show the code or the display text, or both.

In this way, both dependent questions and translations refer to options by id instead of by display text

[iperdomo]

• I guess we'll reuse the code property for QuestionOption kind, right?
• Should we allow the user changing the value of the code property when data has been collected?
• When showing the list of possible option dependent values , we should show also the display text and not only the code,

    1 | Male
    0 | Female 

• Not sure if we're handling it already. What happens when if a Question on which we depend changes its order and is now after the dependent one?

4) In the answer value, the device should serialize both the code, the display text, the translated display text if it was displayed, and the language code. Is not only the code|display text on the Survey base language the one we need?

[ichinaski]

In the answer value, the device should serialize both the code, the display text, the translated display text if it was displayed, and the language code.

Storing and serializing the set of translated values in the app is going to be somewhat challenging. How would the server handle that data? AFAIK what we really are interested in is the option ID or code the user has selected, right? I fail to see how the displayed texts are relevant.

Also, what if the user updates the languages in the middle of an ongoing data collection, wherein some option questions have been already responded?

[mtwestra]

@iperdomo on your questions: 1) yes, the idea is to reuse the 'code' property 2) I think the code property could change after data has been collected - it should be the responsibility of the user, I think. Another question is if we should allow data cleaning on the code - I guess so... 3) yes, in dependencies we should show both the display text and the code 4) The case of a moving question is not handled yet. In fact, I was thinking perhaps of a 'integrity check', which could be run before form is published. It could check for a) questions dependent on a later question, b) faulty dependencies, c) missing translations, etc. 5) Yes, I guess the code + display text in base language should be enough.

[mtwestra]

@ichinaski I think it might indeed be enough to serialize the code + display text. The only question I have is if it might be good to also serialize the option id, so we could export reports in the translated languages easier, as we could retrieve the translations through the option Id (otherwise we'd need to match them somehow)

[mtwestra]

Could reuse the cascading question UI

[Kiarii]

Wireframe illustration of Adding label-value (code) to questions options functionality:

[janagombitova]

Summary of decisions made based on the discussion 30 July. For the upcoming implementation we will:

1. When creating a new option question:
   • always provide the user with the text field for a code and a value
   • codes are not mandatory
   • if the user fills in a code, he has to create a code for all options
2. In reports:
   • if the user added codes to options then show in one cell
     • for a single selected option: code:value
     • for multiple options (pipe separated): code1:value1|code2:value2
   • if the user did not add codes to option, then show:
     • for single selected option: value
     • for multiple options: value1|value2

Additional question In Data/Inspect data tab and Monitoring tab can we not reuse the pipe separation for multiple options, but present each option in a different line? 01:Joe 02:Kate 03:James

[ichinaski]

After a follow up chat with @muloem, we agreed on the following names for options:

• Text: Option label
• Code: Unique value, optional. This property is called value in existing surveys.

Therefore the survey XML definition will be as follows:

<options allowOther="false" allowMultiple="false" renderType="radio">
    <option code="UK">
        <text>United Kingdom</text>
    </option>
    <option code="ES">
        <text>Spain</text>
    </option>
</options>

And the corresponding response:

{
    "code": "UK",
    "text": "United Kingdom"
}

[ichinaski]

Update:

Discussed with @muloem the complexity of having JSON-encoded responses for dependecies management, and after considering the minimal added value this would have overall, we decided the following:

• For each option, you have a text (mandatory) and a code. If the code is not set, it acquires the same value as the text.
• The code becomes the unique identifier within the question.
• If the code is different from the text, the app UI displays both:
  • Y - Yes
  • N- No
• If the code is equal to the text, only the text (or code) is shown:
  • Monday
  • Tuesday
  • Wednesday
  • ...
• It could be the case that some options have code and some don't. I believe is up to the user to ensure data integrity and consistency.
• Dependencies are established the same way, just storing the code instead of the text (It might be the case that we are doing this already). Dependencies on multiple values use pipe-separated format.
• Responses will keep using the pipe-separated format, but the code will be stored instead of the text.

[jonase]

@ichinaski @muloem

Responses will keep using the pipe-separated format, but the code will be stored instead of the text.

So we will not use JSON encoded responses (as we do with cascades):

[
  {
    "code": "UK", 
    "text": "United Kingdom"
  }, 
  {
    "code": "FI", 
    "text": "Finland"
  }
]

and instead the response will only include the code in the QAStore: UK|FI. Does this mean that the reports should only display the codes as well?

[ichinaski]

Does this mean that the reports should only display the codes as well?

Good question. @janagombitova, @muloem Would there be any problem with just dumping whatever the response contains? That is, if the code is provided, just report the code. Otherwise, the value. Basically, using the same content as the response.

[janagombitova]

@ichinaski @jonase @muloem The way I see it is in the report, if the code is provided then I want to see both the code and the value, and if a code is not provided then only the value. Having only the code in the report can fit the case when the data is to be uploaded into another data tool, but for other use cases this will not suffice and the user needs to see the value as well, so both code and value.

[muloem]

@jonase @ichinaski Agree with @janagombitova comment above.

[jonase]

@muloem @janagombitova @ichinaski Sorry, I'm still a little bit confused:

Responses will keep using the pipe-separated format, but the code will be stored instead of the text.

If the response will only ever contain the (possibly pipe separated) code(s) we will never see a case where the code is missing:

That is, if the code is provided, just report the code

and

if the code is provided then I want to see both the code and the value, and if a code is not provided then only the value

Or am I missing something?

[jonase]

Yesterday we discovered that only keeping the code in the response might turn out to be problematic for the “Allow other?” feature. Changing the response is possible but a considerate amount of work is required:

• Dependant questions are tightly coupled to the current response format
• If we’re changing the response format we also need to make sure we render it correctly on the dashboard and in display names
• We still need to support the old format for backwards compatibility reasons

Before we choose this approach let’s see if the current format is something we really can’t keep on using. As an example, let’s say we have the following options for the question “What’s your favorite food?” with settings “allow multiple” and “allow other”:

code	text
PI	Pizza
PA	Pasta
SU	Sushi
SA	Salmon

With the current response format we have the following scenarios:

(1) No ‘other’ specified and all codes found in the Option datastore: A response looks like PA|SA which we will simply render as `PA:Pasta|SA:Salmon``

(2) ‘other’ specified as “Ice Cream”: The response will be PI|Ice Cream and we can render it as either PI:Pizza|Ice Cream or possibly PI:Pizza|other:Ice Cream.

(3) No ‘other’ specified but we have an unknown code: The response could for example be SU|PC. The PC code is not present in the datastore (perhaps it has been removed or changed) so we can not figure out the corresponding text. Our options for rendering are then:

• SU:Sushi|PC
• SU:Sushi|PC:PC
• SU:Sushi|other:PC

Data cleaning

For data cleaning we must send back a string in the response format. This means we'll need to strip out the text.

• If we use the keyword other: Strip the text for all responses except when the keyword is other. SU:Sushi|other:PC => SU|PC and PI:Pizza|other:Ice Cream => PI|Ice Cream
• If we don’t use the keyword other: Strip the text for all responses. SU:Sushi|PC => SU|PC and PI:Pizza|Ice Cream => PI|Ice Cream

[ichinaski]

@jonase That was pretty much the same approach I had in mind, if we are to maintain the current format as much as possible. The app could use a similar logic when it comes to download data.

I also think the response type should maintain its current role, determining the presence of other responses. This is at least how it works in the app. A mixture of Allow others flag and OTHER response type will indicate that the outstanding item in the response, not matching any defined code, can be assumed to be the user-defined answer.

As for the JSON-formatted response, the implications I see are the following:

• Dependencies management: Although somewhat complex, this is pretty much isolated in the app, therefore not a system-wide problem.
• App needs to handle both types of responses, as old values might be present in the database.
• Format inconsistencies in the datastore. Data will be held in both formats, which can be problematic.
• Dashboard visualization needs to support both formats. Data editing should be supported as well, without leaking this format details to the user.
• Reports need to support both formats.
• Datapoint name needs to be handled.
• Allowing users to define a code for others can potentially introduce several inconsistencies, in both the survey definition and in responses provided by the new app.
• Old apps will be present in the field, which will be providing responses with the old format. This means the transition to the new format may not fully happen any time soon.

[muloem]

Discussion with @jonase @ichinaski

• We will use JSON format for option question responses that contain (optionally) code and (always) text names
• For survey definitions the values serialised in the dependentQuestionAnswer will be the text
• For survey definitions we add an extra code attribute to the option element and keep the value attribute in order to maintain backward compatibility with existing apps. (check that old apps are using it)
• We will not allow users to specify a code when allow other is selected but rather we will use an internal code OTHER
• Reports for the old format which is pipe separated will be rendered as is and if JSON is present it will be parsed and rendered in the new format
• For data cleaning / import the reports will always send back JSON formatted responses
• Data point synchronisation for the app the old value format will be sent back. i.e. pipe separated strings
• Display names need to be processed and show the text correctly
• Change the response type to OPTION
• Consider creating data script for transforming the response format to JSON (should run on app engine)

[muloem]

@janagombitova @jonase @ichinaski raises a good question How to handle the default OTHER code when we have a multi lingual setting. We'll use the default for now and figure out how to handle the multilingual aspects of it in a follow up iteration

[ichinaski]

@muloem @jonase Currently dealing with backwards compatibility, handling responses for existing survey definitions (no codes). Just to be sure we're on the same page here: the code assigned to these options, will be the same as the text, right? Using null can be rather inconsistent, as OTHER values will have codes. This approach goes along the lines of the current implementation, where the text == value.

[muloem]

Is it possible to figure out whether or not there are codes in the regular responses and in those cases do not send a default OTHER code?

[ichinaski]

Although it is possible, I feel that would remove a lot of context, specially now that we're removing the OTHER response type in favor of OPTION

[ichinaski]

Basically, having some sort of OTHER, be it a code or the response type, adds a lot of meaning to the response, and makes value association for existing responses more robust.

[ichinaski]

As per discussion with @muloem, @jonase:

OTHER values will include an additional flag, isOther.

Eg

[{"text": "First option"}, {"text": "Something Something", "isOther": true}]

[jonase]

@ichinaski @muloem

Regarding the export/import roundtrip I'm not sure how to preserve the "isOther" property:

[{"text": "First option"}, {"text": "Something Something", "isOther": true}]

would serialize to First option|Something Something which means that the "isOther" flag is lost. I think we need something that preserves this flag for data cleaning.

[muloem]

@jonase @ichinaski

The main issue here is that the flag would be lost during data cleaning import, right?

If thats the case, I can think of two options here:

• Retrieve the options at the FLOW services end as you were doing previously and depending on the set of options and the allowOther flag on the question, decide which one needs the isOther flag associated with it.
• Serialize the set of responses as they are in the excel sheet, and send to the GAE backend, and from there check the options and the allowOther question flag and determine which one needs isOther just before storing.

[jonase]

@muloem Another potential solution could be to split the response into two cells and add the "other" response in the next cell?

[janagombitova]

@muloem Hi Emmanuel, I was checking the option + codes and noticed that the user has to click on 'add new option' in order to create the values and codes. Can you fix this, so it is as in Kiarii's design, that there are already 2 fields open by default?

And is it possible that the 'code' text box there is the word 'code' as Kiarii also proposed and as we have in cascades? Currently as it is, it will not be clear for the users where to add the code and where to type in the option value/name

I am sorry that the design instructions were not clear enough and these changes need to be added now. Will do my best to be more specific next time. Thank you.

[janagombitova]

delivered.