Notable ODK Collect updates missing from JHU Collect

[joeflack4]

General Notes

• ODK Collect release notes history
• JHU Collect repo
• There are so many bugfixes, it's hard to sift through them all. Therefore I'm not sure if there are any "essential" bugfixes I missed and didn't add here.
• Upgrading to 1.5.0 or above would mean that Android 4.1 (Jelly Bean) would now be the minimum supported version.

What JHU Collect has that ODK Collect doesn't

Independent, novel features

Form linking

If I recall correctly, this was developed by @yanokwa. In the source file there is a save_form and save_instance binding (I believe this may be the correct term). The corresponding fields exist in the source XlsForm.

A standard use case is the following:

1. The parent form is a used to fill in a household roster to identify people for further follow-up.
2. Children forms are generated for those who are identified, and certain data is pre-populated in the children forms.

PMA2020 uses this scheme to interview a household (household questionnaire) and identify eligible females for later questioning (female questionnaire) on topics like reproductive health. The household questionnaire might have the following fields:

• 1 hidden string type (similar to a calculate), with the value of fq-v1 in the save_form field of the XlsForm. The value "fq-v1" represents the form_id in the settings worksheet of the child form. In this case, it would be short for "female questionnaire, version 1".
• 1+ fields (I believe any type is possible), with values of the structure XML_ROOT/PATH/TO/NODE in the save_instance field of the XlsForm. The "XML_ROOT" would be found in the xml_root field of the settings worksheet of the child form. If, for example, my "XML_ROOT" was "FQ" (female questionnaire), and I had a question with the name first_name in a group called confirm_info, the following value would be entered in save_instance: /FQ/confirm_info/first_name.

Given this setup, we would expect the following behavior in JHU Collect:

1. An interviewer would enter a household roster in the household questionnaire. Upon saving it, she would receive a toast notification saying that the form had been saved, and 1+ other forms were created.
2. The interviewer would navigate to the new forms or edit forms section. There, she would see a single item that she could select which, in ODK Collect, would simply be the name of the form, which she would press her finger on to enter the form. In JHU Collect, tapping this option would expand into a list of the parent form (household questionnaire) and any child forms (female questionnaire(s)).
3. She opens a child form.
4. When navigating to the questions for which the path to that question was designated in the save_instance in the parent form, the question would be pre-populated with whatever the value was in the parent form (household questionnaire).
5. If she changed that information in the child form (female questionnaire), the change would be synced with the corresponding value in the parent form.
6. Once completing all of these linked forms (parent form and all child forms), she would be able to submit to the server.

Some features of form linking are the following:

1. The parent form may contain eligibility logic to create the child form. If in the parent form, answers are changed such an existing child form is no longer eligible, the child form is deleted.
2. The child form may contain eligibility logic such that if further information shows ineligibility, the child form will be deleted.
3. The parent form determines the instance name of the generated forms upon creation.
   • Example: PMA2020 uses this so that in the instance chooser list, the instance name shows the age and name of the eligible female.
4. The parent form may send values to the child form. These values are linked, so that an update in one form triggers an update in the other form.
   • Example: PMA2020 uses this so that name and age can be updated in the female form.
5. In the instance chooser lists for "edit saved form," "send finalized form," "view sent form," and "delete saved form," linked instances are shown together.
6. In "send finalized form," linked instances are sent to the server together.
7. In "delete saved form," deleting a parent instance deletes all child forms

We at PMA2020 have used the form linking functionality only in the context of repeat groups. If a single repeat group has an associated child form and the repeat group is deleted, then the child form is deleted.

In the future, it may be useful to allow different child forms with different eligibility and to allow form linking outside of repeat groups.

Hidden fields

JHU Collect has fields like "hidden string", "hidden int", etc. However I am not sure if there is an advantage to using these fields over a calculate, other than the advantage of strong typing. @jkpr may be able to say more.

Variations of features that exist in ODK Collect

User in-app behavior analytics

JHU Collect's implementation may be more granular, but the underlying mechanism (plain text file attachment) is probably inferior to what exists in ODK Collect. I do not use either, so I'm not too sure on the differences. @jkpr would be able to speak more on JHU Collects implementation.

Ethiopian calendar implementation

I think this is very similar to what ODK Collect has, and probably not as good.

What ODK Collect has that JHU Collect doesn't

Essential or very desired

New Features

• Randomize select ones and select multiples
• Log submission analytics (if includes features not in JHU collect)
• Guidance hints (could be used in place / in tandem with the manual)
• View sent forms
• Increased formatting options (HTML and maybe markdown)

Bugfixes

• random() repeating the same random number for each device (link)

Nice to have

New Features

• Rank widget to rank choices
• Form updates can trigger notifications and background downloads (General Settings > Form management)
• Geopoint dialog shows satellites used and time elapsed for troubleshooting
• Scan barcodes without needing external Barcode Scanner app

General Updates

• Translation updates thanks to transifex volunteers (depending on how updated the French is)
• Faster loading of large forms
• GPS update for faster locks and precision
• Ethiopian calendar widget with calendar appearance ethiopian (depending on how different from JHU Collect)
• New setting for resizing images for faster submissions (depending on how different from JHU collect)
• Option to set image size from settings
• Date/time widgets with no-calendar appearance allow nulls and are centered for accuracy (link)

Developer updates

• Support for escaping and _ and # in forms using \ and _ and #

Bugfixes

• SQL injection vulnerability

Other updates of possible interest

New Features

• Send submissions over SMS
• In-form SMS/email sending
• Encrypt an instance during scanning if needed
• Attach audio, video, or other file from Google Drive
• Dark theme (General Settings > User interface) (possibly saves on battery)
• Attach any file on device with question type: file
• Google sheets integration (for form development?)
• Support for groups nested in groups with field-list appearance
• Dynamic search bar should be available for Get Blank Form view for Google Drive
• Include a clickable SVG with image-map appearance for single and multiple selects (Android 5+)
• When a blank form or its media is updated, a message is shown in the form download list
• Device-wide setting to prevent moving backwards (Admin Settings, Form Entry Settings)
• Show separators for numbers with 'thousands-sep' appearance
• Autocomplete appearance on select multiple and select one
• Record altitude and accuracy in Google Maps geotrace points
• post server username as metadata
• Range widget with spinner/picker appearance
• Combine and save filtering and sorting on every form list screen
• Separate form preloaded values for username, email, etc
• Quick attribute support for cascading selects

General Updates

• Searching and sorting of form lists now have modern interfaces
• Required questions are marked with an asterisk
• Read-only widgets are much more legible
• All user interface libraries have been upgraded
• General and Admin settings have been reorganized
• Updated looks for draw/signature/annotate, spinner, URL and list widgets
• Bulk configure devices offline via QR code (In Admin Settings, under Import/Export settings)
• APK size reduced by 65% for faster app updates
• Change the app language to one of 47 languages from settings
• Dialog pickers for date and time
• ODK Aggregate server URL history dropdown
• About screen in hamburger menu
• Scan the instances folder for new instances (allows for bulk loading of filled-out forms)
• Scan the forms folder to remove forms that were deleted from the sd card
• Scan the instances folder to remove instances that were deleted from the sd card
• Go to settings if GeoPoint is triggered and GPS is off
• Use OpenRosa server messages when they are available
• Prompt user to install or upgrade Google Play Services when needed
• Google Analytics
• Reset app data
• Copy/paste using long-press on text field
• Removed instruction screen at start of form

Developer Updates

• JavaRosa update to at least 2.5
• Improved markdown support

[joeflack4]

@jkpr Recently had me look through the ODK Collect release note history for the purposes of evaluating possible future upgrades of JHU Collect. Here is what I found.

[yanokwa]

@joeflack4 It'd be good to understand what critical features JHU Collect has that ODK Collect doesn't have. It'd also be good to understand what would it take to get those critical features added to ODK Collect in a generic way so JHU doesn't have the burden of keeping your fork in sync with the mainline.

[joeflack4]

@yanokwa What are you doing here!

I'll confer with James. We can get a list for you. Any preference on where you'd like that list to ultimately be posted?

[yanokwa]

@joeflack4 I'M EVERYWHERE!

I think putting a list at the top of this issue would be a good place to start.

[joeflack4]

@yanokwa Sounds good. I've updated this info with some of the unique implementations of features that JHU Collect has. Let me know if that is helpful.

[jkpr]

@yanokwa: We would like to update our fork of ODK Collect this year, preferably before November. You have said it would be nice to pull this feature into community ODK Collect. What are the next steps to move this forward so that it can become a part of community ODK Collect? What timelines are we looking at? How can we help? It would be great for us if we could get it done before November.

I am putting below some more details about form linking to add to what @joeflack4 has written.

Form linking

A standard use case is the following:

1. The parent form is a used to fill in a household roster to identify people for further follow-up.
2. Children forms are generated for those who are identified, and certain data is pre-populated in the children forms.

PMA2020 uses this scheme to interview a household and identify eligible females for later questioning on topics like reproductive health.

Some features of form linking are the following:

1. The parent form may contain eligibility logic to create the child form. If in the parent form, answers are changed such an existing child form is no longer eligible, the child form is deleted.
2. The child form may contain eligibility logic such that if further information shows ineligibility, the child form will be deleted.
3. The parent form determines the instance name of the generated forms upon creation.
   • Example: PMA2020 uses this so that in the instance chooser list, the instance name shows the age and name of the eligible female.
4. The parent form may send values to the child form. These values are linked, so that an update in one form triggers an update in the other form.
   • Example: PMA2020 uses this so that name and age can be updated in the female form.
5. In the instance chooser lists for "edit saved form," "send finalized form," "view sent form," and "delete saved form," linked instances are shown together.
6. In "send finalized form," linked instances are sent to the server together.
7. In "delete saved form," deleting a parent instance deletes all child forms

We at PMA2020 have used the form linking functionality only in the context of repeat groups. If a single repeat group has an associated child form and the repeat group is deleted, then the child form is deleted.

It may be useful to allow different child forms with different eligibility and to allow form linking outside of repeat groups.

[yanokwa]

I think the high-level process is that JHU files this as a feature request at https://forum.opendatakit.org/c/features so we can start a discussion with the technical steering committee. Paging @lognaturel for feedback.

It's unlikely this can be pulled into Collect as is, so November would be a challenge. What would make it less of a challenge is if JHU could resource a technical person (ideally an Android dev) to take the lead on bringing this functionality into the core.

[joeflack4]

@jkpr Added your comments to the issue. @yanokwa I will go ahead and post this to the features forum, thanks for the direction. As for working internally on this (would come from the PMA2020 grant), I do not know what our capacity is / will be in the near future for that. We are currently transitioning to a new grant. I believe we will know more in the coming months.

Edit: Posted, https://forum.opendatakit.org/t/form-linking/14565.

[joeflack4]

Just noting a bugfix that we'll benefit from when upgrading; something that's been solved in ODK Collect.

Error in choice filter during begin repeat This guy was having an error: XPath evaluation: type mismatch This field is repeated I noticed I was having this as well. Yaw mentioned that this was an issue in older versions. In this case, the guy was using the KoBo Toolbox app.