Implement point updates functionality

[mtwestra]

This functionality should enable updating existing records on the device.

The current device-side mockup is available in dropbox as FLOW / product development / feature desciptions / point updates / balsamiq / APK-5.bmml

[iperdomo]

@mtwestra not everybody has access to that document. Is there a way to download the proposed functionality? PDF pehaps?

[mtwestra]

The mockup for the APK is here: https://dl.dropboxusercontent.com/u/2839850/APK-5.pdf

[mtwestra]

Overall considerations: Point updates should be a way to add information to an existing point. To ensure data integrity, it seems logical to use different surveys to collect data on items. For example, if we did a data collection in 2011 with survey 'waterpoint survey 2011', we don't want to use the same survey again in 2013 to collect data for the update. We would prefer to have a survey 'waterpoint update 2013' instead. Therefore, we need to be able to somehow group surveys together under a header which signifies which type of item they are about.

For example, we could have the project 'Water points', which has underneath it several surveys such as 'waterpoint survey 2011', 'waterpoint update 2013', and 'new waterpoint 2013'. All these surveys add information to entities named 'water points', which are connected to this project.

To allow different surveys to add information to the 'same' piece of information, we need a way to identify individual questions. The proposal is to give each question a 'question identifier', or 'property name'. This would have a similar use case as metrics, but would be more automatic: you can simply give a question a descriptive 'property name'. For example, the question 'What is the status of the water point?', could have the property name 'status_water_point'. If you then want to have a question in a different survey to have access to this piece of information, either to display it or to update it, that question can be given the same property name.

General design: surveys are grouped into named projects surveys in projects can create items (SurveyedLocales) specific to that project surveys in a project can have access to properties of items of that project 'sameness' of questions in different surveys is established by giving each question a 'property name' Previous information on items is not changed, instead new pieces of information are added.

Functionalities: 1) a user can use a survey to collect information on certain items. 2) a user can download a list of existing items to the device 3) on the dashboard, a user can select which question-answers are shown for identification purposes on the phone 4) on the dashboard, a user can select for which questions previous answers are shown in the survey view 5) a user can select an existing item to add information to, based on:

• proximity - ie. geolocation
• a list of identification question-answers 6) when an existing record is used, previous answers to questions are displayed in the survey view 7) when a question which has a previous answer is not filled in, no new value is saved 8) on the dashboard, a user can export a report with the most recent values of all questions

[mtwestra]

testing in flowaglimmerofhope

Dashboard

On the instance, I have created a survey group 'Akvo point update test', with a survey 'Water point survey'. When you open the survey and go into the questions, you will see an extra area called 'Monitoring settings'. Here, you can do two things: 1) assign a 'property name' to the question. This is used to display the question information on the device, and will later serve to couple questions accross surveys. Only answers of questions that have a property name will be send to the device.

You can either create a new one by typing it in the text field (the system will make sure it is unique), or you can select an existing one. In this case, as we use only a single survey, we create new ones. We don't do some necessary checks yet, as making sure that only a single question in a survey has any given property name.

2) when you give a property name to a question, a checkbox 'display in list on device' is shown. If you select this, this piece of information will be shown in lists of records on the device, to help you identify them. So only a few questions in the survey should have this checkbox checked, for example the ones that will help you identify a point. In this case, I have checked the question 'water point type'.

Device

You can install the apk in the usual way, for example by sending it to your gmail account and installing it from within the gmail app.

When you have the survey on your device, create and select a user and click on the survey. You will see the following items

1) Start new record - this starts an empty record in the usual way. 2) Find existing record - this leads to screen where you can search existing records 3) a list of numbers - this is not implemented yet, but the idea would be that you show some numbers of interest to the enumerator: surveys collected today, pending for upload, this kind of thing. 4) Upload - doensn't do anything yet. 5) List - will show two lists: Pending (ready for upload + saved + error in upload) and Completed (all successfully uploaded records) 6) Download - here you can download records from the server.

More on 6) Download

When you click download, a screen appears with two counters and two buttons. 1) Total on server - when you have an internet connection, the device will attempt to contact the server and get a list of existing records. This will be 0 at first, as you have not collected data yet 2) Cached on phone - this is the number of records you have cached on the phone 3) a button 'Download records' - when there are records available and you click this button, the phone will cache the data on the phone. This is not optimised yet - it will redownload all data each time. 4) a button 'Delete cached records' - delete all cached records from the phone

More on 2) Find existing record

The Find existing record screen shows three things: 1) a search field. This will search for records which have a value which starts with the search term. The search starts when you have entered at least 3 characters. 2) a button 'Show on map'. This will show the current collection of records on the map 3) a list of records. Currently, what is shown is: the id of the record, the date when it was last updated, and the values for which 'Display on list' was checked (see above) When you click one of the items, a detail screen is shown. The list ONLY shows records within 100km, so even if the count is larger then zero, you might still see 0 items in this list. Please record some new records first, including geolocation.

More on Record Detail

The record detail screen shows: 1) the id and the distance from the present location 2) an arrow, which points in the direction of the location in the record 3) a button "Show on map" - shows the current point on a map 4) a button "Update this record" - this opens the survey and populates the known values 5) a list of all the values known, for which the question has a property name.

After the survey with the existing values populated opens, you can finish it as a standard survey. In the backend, it will be recognised that this is an update of an existing record, and handled accordingly. What we don't have yet is the dashboard interface to see the records, this is one of the next points.

Open questions

While building the functionality so far, we encountered a number of usability questions. 1) when you select a record to edit, it is unclear what should happen with the available values. At the moment they are put into the fields. Another option could be to display a 'previous value' above the anwers, so the enumerator can choose to change it or not. A mix could also be possible, although that might become complex 2) What should the syncing policy be? at the moment, syncing cached records needs to be done manually.

General remarks

Things that we will implement next are: 1) quality check of the present code 2) better display of Id and dates 3) make the lists work 4) display records in the dashboard

[caetie]

Some high level feedback questions from a round of field testing in dc today with @henryjewell

• Go out with two phones and collect two baseline points (initial record creation) with 736002/Water point survey
• Return to each point and submit one update per baseline point
  1. What is the specific purpose of the Upload button, and how does its function relate to just Submitting data? Does user need to Submit and Upload? Perhaps for situations where collection is offline? But then users would also press Submit, and the data would be queued up anyway. Seems that there is a lot of overlap here, and it's a bit unclear for the user how they are distinct.
  2. Similarly, within a survey we now have a List, but outside the surveys we have "Survey Status," but both are achieving essentially the same thing, on different scales in the app. List is also too generic of a name. Suggest Status.
  3. The Update Existing Record function creates a situation where user could submit an update without being at the point, but still presents the geo question as blank in the update form, which makes it look like the user should redo the GPS. The GPS should probably not be editable from the app, but then how do we confirm the update is actually occurring at the point?
  4. In the update record survey, autofill all answers. Displaying them blank makes it look like they should all be filled in again.

[mtwestra]

Commits 0f97fcf and 891e04d accidentally given wrong issue tag, they belong here.

[mtwestra]

Issue #346 Sync issue (uploaded points missing on certain device screens)

Firstly, download records screen on device still shows "total on server" as the same figure (12) it was before i submitted four additional points, even though these actually show up on the server dashboard as well as on the phone's "list" and "find a record"

Secondly, one point showing up on server (1067000) as well as on the device "list" (successful upload with green hook), has not shown up on the "find a record" list yet;

[mtwestra]

issue #347 Point IDs don't match between device and server lists

ID cited on device under "Find existing record" ("ty8254cs4") does not match ID cited online for same record ("1034000"), which is rather confusing.

[mtwestra]

Issue #348 "Find a record" vs. "list"

It is not intuitive that under "List" you can't search, nor jump to records, but have to go to "find existing record" instead; it would feel more intuitive to just have one overview list, or at least be able to jump straight from the "list" to a record;

also, under "list" there is a "pending" and completed" tab, but they both show the same results - so why have two?

[mtwestra]

Issue #349 "Find a record" screen issues

On the "find existing record" screen, it is very hard to keep the various entries apart. It would be much easier if every second was shaded differently;

Also, the search function doesn't seem to support searching by date (if you type "sep") you get no results even though on the list there are "Sep" records; searching by date would be very useful.

Finally, if I click on "update this record" on the "find a record" screen, the geolocation and the "name of the owner" text box question are empty i.e. originally submitted values that still show on server are missing; Also, prior to clicking on the "update this record" button, the "distance" field is not filled in on my device;

[mtwestra]

Issue #350 Collected vs. "updated" records

The overview and "list" is a bit confusing, because the updated record registers as a separate "collected" record; i.e. it says "Complete records collected:3", and shows three on the "list", but the third was just an update of the first; the searchable "find existing record" screen correctly shows two records; I understand this, but the way it is phrased is a bit confusing;

the same goes for the dashboard, where updates seem to show up as a separate lines and there is no easy way to see the total number of independent points mapped;

[iperdomo]

@mtwestra to enable the built-in rest security, you need to include your new endpoint filtered by restAuthFilter in web.xml

    <filter-mapping>
        <filter-name>restAuthFilter</filter-name>
        <url-pattern>/rawdatarestapi</url-pattern>
    </filter-mapping>

[iperdomo]

Just to be clear, the file is GAE/war/WEB-INF/web.xml and the new endpoint is /surveyedlocale

<filter-mapping>
    <filter-name>restAuthFilter</filter-name>
    <url-pattern>/surveyedlocale</url-pattern>
</filter-mapping>

[mtwestra]

ok, clear. Thanks! On Nov 13, 2013, at 11:58, Iván Perdomo notifications@github.com wrote:

Just to be clear, the file is GAE/war/WEB-INF/web.xml and the new endpoint is /surveyedlocale

restAuthFilter /surveyedlocale

— Reply to this email directly or view it on GitHub.

[mtwestra]

Test round 28 November:

Bugs

1) When a user is selected, and the phone is turned to landscape, sometimes the user disappears. Not clear when exactly.

2) When a record is selected, and the phone is turned to landscape, the selection disappears

3) When multiple questions are marked 'show in list on device', the values should be appended to the name. Currently, only the first is used in this way. This is probably something that needs to be fixed in both the device and the backend.

Usability issues

1) when a user presses 'sync' and there is no internet connection, the user should be notified that there is no internet connection. At the moment, it just says 'Syncing', and then nothing happens

2) When a user completes a registration form to create a new Record, the record should be deselected by default. At present, the user comes back in the new record, and the tendency then is to immediately use the registration form a gain for a new locale. This then creates multiple surveys in the same record, which is not the users intention. Durng testing, 3 out of 6 users did this wrong. As a compensation, in the 'select record' screen, we could have a tab with 'recently selected', which should contain the records that the user has just created / worked with, so they are easy to retrieve and select again. An alternative is to always show all the surveys as disabled when no record is selected, and let the user click a 'new record' icon (for example a '+'), which then creates the new record and enables the registration form.

3) Maps: The geolocations are often inaccurate, so it is important to have multiple ways to locate a point. In future, we will move to OSM maps, which can be downloaded to the phones. For now, the suggestion is to also enable a 'offline grid', which just shows a grid and some distance indication, together with the points on the map, and the position and direction of the user, in the correct relative locations. If there is no internet connectivity, and hence no map, this would still be very usefull for the enumerator to orient him/herself.

4) Currently the map start on zoomlevel 0. This should start at zoomlevel 18 probably.

5) There was a big demand for letting users select the same survey (or an updated version of that), and then have the values prefilled so the can be checked and updated. The idea would be to have an option that determines this, which is on by default. So the setting would be "automatically prefill values from previous surveys", and the behaviour would be that if a user is in an existing record and clicks the registration survey (or an another one), and there are already values for this survey in the downloaded responses, the values are prefilled.

6) One common workflow is for the enumerators to collect the data and save it for later completion, or for the field manager at the end of the day to go through all the collected data and submit them. For this, it is useful if the saved survey can be found back easily. In this case, the survey should get a name (as it does now), and should be able to be retrieved easily for later completion. So perhaps on the 'search records' screen, we could also have a tab with 'saved' records.

Dashboard

1) we need a way to see the list of records, and underneath that have the list of data.

2) we need a way to export a raw data report. One structure that might make sense is to have the questions as tabs, and to have LocaleId, instance Id or survey name as date, and value as columns.

Other findings

One device, a HTC Experia 4G didn't allow any 3rd Party apps, and the option that is usually available to allow that was not there. So it was inpossible to use FLOW with that phone

[mtwestra]

Functionality needed on the dashboard:

• A user needs to see a list of surveyedLocales
• In the list of surveyed Locales, you can click on one and see the containing surveyInstances
• The surveyInstances should be editable as in the list of surveyInstances
• A user should be able to export data from surveyedLocales
• A user should be able to export either the latest status or all the data

Questions: What is a logical format for exporting this type of data?

[mtwestra]

Two issues:

1. In the Monitoring Tab, the display name and the survey group are not shown in the surveyedLocale table.
2. When I select 'Watermeters Monrovia' and click 'Find', nothing happens. The points shown belong to the PatientsDB database.

[iperdomo]

Test Plan

NOTE: Monitoring is enabled via configuration file See #607

• SURVEYS
  • Survey Group
  • A survey group can be configured as Monitoring Group using the option Make Monitoring Group, available when displaying the list of surveys within the group
  • If a Survey Group is configured as Monitoring Group a M icon is shown next to the survey group name
  • If a Survey Group is configured as Monitoring Group, a label is shown Monitoring Enabled
  • If a Survey Group is configured as Monitoring Group an user should be able to pick a Registration form ("Survey that creates new monitored entities"), from the dropdown
  • NOTE: A Monitoring Group must have a registration form
  • Questions
  • If a Survey belongs to a Monitoring Group a FREETEXT or OPTION questions can be configured as User in record name
  • If a Survey belongs to a Monitoring Group a GEOLOCATION question can be configured as Use in record location
• DEVICES
  • No major change in the functionality. The only remark is that downloading survey definition via the survey id is not allowed if the survey is part of a Monitoring Group
  • Survey assignments is the way to define what data gets synced to the mobile device
• DATA
  • Monitoring tab is now available
  • A user should be able to show the latest data collected filtering by:
  • Survey Group (only monitoring groups should be available in the dropdown)
  • Identifier
  • Display name (Note: the display name search is exact match, including dashes)
  • A user can see all the data collected for an specific data point, using the View details option on each row
• REPORTS
  • EXPORT REPORTS
  • When exporting data of a Monitoring Group a new option is shown Export only last collection, if chosen only the last data collection for each data point is exported, otherwise all data is exported
  • The Raw data report now contains the data point identifier and display name
• MAPS
  • No change
• USERS
  • No change
• MESSAGES
  • No change

[rumca]

As per the test plan and comments on #607 I have tested that enabling monitoring features via configuration works as intended. @mtwestra given that this will be an incremental roll-out - I'm not sure it makes sense to close this issue yet?