Refactor Flow - Caddisfly integration

[janagombitova]

Context

Caddisfly is a water and soil quality testing solution that can be added to any Flow form. The user selects the Caddisfly question type and which parameter she wants to test out and saves the question. In the Flow app, once she reaches her water quality test question with a click of the button the Caddisfly app will open up on the defined test type. She follows the instructions on the Caddisfly app, runs the test and sees the result. The Caddisfly app brings her back to Flow app to the question where she also sees the test result. This result is now part of her form, part of her dataset.

Akvo Caddisfly is now fully integrated with Flow from the user's perspective as well from the technical side. Adding new tests to Caddisfly means once the Caddisfly team is ready, Flow team needs to take over the work. The integration is somewhat low impact in terms of new code that needs to be written, but there is a dependency in term of the integration being deployment time dependent. When the Caddisfly team adds a new test, all Flow instances need to be redeployed to make the tests appear in production.

Problem or idea

Caddisfly is a fast growing product, with new test types being added at high pace. The current integration does not allow the Caddisfly team to progress quickly due to Flow lock down. Secondly, this causes a constant source of disturbance to the Flow planning. This integration is disruptive for both products.

Solution or next step

We would like the user visible integration between the two products to remain. Flow users should easily use Caddisfly in their data collection exercises (adding the test, testing, seeing the result in the app and on Flow's space, exporting the results, not cleaning as Caddisfly is a non editable field).

But we would like Flow and Caddisfly to know as low as possible about each other technically, and become more disentangled. We would like Caddisfly and Flow work on a "plug-n-play" mode.

We are thinking of following these principles

• All Caddisfly tests follow a unified and structurally defined format
• Caddisfly team can add a new test (change an existing one or remove on) at any time, not being dependent on Flow's releases
• Flow team is not responsible for Caddisfly tests
• Flow reads in the tests only for the survey editor to know what is the latests Caddisfly offer for users to choose from (for now Flow only reads in the name and UUID, the other information is ignored)
• Flow ignores all extra information in the file (for now) and if the file is broken Flow ignores the entire file
• Caddisfly test results follow a given pattern (given by Flow) that is uniform
• Flow ensures that the results are user readable in the app, Flow space and exports and exposes the data in the API
• Caddisfly team is responsible for ensuring that all tests work well through all the tasks the user does collecting data
• All Caddisfly tests are shown to all Flow users

Benefits

• Caddisfly and Flow can evolve independently
• Flow handles Caddisfly tests in a unified way
• No close coordination needed between the two teams once the changes are implemented
• From users perspective Caddisfly and Flow are connected tools (remains the same as now)

Solution Ideas

to be added by @muloem

Complications

How will we handle data that is already submitted? Caddisfly has been implemented in multiple stages causing data being stored in different forms. How will we deal with backwards compatibility?

[LarsSjogreen]

Notes on solution (taken from memory from the discussion that was had a couple of weeks ago):

The main idea around the integration is that the only point in which Flow needs to be aware of the available tests is during survey design. In this case, this can be seen as a pure front end problem and if the list of available tests can live somewhere separate from Flow (for example as a JSON file in a CDN), it will make it possible for Caddisfly developers to add new tests to the list at any time.

• The list of available tests is a JSON file living in a CDN. The Flow survey designer reads this list from the Javascript at runtime. The file should either have some smart cache busting strategy or the caching of the JSON file could also be disabled altogether.
• The records in the tests definitions JSON file should contain information on name and UUID. Other possible properties for the records are one flag for setting test visibility in the list to enable some tests (deprecated tests or new ones under test) to be hidden from ordinary users. It could also be a good thing to have a metadata or description property that could be used in the survey designer UI, but this functionality can also be added at a later stage if needed.
• The Caddisfly test data coming back from the app is in a well structured format. Flow will store, display and export Caddisfly data in a uniform way without needing to do code changes when new Caddisfly tests appear.

(I've probably missed details but I decided to write this down as a reminder of what we discussed. I still expect @muloem to do research and to describe the solution in more detail.)

[janagombitova]

Issues with the code changes: (more to be added)

• https://github.com/akvo/akvo-flow/issues/1825
• https://github.com/akvo/akvo-flow/issues/2250

[muloem]

The integration points between Flow and Caddisfly are during survey design, data collection and browsing, and report generation.

Looking at all these three areas, we find that Flow essentially needs to know about specific fields in caddisfly test definitions, as well as in caddisfly responses, which does not require that we keep entire copies of caddisfly test definitions embedded in Flow, but can retrieve them from a CDN-like setup.

The Flow-caddisfly integration therefore consists of:

1. Correctly parsing a caddisfly test definitions file to extract a list of tests and identifying information like that name, brand, and UUID of the test.

2. Saving and passing this UUID on to a Flow caddisfly question in order to launch the correct tests during data collection

3. Correctly parsing the responses that come back from caddisfly to render them on the dashboard and in spreadsheet reports; specifically the value and unit of each of the test responses, as well as the image where appropriate.

To enable a CDN-like setup for test definitions, we created an AWS bucket where the test definitions file can be uploaded and updated by the caddisfly team as and when new tests become available.

The tests definition file is a JSON file that is loaded directly by the dashboard UI and parsed and rendered in the survey designer. Incase this file does not conform to the expected format, we do not load any caddisfly tests.

The default tests definition file is loaded by all instances unless an instance has a configuration item that specifies that it should load a different file. We implemented this to enable the caddisfly team to use specific instances only for testing before they move into the live environments.

[janagombitova]

Refactor completed and all done. Closing off the issue.