Additional field submission when data is read in from an external instance

[pbowen-oc]

We are seeing an issue when we read data in cross-form data (i.e., from an external XML instance).

We have a simple form that reads in a value and then there is a second item that is calculated to be the value that was read in.

When I open the form the first time, suppose it reads in "TEST". Now when I close the form and open it again, it first submits null for that item and then a fraction of a second later it submits "TEST" for the item (as the value that was read in). This happens nearly every time we open the form in edit mode. The data is correct in the end (assuming the queue submits in order), but it is cluttering our audit logs with the value changing form "TEST" to null to "TEST" to null to "TEST, etc. When I look at the network calls going out, the first submission of null is less than .5 seconds before the submission of the actual value. Can you look into this and see what can be done to not update the model and trigger submissions until the external instance has been read (at least for any items that are dependent on the external instance)?

[MartijnR]

Ah, ok, so this is likely dependent on the order of the calculations.

and see what can be done to not update the model and trigger submissions until the external instance has been read

I don't think that's feasible, but will check.

It sounds like this is caused by the fields for which the values are not submitted (oc:external="clinicaldata" as bind attribute). So it's all correct behavior as far as Enketo is concerned.

[pbowen-oc]

Yes, I only observed this behavior with calculations dependent on oc:external="clinicaldata" items so far.

I'm open to other solutions if you can think of other options to prevent the behavior I described.

[MartijnR]

I will think about an approach. One approach would be to submit those "clinicaldata" fields, but I'm sure you have good reasons for not wanting to do that.

[MartijnR]

Question B has a calculation that reads a value from external data. It has attribute oc:external="clinicaldata" so its value is never submitted. Question A has a calculated value that depends on B, e.g. calculate="/data/B"

First record: B is set to value '123' from external clinicaldata, and A changes to that value too. Only A is submitted

Editing sessions: The record has value "" for B. It has value "123" for A. Because the calculation for A comes before the calculation for B, A is set to "" and triggers a fieldsubmission. Once the value for B is calculated (from external clinicaldata), it is set to e.g. '234'. Question A will subquently also be set to '234' and triggers another fieldsubmission.

Solutions:

I: A solution that tries to change the order of the calculations is not a good approach. This is not easily predictable in Enketo because calculations without form controls are dealt with separately from calculations with form controls. There is an issue to make this predictable: https://github.com/enketo/enketo-xslt/issues/31, but that is really quite hard.

II: A solution that does not set the value to "" in the model during the editing session seems impractical. There isn't a clear way of determining when to stop blocking model changes.

III: Submitting clinicaldata seems the only practical solution, so far.

IV: Changing the calculation module to first do all calculations with the clinicaldata attribute is worth exploring. TBC

V: Or simpler to implement: add a separate (duplicate) calculation of only fields with the clinicaldata attribute, during initialization before regular calculation initialization is done.

[MartijnR]

I am stuck with this. IV and V are both very difficult to implement (in a way that is maintainable).

[pbowen-oc]

We may need to resolve this on our end by storing the external data values and passing them back in as part of the instance so that the slight time lag is not an issue.