External validation with HTTP/database services

[avernet]

Right now this is possible, but only indirectly:

1. Define the service to call in Form Builder.
2. Store the result of the service in a hidden field, i.e. a field which visibility has been set to false().
3. Use the value of that field in the constraint expression of the field you really want to validate.
4. Define an action that calls the service when there a value change on the field to validate.

This approach has some limitations:

1. It is convoluted – If a number of fields need to be validated this way, as many services, actions, and hidden fields need to be defined and maintained.
2. It isn't well-suited to validating multiple fields – Often, customers would like to use the same service to validate several fields.
3. It doesn't allow for the validation to run on save or submit – As calling the service can be an expensive operation, it might make sense to just do it on save or submit, not on value change.

[ebruchez]

This comment contains ideas moved from the wiki.

Rationale

Currently, Form Runner only supports "internal" validation, that is validation expressed with types and constraints (like in plain XForms).

Often, you need external validation, which might involve complex calculations or lookups that are hard or inconvenient to implement with XForms. E.g.:

• credit card number/expiration is validated against an external service
• a value must belong to a large, changing set of data stored into a database etc.

A possible way

A possible way of implementing this is documented in this how-to: Perform external validation.

This works in limited cases but has limitations, including:

• attributes are not supported
• annotations pollute the document to validate

These limitations are likely not acceptable for Form Runner as we want this feature to be future-proof.

New thoughts

Requirements

• supports attributes
• does not pollute the XML data
• future-proof in Form Runner

Lifecycle

With internal validation, data is frequently revalidated, typically whenever a field changes.

With external validation, this by default might be too costly. In that case, external validation:

• takes place upon explicit user action, such as "Save", "Submit" or "Refresh"
• upon the user changing the value of an externally-invalid field, the field is marked as valid again, until the next external validation takes place Internal validation can still be applied incrementally.

Service

Validation information is provided by a service:

• in: instance data as XML
• out: invalidity information if any
  • invalid nodes
  • custom error messages

The exact format of the response is TBD, but could be:

• annotated XML instance data
  • ⇒ drawback: attributes not supported in an elegant way
• list of invalid paths using canonical XPath, e.g. /foo[1]/bar[3]/@gaga
  • ⇒ drawback: need to produce and evaluate such expressions

Persistent/imperative MIPs

In XForms, Model Item Properties (MIPs) are declarative, so don't need to be "persisted" in an instance DOM as they can be recomputed as needed.

With external validation, we might need to introduce a type of persistent, imperative MIPs, settable with an action:

<xxf:setmip ref="foo/bar" name="valid" value="false()"/>

and readable as an XPath function:

xxf:getmip($name as xs:string) as item()

An action to clear all MIPs, or all MIPs of a given type, might be needed:

 <xxf:clearmip ref="instance()" name="valid"/>

NOTE: Persistent MIPs do not replace declarative MIPs: they combine with them.

When nodes are removed from an instance, these MIPs are automatically removed.

Impact on the XForms engine

This has the following impact on the implementation:

• implement action
• implement function
• add support, on data nodes, for holding such imperative MIPs
• add support for this to XPath analysis (parallel with setvalue)
• serialize/deserialize this in dynamic state

Impact on Form Runner

• property enables this feature with a service URL
• Form Runner modified to call service when needed, parse result, and set custom MIPs

Impact on Form Builder

None for now, this has to be configured through properties.

[ebruchez]

At this point, I am not 100% sure that the "persistent MIP idea" is the best one. But the idea that external validation could be triggered by default only upon save/submit is good, and maybe configurable via independent properties or directly in the save/send processes properties.

There should probably also be ways to trigger external validation separately, maybe by adding an action once we have separated Form Builder actions from services.

As a first step, it might be ok to say the following:

• "external validation" is a built-in feature
• there is a new process action to trigger it
• save/send processes can call this action
• the action
  • sends the entire instance document
  • expects a format TBD in return
  • uses the result of the action to mark fields as valid/invalid (exact way TBD, with persistent MIPs or not)

[avernet]

+1 from customer

[avernet]

+1 from customer

[ebruchez]

+1 from evaluator

[ebruchez]

Comment on the proposal above: we could send the whole XForms instance, or possibly pass a list of control names to send. Then only those values would be sent, as URL parameters or in a application/x-www-form-urlencoded POST.

Clearly an important step is to determine the format(s) of the request/response.

[ebruchez]

+1 from evaluator

[ebruchez]

• Re-reading the above, I am not sure we need to support validating attributes. Some of those are used by Form Runner, such as filename, size, and mediatype for attachment, but it is not a very general requirement besides that.
• Another requirement the ability for external validation to provide error messages. Ideally, those could be in multiple languages.
• We might want to support global error messages, although this can be implemented separately.
• Should Form Builder allow specifying custom error messages? Scenario: external validation only marks the element as invalid, without a custom error message.

[ebruchez]

With datasets, the workaround can look like:

• call service and store annotated XML data in a dataset
• each control must have a constraint using data in that dataset

This is still heavy but doesn't require hidden controls. A single service call can return all validity information.

[ebruchez]

Two scenarios to consider:

• single service call to validate one or more or all controls
• multiple service calls to validate individual controls

Both are probably reasonsable depending on the situation.

[ebruchez]

With datasets:

• add "Validate" button to form (see also #3236)
• send the form's XML data and set a dataset via a service activated by the button
  • NOTE: This is not possible yet! What about from a process?
• create a custom bind which validates all controls based on information in the dataset

Example:

<xf:bind
    ref="instance('fr-form-instance')//*[empty(*)]"
    constraint="
        for $name in name(.), $ds in fr:dataset('validation-dataset') return
            empty($ds/*[name() = $name][1]/[@valid = 'false'])
    "
/>

Above assuming a validation document like:

<validation>
    <name valid="false"/>
    <foo valid="false"/>
    <bar valid="false"/>
</validation

You can vary the format of the validation document.

[ebruchez]

The above might work if we add the ability to set a dataset with the result of a process send() action.

[avernet]

+1 from user

[ebruchez]

+1 from customer