When a form can create an entity or update an existing one, the UUID for creation is not automatically generated

[lognaturel]

Software and hardware versions

acbfc439528a45b59392f0278705fbc7adba6630

Problem description

When the entity create action is specified by an XLSForm, the generated XForm includes a setvalue to generate an entity UUID. When the update action is specified, the expression given in the XLSForm entity_id field is used as the entity UUID.

When a form can both update and create an entity, the pyxform behavior is the same as for the update-only case. That means that if the entity_id field is set directly to a form field that represents an existing entity's UUID, submissions that attempt to create entities will be rejected. We do want UUIDs to always be generated client-side to support the offline case so the reject behavior is good.

We need to decide how much responsibility pyxform should take for this case.

Steps to reproduce the problem

Use a form that can both create and update entities such as https://docs.google.com/spreadsheets/d/1poe2lx_MUzbN0ySuZlRX8_fuKqM4LdsBMwKORQW1QRk/edit#gid=1068911091

Make sure the entity_id is just ${existing_participant}, see that the create case is rejected. Set the entity_id to coalesce(${existing_participant}, uuid()) instead and see that both creation and update work as expected.

Expected behavior

The options I see:

1. Leave things as-is and document that when writing forms that can create or update, the form creator is responsible for generating the UUID for the create case. I'm hopeful that these types of forms will be relatively short lived: once Collect has a way to show entity lists outside of forms, it will be easier to split the update and create functionality. Also, creating these kinds of forms will likely require using coalesce expressions elsewhere if there's the possibility of updating existing entity properties (see id_number_to_save) or displaying a property that could be new or existing (see display_first_name).
2. Given an entity_id expression of ${foo}, have pyxform generate coalesce(${foo}, uuid())
3. Figure out some other clever way of generating a static new UUID instead of one that would change on recalculation. I don't know how important this really is but it feels subtly wrong to me that the entity id wouldn't be fixed until form finalization/submission time. There might not be a way to do this without injecting an additional node.

The reason I'm hesitating about having pyxform generate something in that case is that it does feel a bit magical and it could be more confusing than helpful. I think it's likely someone writing this kind of form would come up with something like coalesce(${foo}, uuid()) and then we could end up with coalesce(coalesce(${foo}, uuid()), uuid()). That would work but certainly be a bit strange.

I also do generally feel a bit uncomfortable with the fact that uuid() could be recomputed at any time. As I wrote in the last bullet point, I don't know that there is an alternative that doesn't include injecting a node. I generally prefer not to inject nodes because that has impact on users' data exports. If we go with the first option, a form creator who really cares about entity ids being stable throughout a form-filling session could add a form field with a dynamic default value of uuid() if they want.

[ktuite]

What would option 1 look like in practice? That they must fill entity_id column with coalesce(${existing_participant}, uuid())? This feels okay to me. I like that it is less magical. Hopefully people that are trying to go this route with their form design can handle/would appreciate less magic.