Remove all ocdsMerge stratagies

[kindly]

I think there is a case for removing all OCDS merge stratagies and just documenting how the merging is expected to happen. I think it over complicates the schema and is more confusing to implementers.

Merging can be broken down into the following.

• If a value is an object (or top level object) merge how you would expect, by updating the values when the keys match in the new object and adding them if there are new keys.
  • The exception to this are the top level keys "ocid, id, date, tag" discussion for this found #284.
  • For the versionedRelease case for each value that is not an array or a list add the versioned list

     [{"releaseID": <id>, "releaseTag": [<tag>,...], "releaseDate": <date>, "value": <value at release>}, ...] 

• If the value is a list of objects, look to see if there is a corresponding id in the objects of the list you are merging to. If there is then merge as an object above, otherwise add the new objects to the list.
  • The exception to this is the following lists. In this case version the lists as a whole i.e use the whole list as the value in versioned list above
  • Award.suppliers
  • Tender.tenderers
  • Organization.additionalIdentifiers
  • Item.additionalClassifications
  • Amendment.changes

[timgdavies]

Related to #284

[kindly]

I suggest we replace all the merge strategies and just have the following two extra properties as booleans.

wholeListMerge

This will be used for:

• Organization.additionalIdentifiers
• Item.additionalClassifications

omitWhenMerged

This will be used for:

• release.date
• release.id

In version 1.1 as organizations have a top level id, they should use normal merge rules.

[mpostelnicu]

are we still using omitWhenMerged and wholeListMerge described above ? I am having a sense that removing both ocdsVersion and arrayMergeById and treating all arrays in the same way may not work for some corner cases, also ocdsOmit for the release.date.

We have a full merging routine implemented for 1.0 that traverses the release trees and performs the merge based on metadata of each field (the type of merge strategy specified for each of the fields). This assumes zero knowledge of the documentation of how a merge needs to be performed as a whole but only knowledge about how each atomic merge operation is handled, thus quite easy to extend to more fields. As the standard can be extensible to allow customized implementation-specific fields, this is a plus because we dont have to change the merging routine, just make sure the new fields have the type of merging specified in their definition.

If we are planning to abandon this and have no field level metainformation about merging strategies, do we have in mind developing a separate detailed documentation page about how to handle the merge, what fields to omit what to include and what are the exceptions to the general merging rules?

[kindly]

@mpostelnicu The plan is to still do this.

The main rationale for this change is to make creating OCDS extensions easier for people who do not understand the merge strategies as they currently stand. It is a fairly large burden for any extension creator to make sure they understand them and use them correctly. To be honest I am surprised and impressed that anyone else outside the OCP team actually understood them at all!

So for clarity there will be only the two field level merge stratagies: omitWhenMerged (this is equivilent to ocdsOmit now) wholeListMerge (this is equivilent to ocdsVersion on an array of objects)

All other merge strategies will in essence be defaulted:

arrayMergeByID: for all arrays of objects that do not have wholeListMerge overwrite: for all id fields on arrays of objects that do not have wholeListMerge
ocdsVersion: for all other fields

So when people are writing extensions they do not have to worry about merge strategies at all for the most common use cases:

• It is rare to want to use omit in an extension and can not think of any use case for this.
• It is rare and discouraged to have arrays of objects without them having an id and therefore want to be merged as a whole.

This of course should/will be documented.

I can see how this change is slightly harder to implement for people implementing merging (who themselves must be very technical) as it means inferring the merge strategies from the schema and not having every field explicit.

However, I think it is important to make it as easy as possible for domain specialists who are making the extension schemas. It is very difficult to describe what they mean at the moment. Also even for people who know what they are get them wrong, so defaulting the most common cases I think makes sense.

[mpostelnicu]

OK, as long as we are documenting this nicely, it should be fine.

[timgdavies]

I've put together updated merge documentation here: http://standard.open-contracting.org/1.1-dev/en/schema/merging/

[kindly]

After rewriting the script to make the versioned_validation_schema I have found that we need one more rule to say explicitly that we need to version a particular id field. This has no effect on the merge rules as outlined in the new documentation, but it should be mentioned in the versioned release schema section at the bottom.
So even though this is late notice for 1.1, for clarity I think we need to add another rule.

versionId

At the moment this is used in two places:

• tender.id
• item.unit.id

i.e it is used when an object, that is not in an array of objects, has an "id" field.

This new field is not strictly necessary (as it could be implied from schema by looking to see if the object is a single object or is in a list of objects) BUT this implication is very hard to write for anybody implementing versioning and an implicit marker is much easier.

The changes to the schema are here along with the new code that generates the version schema.

https://github.com/open-contracting/standard/pull/412/files#diff-0eaeaf6a8f844dc3725590c958f7c127

[timgdavies]

Thanks @kindly

I've added some additional documentation updates to the branch - and will look to merge this now.