Amendment handling

[timgdavies]

This issue is under consideration for the 1.1 milestone.

The problem

Currently the amendment block is structured to contain:

• A date
• A rationale
• An array of changes

The changes array consists of:

• A JSON Path
• The former value found at that path

The reasoning for this structure was to:

• Encourage explicit declaration of the details being changed in the formal amendment to a tender, award or contract;
• Provide a way for systems which do not provide persistent releases to indicate the history of a contracting process;

However, there are a number of issues with the approach taken:

1. The former_value field permits a full range of property types (string,number,integer,array,object,null). This creates challenges for flattening the data or creating alternative serialisations.
2. Applications need to apply considerable logic to parse the former_value information meaningfully, effectively building their own mini-release and comparing it to the current release.
3. Amendment data is nested in each of tender, award and contract, but there is nothing, for example, to stop a tender.amendment including former_value entries for (e.g.) contract.value
4. We have no evidence of former_value being used in the wild.
5. With versioned releases we already have a way of showing the history of a field. The approach in amendment appears to provide a competing approach.

   Proposal

We will develop a new approach to amendment in 1.1 that involves separating out two distinct needs:

1. Having a justification for any amendment;
2. Having structured data on what has changed;

For (1), we should maintain amendment date and rationale and add a field for description. The description would allow for free-text or semi-structured information about the changes made.

For (2), we should document how a single release package can contain two releases, one with the former values that are being over-written or propose use of the data model from the versioned release to provide historical information.

The details of (2) will be further worked up for additional feedback.

Engagement

Please indicate support or opposition for the general direction proposed here using the +1 / -1 buttons or a comment. If opposing the direction, please give clear justifications, and where possible, make an alternative proposals.

[ColinMaudry]

I'm really curious to see how OCDS will model amendments. Currently, in France, we plan to have, for a given tender, an array of "amendments", each containing a date of signature, a comment, and the new values (only a handful of properties of the tender can be amended).

The issue I see with our approach is that to compile the actual current latest tender data, one must compile the amendments array of objects to populate the latest value for each property.

However, this approach enables to cleanly store amendments data.

Another approach would be to always have the latest data in main tender object, and instead of storing amendments in a separate array, storing an array of historical data, with dates and previous values for given properties.

[timgdavies]

Our approach to this would come via the existing releases and records mechanism where I think we pretty much have what you describe in the last sentence @ColinMaudry. We've just not documented it very well or got good worked examples.

In the merging pages pages of the docs there is a description of creating records that represent the latest state of a contract, and then either including:

• An array of past releases of data (so the users need to compare releases to find changes)
• A versionedRelease in which each field becomes an array of the values that field has had over previous releases (essentially pre-computing the changes to make it easier for consuming applications to spot changes)

So, for a tender that gets amended, we should have:

• A tender release with the initial details
• A later tenderUpdate release which contains an explanation of the changes in the remaining amendment block - and simply contains the changed values

Then the amendments details could be captured either by:

• A separate compiled record being available;
• Or, a versionedRelease being included in the release package along with the second release.

I'll see if we can work on more concrete examples of this soon - but hope this makes rough sense. Comments very welcome.

[chrisalexsmith]

Some contracts, particularly for commodities, may have an allowable quantity tolerance e.g. +/- 5% of tonnage supplied, meaning that the value of payments made may exceed or be below the contract value without the contract needing to be amended. This does not constitute an amendment but flagging up in case not covered elsewhere.

[timgdavies]

NSW use case suggests that rather than an object with date and rationale, and then array of changes, we need an array of objects - to allow that a single release could declare a number of different amendments from different points in time.

[timgdavies]

A draft patch and documentation (in need of some worked examples) is now available here: https://github.com/open-contracting/ocds_upgrade_357

@ColinMaudry This introduces the ability to provide a description of each of the amendments in a contracting process, but suggests that the very structured information should be captured through multiple releases (for each before and after of an amendment process). However, in cases where you just have a small number of values that can be amended, it would theoretically be possible to introduce specific fields to the amendment object for these. What fields can be amended in your model?

@andrewlorien Just flagging that we're looking at moving towards array for amendments, but by deprecating 'amendment' and using 'amendments' (plural) - as I think you were preparing some data along these lines.

[ColinMaudry]

Hi @timgdavies, thanks for following up.

In the French model, only the following values can be amended after the first publication of the data:

• the awarded value (if the modifiability of the awarded value allows it)
• the list of the compani(es) that were awarded the contract
• the duration of the contract

Reading the ways to provide data for an amendment, I advise against providing a delta between two values as it adds quite some complexity to rebuild the data.

If you want to work with deltas, work with Git :)

Given the current data practices I would suggest an hybrid model:

• either one provides the new and the old value (dumb and simple method)
• or one provides the new value and a reference (URI) to the old state of the data (smart and advanced method)

[andrewlorien]

Thanks Tim, I'm publishing an array of amendments now. About the delta thing: The Australian Federal government mandates that contract value amendments be reported as a delta. I think that's because a budget change in one financial year should only be reported in that financial year. It's incredibly difficult to report on, and very easy to get wrong, especially in aggregate data. We publish the amendment with its value and the original contract with the original value in tiny text.

Reading the 1.1 upgrade suggestion, I think that's fine. It makes much more sense to publish the new value as an update, and to let users calculate a delta if they need it.

This also allows for the possibility that an amendment could simply consist of a new document which doesn't have a "former value".

[timgdavies]

Thanks @ColinMaudry @andrewlorien - this is really useful feedback.

So - I think in the new model, if law requires a delta, this can be given in the unstructured or semi-structured description field of an amendment in the amendments array - but we would not be asking for this in structured form in OCDS.

For the hybrid model, I think we could accommodate that with guidance around the 'amendsReleaseID' and 'releaseID' fields in each amendment, which should either:

(a) Cross reference the id of a release in the current release package;

or

(b) Provide the URL to a release package and release contained therein (we need to firm up the approach to cross-referencing releases in packages, which I've envisaged as possible with #id identifiers on the URL, but never written up in full afaik)

That way the application can show where the before and after are either in the current file, or across the web.

[siwhitehouse]

Looking at the eForms Technical Specifications for the EU it appears that they also have specific meanings for 'changes' and 'modifications'. My assumption is that 'changes' are made to the tender and 'modifications' are made after the award/contract, but I want to have that checked.

Changes/Modifications are part of the EU's rationales for amendments. These are structured whereas rationale is a free text field in OCDS. So, OCDS can accommodate the EU, but without a codelist for rationale it may prove difficult to report against from OCDS.

One other thing to note is that EU publishers may 'bundle' their amendments. This is from the eForms technical specifications:

In case of multiple contract modifications, if there have first been small modifications to the contract (which did not lead to a publication of a contract modification notice) and then a bigger contract modification (which did lead to a contract modification notice) then the contract modification notice should inform about the nature and extent of all the changes - both the earlier small ones and the later larger one.

For more information, see 2014/24/EU Art. 72.

[ColinMaudry]

@siwhitehouse I emailed you the email address of someone closely involved with eForms.

[JachymHercher]

My comment from the 1.1 peer review:

1) It is not clear to me that an explicit releaseID ("values after the amendment") is necessary in the amendment section. I'm not 100% sure I know how amendment work in OCDS (I'm also not sure it's quite clearly documented), but to me it seems the simplest that every release which is amending a previous release should contain an amendment block. This amendment block then does not need to contain a releaseID, because it already is within the relevant release. 2) There should be a codelist for the rationale of the amendment. This role is already partially played by the releaseTag codelist, but there are some shortcomings.

• The codelist is at the top level of the release, while it refers to information relevant for the amendment section. Users may not be looking for this information there.
• Currently, the codelist essentially contains two types of information: section names and amendment-related info. Having all the elements of the codelist at the same level may be more understandable. This could mean e.g. only section names in the codelist, and the amendment related "subcodes" would be within the amendment section. (By amendment related subcodes I mean: tenderAmendment, tenderUpdate, tenderCancellation, awardUpdate, awardCancellation, contractUpdate, contractAmendment, implementationUpdate, possibly contractTermination).
• I'm not sure having all the section-action combinations in the codelist is the simplest possible. First, it doesn't cover all possible options (e.g. planningUpdate). Second, since the tag is an array, couldn't the same information be gained by just having three tags "update", "amendment" "cancellation" and if a release contains a "contract" and "update" tag, the users know it's a contract update? This would significantly shorten the codelist as well as allow full flexibility. (As mentioned above, the "action" part could be a separate codelist within the amendment section.)
• Besides the current amendment-related values in the releaseTag (~change of notice, change of reality, cancellation), there should be also values for any amendments to a release which are required by the OCDS because of it's structure. Concretely, I'm aware that tender.hasEnquiries and tender/contract.relatedProcess (see below) require this, but there may be more. (This matters e.g. because the interpretation of a buyer with many amendments then isn't "buyer is a bit overwhelmed and keeps making typos in his releases" but "buyer is very diligent and carefully includes all relatedProcesses in his releases").