Exceptions to immutability

[jpmckinney]

From https://github.com/openownership/data-standard/discussions/475#discussioncomment-7106304

tiredpixel: how can something like a clerical error be distinguished from an actual change?

@jpmckinney: This can be handled in different ways, depending on user needs.

As you suggest, the simplest is to just change the original data. This breaks immutability – but, honestly, unless there are very strong use cases for preserving immutability, then it is so much simpler (for both users and publishers) to allow clerical errors as an exception to immutability.

Another option: Today's procurement systems follow the same structure as their original paper processes. So, a notice is published, containing a clerical error. Being a physical paper posted on notice boards and distributed to offices, it's not possible to simply change it. As such, the EU, for example, publishes a special type of notice ("Corrigendum", or "Change" more recently) with such changes. Publishers are not allowed to make non-clerical changes via such notices. In BODS, perhaps a new field on a statement would serve as a flag, like "correction": true.

@kd-ods: Yes - we will need to treat certain types of correction and post-hoc redaction as special cases. So we will be outlining in future versions of the standard the circumstances under which statements need not be immutable.

Emphasis added. Opening the issue as otherwise this intention for future BODS versions is untracked.

[StephenAbbott]

Thanks @jpmckinney. The BODS team is gearing up to the release of version 0.4 in June and will be spending the next couple of months documenting all the outstanding feedback we received where new issues need to be raised to capture future development possibilities. We have internal logs of some feedback which will be raised as issues where relevant on the BODS Github repo

[jpmckinney]

Thanks, @StephenAbbott. Maybe a process change to consider is to record suggestions and discussion in this issue tracker, rather than in an internal tracker. Open standards typically record everything in their public tracker – especially if the suggestion or discussion already previously occurred in that same public tracker. While going through past issue, you also noted renaming had been recorded internally, but this should just be noted here, in this tracker.

[kd-ods]

In BODS 0.4 we refined the messaging around immutability. In particular:

On the Generating statements page:

Statements SHOULD be treated as immutable: once a Statement is published it SHOULD NOT be republished with the same statement identifier (statementId) and different property values. See Information updates for more information.

And then on the Information updates page:

Error correction

Errors in published data may be due to mistakes at the point of information disclosure, or the incorrect processing of information by the data management system.

In either case, errors SHOULD be corrected by the issuing of new statements including an annotation, with the motivation ‘correcting’ and a description of the correction, and an updated publicationDetails.publicationDate.

See the example in Dates guidance.

And

Redaction

If sensitive information is accidentally published in a Statement, the Statement MAY be republished with the same statement identifier and updated property values.

We've kept the requirements a little loose with SHOULDs and MAYs. The rationale is that the precise way that errors and redactions will be handled will be a function of: the nature of the error or redaction, the way the data is published, and any relevant privacy legislation. There is a limit, by this rationale, to how much error handling and redaction practices should be standardised as part of BODS.

Any feedback on this guidance as it appears in BODS 0.4 will be welcomed.