Acceptance criteria - Kingfisher Process

[hrubyjan]

At the end of each phase of data processing we should evaluate whether it ended well, there is something suspicious or this particular phase failed. For process phase define criteria that will a) prevent a dataset from being published in data registry

• we shouldn't be too defensive and try to detect obvious problems

b) raise a warning but will not prevent dataset from being published

• this should serve to notify admin that there is some issue worth inspecting

We should not insist on having some criteria if we will not see some meaningful rules

[jpmckinney]

On the call, I think we said unexpected exceptions (e.g. from OCDS Kit during upgrade or OCDS Merge during compile) would be considered errors (though I don't know if these caused a "failed" or "warn" state per #32).

[jpmckinney]

@jakubkrafka Other than Sentry (which is disabled), where can I find/query a log of warnings and errors?

[jpmckinney]

Assigning for question above.

[jakubkrafka]

Each docker container (besides of scrapyd logs for particular jobs - scrapy in general lacks this capability) logs into the stdout/console - thus they are captured by the docker itself. All the logs can be inspected via

docker logs <container_name>

[jpmckinney]

Needs to be open for the original issue :)

[jpmckinney]

Kingfisher Process stores messages in collection_note with warning and error levels. We can:

• [ ] Add these to the Kingfisher Process metadata endpoint https://github.com/open-contracting/kingfisher-process/issues/421 cc @yolile, and then store them in the Job context, so that they are not lost when the collection is automatically wiped
• [ ] Consider a weekly or monthly email from the data registry, with the warnings and errors from the last month's jobs, so that we can investigate.

Errors are:

• file_worker
  • "%s has disappeared, skipping"
  • "Source %s yields an unknown or unsupported format, skipping"
  • "Source %s yields invalid JSON, skipping"
• record_compiler
  • "It has no compiledRelease and no releases tagged 'compiled'. It was not compiled."
• record_compiler and release_compiler
  • "OCID %s can't be compiled, skipping"
• loader
  • FileError from Kingfisher Process https://github.com/open-contracting/kingfisher-process/issues/366

These errors aren't individually severe enough to prevent publication, but maybe there's a % threshold.

Warnings are:

• record_compiler
  • "OCID {ocid} has {undated} undated releases. The {len(dated)} dated releases have been compiled."
  • "Its compiledRelease was used."
  • "Its only release tagged 'compiled' was used."
• record_compiler and release_compiler
  • ExtensionWarning from ocdsextensionregistry

These warnings are indeed fine. By getting notified, we can update quality descriptions.

If emails get noisy, we can maybe think about logic for silencing some alerts (e.g. silence unless threshold increases), similar to how Sentry does it. (This would be for a new issue.)

Related: https://github.com/open-contracting/kingfisher-process/issues/222