Data Review Tool rewrite

[jpmckinney]

• [ ] Create new repositories for non-CoVE version, e.g. plover and plover_web (still Django)

Web frontend (templates and text)

• [ ] (Quick fix) Adjust the header and footer to match the Data Registry or Spoonbill
• [ ] Update error messages
• [ ] Implement new designs and requirements (folder)

Web backend

• [ ] Use Celery (with Redis queue) for background tasks
• [ ] Implement async upload and async processing
• [ ] Don't use the filesystem as the cache
  • Once we cache validation results, we no longer need to write/cache the metatab, conversion warnings, extended schema, cell_source_map, heading_source_map.
  • See Unflatten in Kingfisher Collect to get unflatten results from temporary directory.
  • We do want to write the original file and the converted file to the media directory, as it is helpful to users and analysts to download the file (especially if they did not upload it, lost track of it, or if the data at the URL has changed).
• [ ] Use Django page cache (with Redis, since it's already installed)

Library (libcoveocds)

• [x] Extract relevant logic from lib-cove (only common.py is very relevant)
• [ ] Simplify and refactor the extracted code
• [x] Stop writing to validation_errors-3.json and remove corresponding logic from web backend
• [x] Remove keys from output/context that are unused (check what other projects reads this library's output)
• Remove JSON serializing of errors (originates in lib-cove) [This is needed to aggregate similar error]
• [ ] Try switching to jsonschema-rs https://github.com/open-contracting/lib-cove-ocds/issues/123
• [ ] Ask ODS about dropping AGPL in our code

Learning

While everything is fresh, read latest JSON Schema to see if anything can be simplified by adopting new versions

• https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-validation-01
• https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-01

[jpmckinney]

Ideas for new checks from internal discussions, that better solve for what data support managers find useful in the current Key Field Information:

• “who bought what from whom, for how much, when and how”

Unlike other checks, it is useful to report on the details of these checks even when they pass.

It is also useful to report (similar to Pelican):

• number of contracting processes
• stages covered
• date ranges

Plus, if possible (Slack):

• tag counts
• party role counts

An important design caveat is that users are not uploading full datasets or representative samples. The checks need to make sense even for a sample. We'll need to word any messages carefully. e.g. "the sample doesn't contain awards" not "your dataset doesn't contain awards" or something.

In general, along the lines of the earlier user research, we need the DRT to be useful, interpretable and actionable for OCDS implementers. If something is needed for data support, we might prefer to implement it as a notebook. (Of course, it is more convenient for team members to not load another tab.)