make all dataset upload errors visible to admins

[jbothma]

• it should be convenient to upload a dataset file with the right metadata
• you should not easily forget to do that
• uploaded dataset files should be imported, creating PurchaseRecord objects
• if uploading fails, it should be clear
  • that it failed - that the dataset is not up to date
  • why it failed so we can resolve the issue

As a data admin, I would like to know whether uploads were successful so that I can tell whether it is complete or whether I need to change something and try again.

Options for satisfying that

• error feedback on creating a new dataset or updating a dataset file
  • requires access to the request variable
  • only works until an import takes longer than is acceptable for a browser form submit
• queue with error feedback and clear overview of datasets, upload success, and current versions
• do the import sync in a signal for now and just save the error message on the DatasetVersion object as we might if it was done async.

e.g.

Admin error feedback: tablib.exceptions.InvalidDimensions

e.g.

django.core.exceptions.ValidationError: ['Missing field(s): buyer_name, supplier_name, order_amount_zar']

[jbothma]

django-import-export errors as well

Basically we have a model Dataset with a field current_version foreign key to DatasetVersion and DatasetVersion has a foreign key to Dataset and a file field.

The idea is that a dataset can have multiple versions - each file is associated with a DatasetVersion.

When a dataset is created, the admin should also upload a file as the initial DatasetVersion. This is currently done using an inline on the dataset admin, but there's also a datasetversion admin (which is less convenient because you have to select from a growing list of datasets with imprecise names, and it's a separate page from where the dataset is created so it's easy to forget)

Every time a new DatasetVersion is uploaded (e.g. to fix a mistake or improve the dataset somehow), the new file needs to be imported (we use django-import-export via its API rather than its admin integration). This is currently done using a signal, which is not ideal because it doesn't have access to the request to provide error feedback. https://github.com/keep-the-receipts/procurement-portal-backend/pull/44 had to be reverted because it wasn't working from the dataset admin and was silently allowing dataset versions to be created despite importing failing.

Creating a new version should set it as the current version.

Editing a random version should not set it as the latest version, since you might simply be fixing metadata on an old version.

[jbothma]

https://stackoverflow.com/a/31431103

[jbothma]

fixed by https://github.com/keep-the-receipts/procurement-portal-backend/pull/50