Catalogi import should run asynchronously

[sergei-maertens]

The default request-response time-out is set to about 60s. The catalogi import (a full catalogue) runs longer than that if you have some real data (tested with Utrecht proeftuin catalogus).

A solution would be to run the import in a celery task and have the admin/ui poll the completion status of the import task.

Currently the web-interface for export/import is not usable in this form and I had to resort to using the CLI, which is not a pleasant experience on Docker/Kubernetes.

Tasks (MVP)

• [ ] Add the celery stack to Open Zaak
• [ ] Make the import a celery task
• [ ] Record the import start + end (success or failure) in the database, any feedback (validation errors) should be recorded in said database table to the user can view this information.
• [ ] Add permissions to view import logs to the Catalogi write/admin groups (update fixture)

Nice to haves

• Have a periodic check in the browser (Javascript poll) to check the import task status/progress

[ericherman]

Making this asynchronous is clearly the way to go.

A follow-on could be to consider how to notify of success/failure upon completion, but notification should not be required for the first asynch implementation.

[sergei-maertens]

We could simply track import runs in a database table and show success/failure in there, as users may run into validation errors (spent the better part of the day fixing those yesterday). That would be the MVP approach for me.

[ericherman]

Sounds good.

The very first version may not even need the logging, but recording what happened shouldn't create a lot of scope creep either.

Even if you deliver it in one chunk, I suggest transforming any add-on ideas -- especially thoughts like this which are not coming from someone with access to the purse strings -- into distinct deliverables/todo-items/features/whatever to avoid snowballing requirements from the peanut gallery.

[sergei-maertens]

Agree, however this feedback is really about actually making the interface useful. Making it async so that the import actually completes is worthless if there are errors in the import data and the end-user cannot see what the errors are :-)

I'll add a checklist with MVP actions