Open jpmckinney opened 8 months ago
From 2024-09-28 email from Michael, about adding "queue" functionality without migrating to lib-cove*-2 packages.
The commit adding the "queue" functionality is https://github.com/ThreeSixtyGiving/dataquality/commit/92a3a16fa059c240731bef09383e89b48320e517 [which includes refactoring/tidying].
It uses the fact that lib-cove-web has a small database and that calling the explore page with a parameter starts the processing by looking up the parameters in the database.
So in 360 cove what happens is:
[360 Cove] Form POST [lib-cove-web] As normal the input view creates a database entry for the input data (and downloads the data if needed) that view then redirects to 'explore' named URL (data.get_absolute_url() ) [360 Cove] Re-implement 'explore' named URL's view as a new separate view (cove_360.views.data_loading, name='explore' template: data_loading.html) [360 Cove] In the 'data_loading.html' template there is some JS that does a GET to 'results' which is the old explore page (thus triggering the real processing as explore page used to) [360 Cove] 'data_loading.html' polls a new JSON endpoint which polls the database to get the status of the processing and redirects to the 'results' page (the old explore page) when it's complete (showing a spinner in the meantime) I also added a cache in the mix so that re-visiting the results page doesn't need to re-run the processing which would be annoying for people if they're sharing the results link etc.
This has been in place for 3 years now(!) so it was one of those things that accidentally stuck, however as of last week we've started on a program of improving 360 Cove, one of the tasks I am going to look at is to make this flow less convoluted, my first port of call will be looking at new async api in django to tidy this up. It should be pretty easy as the current cove has all the pieces needed for this and we aren't anywhere near a scale where we need more complicated queue management.
From 2023-05-19 email from Duncan.