Have an overview of system, across Collect, Process

[odscjames]

(From discussion from https://github.com/open-contracting/kingfisher-process/pull/152#issuecomment-491908018 )

[duncandewhurst]

In particular, we need to be able to see whether checks are still running or complete.

[jpmckinney]

Which collection ID are you interested in? For now I can just share a SQL command for checking this.

[duncandewhurst]

It was 1368 (Moldova). I think they've completed now, but it would be good to see the SQL command to double-check (and to add to the Kingfisher troubleshooting doc).

[jpmckinney]

Get the number of checked releases:

SELECT COUNT(*) FROM release_check WHERE release_id IN (SELECT id FROM release WHERE collection_id = 1368);

Get the number of releases:

SELECT COUNT(*) FROM release WHERE collection_id = 1368;

Both equal 16498.

[duncandewhurst]

Ah yes, I'd looked at release_check but originally it was empty and I wasn't sure if that was because checks were still queued/running or because they had failed completely (as had happened with an earlier Moldova collection). I guess being able to check that depends on https://github.com/open-contracting/kingfisher-process/issues/301.

In the end I ran check-collection to be sure and once it had finished running release_check was populated.

[jpmckinney]

Adding some miscellaneous notes. I have a paper mockup from December that's more complete.

• Show the process status of the workers (i.e. to know whether they are running)
• Make it easy to identify empty or small collections, so that they can be easily deleted
• Scrapyd integration
  • Web form to start a crawl (and select configuration options, like whether to skip checks)
  • Option to display all source IDs in the UI, even if no collection exists for it

[jpmckinney]

In case I forget, I put a cleaned-up paper mockup and notes document here: https://drive.google.com/drive/folders/15TeYUmfg2LN_hIM0Szw6uaUsz7mGV74Y

https://drive.google.com/drive/folders/1R5vnh1vMySuTPaFL8NIQSNcipu97TysQ

[jpmckinney]

I'm removing Views from this issue, because it's run manually, and it generally takes less than 2 mins.

For the few cases where it takes longer, we should make it fast rather than build infrastructure to monitor its progress: https://github.com/open-contracting/kingfisher-views/issues/155

[jpmckinney]

When we rebuild the analyst's server, we can add the registry, whose Django backend makes it possible to see which jobs are running, and which steps have completed.

I think the main thing analysts want is #273 - it doesn't really help to know "oh, it's at 63% of the check step", etc. By reusing the registry's code, we have one less thing to maintain.

Okay to close this issue, @yolile ?

[yolile]

Yeap, that, plus the now documented queries for checking if the steps are complete, sounds good.