QA 2.0 (to go with Archiver 2.0)

[davidread]

This takes substantial improvements from the data.gov.uk team. It is intended to work with the Archiver 2.0 https://github.com/ckan/ckanext-archiver/pull/15

Key changes are similar to the archiver:

• results of the QA are now stored in a dedicated QA table, rather than in the TaskStatus objects (which were awful to query) and ResourceExtras (which makes it look like your dataset changed every time you archive, and if you're not careful this change triggers another archival and you get an infinite loop)
• the Openness report is done using the neat ckanext-report infrastructure, generated nightly, rather than generated on the fly in ckanext-qa templates. This avoids the big back-end load when you click on a report. Now the report is available to not just admins but the public too (let's be open about broken links - it helps to add pressure to getting them fixed)
• lots more tests
• rather than the data.gov.uk team working on their own fork, they contribute back to the whole. Whilst some teams have been using the root fork of ckanext-archiver, no-one has taken much responsibility for it and it is not well maintained.

QA specific:

• file format detection (rather than just looking at resource.format)

To do:

• [x] Merge changes from okfn master
• [x] Change report HTML from Genshi to Jinja

[KrzysztofMadejski]

As for "results of the QA are now stored in a dedicated QA table, rather than in the TaskStatus objects (which were awful to query) and ResourceExtras":

Having QA score in ResourceExtras allows to easily filter datasets (look for Openess Score on https://danepubliczne.gov.pl/en/dataset). Would it be possible to do in new approach (for example through some hook?). BTW: Seee https://github.com/DanePubliczneGovPl/ckanext-danepubliczne/issues/74

[davidread]

Yes, the providing the Openness Score as a facet in the search can be done from the QA table too - see https://data.gov.uk/dataset.

The QA information is accessed via the qa_package_openness_show action but I'm quite keen that it is pulled into the dataset using the schema. So it wouldn't be a resource_extra in the database (and therefore update the metadata_modified date every time the resource.url's server goes down etc) but it would be included when you do package_show and you can add a facet using IFacets as you've done.

Process-wize, you've come across this annoying problem with ckanext-qa and put in a fix that works for you into your fork, but no-one is tending to the core so it got ignored. Meanwhile at data.gov.uk we did a separate fix for this a couple of years ago because the main development went on on our fork. I'd much prefer that we merged all our efforts together, which is why I'm proposing we merge the data.gov.uk fork in now, seeing as the core has been left to rot.

[KrzysztofMadejski]

Sure, let me know where I could be of assistance. I've also done some work on translation.

[davidread]

Thanks @KrzysztofMadejski . Got you - the translation work here: https://github.com/ckan/ckanext-qa/issues/17 Am very happy to merge this in, assuming we do indeed unite forces and everyone agrees this big PR can be merged.

[davidread]

Thanks for all the responses. No-one's come out against this merge, 4 parties have stated their support at https://github.com/ckan/ckanext-archiver/pull/15, plus 1 here plus the support of @amercader and the CKAN tech team, so I'm going to do the merge this week and I look forward to working with you all on this better maintained main-line of Archiver & QA!

[florianm]

@davidread looking forward to testing once merged! thanks for the hard work!