Improve cert_id heuristics

[J08nY]

Currently, we have a bunch of cert_id duplicates and a bunch of certificates without a cert_id matched.

Duplicates

Current list: https://gist.github.com/J08nY/d03714234198d16a41aa931e956ee647 MongoDB command: db.cc.aggregate([{$group: {_id: "$heuristics.cert_id", count: {$sum: 1}}}, {$sort: {count: -1}}])

• Do not allow a cert_id assignment of a wrong scheme (e.g. ANSSI cert_id to a cert that has scheme != FR).
• Parse year from the cert_id and do not allow if not the same as year of valid_from. This needs some verification that this will not remove valid cert_ids.
• Improve frontpage matching.
• Have a post-processing step that removes cert id assignments from a (deterministic) part of the duplicates (just to have a clean but equally bad dataset).

Without matches

Current list: https://gist.github.com/J08nY/9162e89a60f7381a381d1c1c4e1341c4 MongoDB command: db.cc.find({"heuristics.cert_id": null}, {_id: 1})

• Add more regexes for the broken cert_ids (like a space inside) and code to reformat those into valid ones.
• Improve frontpage matching.
• Extract cert_ids also from document filenames and PDF fields (title/subject).

[adamjanovsky]

Can we search for the cert ID in the filename of the PDF documents? I think we change their filename, but we should keep the track of the original one. This was done previously by petrs. I guess he was doing it for a reason, maybe it could identify some missing cert IDs. Besides, in methodology we claim that we do this :D

[J08nY]

PR #258 did most of the work here (the part that made sense), check it out for more details.