Write daily data to JSON

AntoineAugusti commented 4 years ago

Write the latest consolidation data to a JSON file and publish this over HTTP thanks to GitHub Pages.

Goal

Enables usage from other systems through an API (well, a JSON file). Candidates: schema.data.gouv.fr, data.gouv.fr

Results

Badges preview. As you probably guessed, the cleaner your data is, the greener your badge is.

Sample report.json

Here is an example file, for a single dataset. ```json { "5448d3e0c751df01f85d0572": { "date": "2020-04-29", "dataset_id": "5448d3e0c751df01f85d0572", "name": "Fichier consolidé des Bornes de Recharge pour Véhicules Électriques", "schema_slug": "etalab/schema-irve", "schema_version": "1.0.3", "file_url": "https://static.data.gouv.fr/resources/fichier-consolide-des-bornes-de-recharge-pour-vehicules-electriques/20200420-224406/bornes-irve-20200420.csv", "report_url": "https://go.validata.fr/table-schema?input=url&schema_url=https://schema.data.gouv.fr/schemas/etalab/schema-irve/latest/schema.json&url=https://static.data.gouv.fr/resources/fichier-consolide-des-bornes-de-recharge-pour-vehicules-electriques/20200420-224406/bornes-irve-20200420.csv&repair=true", "nb_rows": 12131, "nb_errors": 4359, "nb_rows_with_errors": 4012, "errors_report": "{\"structure-errors\": {\"count\": 1, \"count-by-code\": {\"extra-header\": 1}}, \"value-errors\": {\"count\": 4358, \"rows-count\": 4012, \"count-by-code\": {\"required-constraint\": 4239, \"type-or-format-error\": 117, \"pattern-constraint\": 2}, \"count-by-col-and-code\": {\"n_enseigne\": {\"required-constraint\": 397}, \"id_station\": {\"required-constraint\": 446}, \"puiss_max\": {\"type-or-format-error\": 114}, \"Ylatitude\": {\"type-or-format-error\": 1, \"required-constraint\": 2}, \"nbre_pdc\": {\"required-constraint\": 1910}, \"Xlongitude\": {\"required-constraint\": 2, \"type-or-format-error\": 2}, \"code_insee\": {\"required-constraint\": 917, \"pattern-constraint\": 2}, \"acces_recharge\": {\"required-constraint\": 3}, \"accessibilit\\u00e9\": {\"required-constraint\": 560}, \"n_station\": {\"required-constraint\": 2}}}, \"count\": 4359}", "status": "invalid", "error_percentage": 35, "badge_url": "https://img.shields.io/static/v1?label=Consolidation&message=4359+erreurs&color=red" } } ```

TODO before merging

[x] Rebase master to get latest data
[x] Backfill schema_slug to data.csv

AntoineAugusti commented 4 years ago

@abulte It seems like a nice idea. Two issues I'm not sure how to resolve:

if you git pull, changes from gh-pages will still be pulled
on master you're missing a CSV file to append your daily data. You new to get access to this file by switching branches or adding an extra step in CI + locally

geoffreyaldebert commented 4 years ago

Ok for me. Apart from that, I will think about this pulling issue...

etalab / monitor-consolidation