search

SFOE / OGD_qualitychecks

Automated pipeline to check OGD data quality using Frictionless
0 stars 0 forks source link

Testing data #12

Closed aresssera closed 8 months ago

aresssera commented 1 year ago

I ajusted the main.py as well as .github/workflows/actions.yml.

All csv files from the staging folder with ogd in the name are tested. I load the corresponding datapackage.json from the correct ogd folder and create a temporary JSON in which I replace the path to the destination file with the path to the file from the staging folder. For testing, I use the frictionless library for Python.

(includes #4 and #5 )

AFoletti commented 1 year ago

Code looks ok, but that should be reviewed by @cype too. More importantly... how can I test it?

aresssera commented 1 year ago

You can manually start the action by choosing testing_data for both, the actions (actions.yaml) and the main.py.

image

The results are printed in the console, after the workflow is finished you can look it up.

AFoletti commented 1 year ago

So, I see a report telling me "valid" or "not valid". This is however a little scarce information if we want to make the Fachsektionen independent. Is there a way for the report to output a little more detail about what is actually the problem in the data?