Pipeline Dev: Investigate and plan validation

[RedWalters]

Currently the only validation in the airflow pipeline is csvlint which is better than nothing but not enough. We need to be able to check that:

• the files (source CSV and any supporting metadata) we are pulling into the pipeline are correct for the data being processed (eg if its a 4g-coverage pipeline all the files should be the source data, and 4g-coverage related)
• that the correct processes are being run for the amount of metadata given (the pipeline is likely to need to run different if there is metadata or if its just a csv on its own)
• check that the URIs are correct so that the data will display correctly on PMD without having to upload before seeing if its broken

Timing on implementation of these steps will also be a consideration, will any of the implemented validation steps need to be adjusted once we change to running Airflow on GCP, and if so is it worth waiting and only doing it once or getting it implemented and then adapting it.

If I've missed anything or anyone feels like more/other validation would be good let me know.

[RedWalters]

Another step in this task would be checking what validation the old pipelines run through and seeing what we want to adopt and what we want to leave behind.

One of the main things in this will be SPARQL testing, which - I believe - constitutes the main validation in the jenkins pipelines.