Are the number of files available on S3 correct / the right order of magnitude?
Are the individual file sizes sane?
Check datapackage.json for inclusion of all sources + files.
This would ideally be forced to run before / integrated with the aggregation script. Any warnings and errors would need to be acknowledged or resolved before kicking off the aggregation process.
Create an independent script which checks:
This would ideally be forced to run before / integrated with the aggregation script. Any warnings and errors would need to be acknowledged or resolved before kicking off the aggregation process.