Open roll opened 4 years ago
Hey, is our best bet on this just to wait until frictionless-py is fully implemented and dataflows has migrated?
Yea. I think so. Now this atomic checks architecture works with Frictionless so we can apply checks on streams (as a dataflows processor without consuming a stream separately)
Overview
From
goodtabes@3
(now early alpha) it will have a function likesystem.create_check('baseline/integrity/etc')
returning a check object withcheck.validate_headers/row/table
available. The amount of checks have been drastically reduces (only 2 core checks and 6 advanced).So we will be able to integrate it into the
validate
processor like (maybe adding core checks baseline/integrity by default):As an output, I think we just can add the
errors
list to a resource descriptor with a list of found errors.Initially
dataflows-goodtables
integration is a @cschloer's idea but, for now, I don't see how we can friend them due to their streaming-nature without going to lower-level (the level of the individual check).