LibreCat / Catmandu

Catmandu - a data processing toolkit
https://librecat.org
177 stars 31 forks source link

Make the Catmandu::Validator part of the command line tools #274

Open phochste opened 8 years ago

phochste commented 8 years ago

E.g. something like

$ catmandu import JSON to MongoDB with JSONSchema --schema data/myschema.json < data.json

This will send all items, after optional fixes, before being stored in MongoDB, through a JSONSchema validator and reject any records that are not valid.

nics commented 7 years ago

This will probably be implemented as a fix condition

nics commented 7 years ago

dev has a new Fix::Condition::validate

nichtich commented 6 years ago

Validation is still complicated to set up. Workflows based on validation results can be created with fixes but the basic use case "are these records valid or not (and if not why)?" is still difficult to set up. The best I could find is:

catmandu convert JSON to JSON --fix "validate('', JSONSchema, schema: schema.json); retain(errors)" < data.json

But the exit code is not set on validation error. A dedicated validate command should be useful for instance like this:

catmandu validate JSON with JSONSchema --schema schema.json < data.json

This requires a way to throw custom errors from a fix as implemented in https://github.com/LibreCat/Catmandu/pull/334. With this fix, the validate command could be a shortcut for such fix:

 validate('author', JSONSchema, schema: 'my/schema.json')
 if exists(errors) join(errors,"\n"); error($.errors) end

If no errors are found, I expect no output (Null exporter), otherwise error exit code.

nichtich commented 3 years ago

Validating on the command line is also not feasible because the error fix will quit immediately. Validating a single record works this way:

catmandu convert Null to JSON -fix 'validate(.,Mock,reject:1);if exists(errors) join(errors,"\n");error($.errors) end' || echo ERROR

Same with validate command could be:

catmandu validate Null with Mock --reject 1 || echo ERROR