Implement Streaming Input For Large Files

CMSgov / price-transparency-guide-validator

Validation tool to check output files required by the price-transparency-guide

Apache License 2.0

29 stars 13 forks source link

Implement Streaming Input For Large Files #3

Closed shaselton-usds closed 2 years ago

shaselton-usds commented 2 years ago

Allow for the loading of files (local file system) in a streaming fashion that can be validated against the current schemas

awatson1978 commented 2 years ago

Syntax below:

# stream an ndjson file and validate along the way
price-validator --stream ../output/allowed-amounts.ndjson --schema ../schemas/allowed-amounts.json 

# stream an ndjson file, validate, and output the results into a separate file
price-validator --stream ../output/allowed-amounts.ndjson --schema ../schemas/allowed-amounts.json --save ../output/errors.txt