Make CSV reader have a "strict" schema enforce option

lanl / dsi

LANL Data Science Infrastructure Project

https://lanl.github.io/dsi

5 stars 3 forks source link

Make CSV reader have a "strict" schema enforce option #76

Closed qwofford closed 8 months ago

qwofford commented 10 months ago

Currently the CSV reader plugin will allow arbitrary csv formats to supplement a single table middleware data structure. None values are back-filled to accommodate schemas which are not 100% the same.

The Csv plugin should have a strict mode, that fails if the schemas between two files do not match.

qwofford commented 10 months ago

For an example of how to test multiple csv files using data internal to thsi repository, see:

https://github.com/lanl/dsi/blob/c2419db8117e9229ea1c34c40abb4f10b6b9338d/dsi/plugins/tests/test_file_consumer.py#L54-L59

qwofford commented 10 months ago

@kchilleri

qwofford commented 9 months ago

Strict mode should be given a default value of False. It should be a parameter taken with the __init__ method.
Strict mode should be implemented within the Csv Plugin.
Strict mode should be tested in a unit test.