search

lisad / phaser

library for batch-oriented complex data integration pipelines
MIT License
4 stars 1 forks source link

When loading data, collect all errors #28

Open lisad opened 8 months ago

lisad commented 8 months ago

Right now we use pandas.read_csv and pandas.read_csv implements "on_bad_lines=warn", so we could use that to report more errors before stopping.

If we implement our own CSV reader, should do the same.

lisad commented 5 months ago

With CleverCSV , our current default CSV reader that we changed to, surprises are less likely - but error collecting is now not possible. argh.