search

Arp-G / csv2sql

A blazing fast fully-automated CSV to database importer
MIT License
54 stars 4 forks source link

Better error handling #26

Open Arpan-Kreeti opened 2 years ago

Arpan-Kreeti commented 2 years ago

Ideas on error handling

If an error is encountered during the insertion of data we should not fail the entire process.

Ideally, we should localize the error only for the file, or even better we can continue with that file after logging the error. If a file logs more than X errors we can choose to stop processing the file.

Errors can be written to some error log file which we can save in the user's CSV directory. The CSV directory can have a results folder with the schema SQL file and any error logs. On the processing end, we can inform the users that errors were encountered in the following files and point then to the log file for details.

Changes required here: https://github.com/Arp-G/csv2sql/blob/74c8facb553fcb6b080fa093bb7e4ea24c5b95a5/apps/csv2sql/lib/csv2sql/database/database.ex#L97

In case of errors during schema inference that particular file can be skipped.

Arpan-Kreeti commented 2 years ago

Running iconv is goo idea to get rid of incorrect bytes of data