search

gitdagray / csv-to-sql-insert

Provide table data as a CSV (comma-separated values) file and output a SQL insert statement for a table with the same name as the file.
MIT License
99 stars 120 forks source link

Stream CSV file reading & SQL file writing #23

Open Vikram0118 opened 9 months ago

Vikram0118 commented 9 months ago

The CSV file can get very huge. So, depending on the size of your CSV file, we might want to consider streaming the file instead of reading the entire content into memory.

gitdagray commented 9 months ago

This is a good idea!! 💯

Vikram0118 commented 9 months ago

Processing them sequentially might lead to performance bottlenecks. Parallelizing the processing of multiple rows simultaneously can significantly improve overall performance. I once came across promise.all() for a project doing parallel reading. we can add that.