Support CSV parallel encoding

multiprocessio / datastation

App to easily query, script, and visualize data from every database, file, and API.

https://datastation.multiprocess.io

Other

2.9k stars 112 forks source link

Closed Posrabi closed 2 years ago

Posrabi commented 2 years ago

Add a toggle in the UI to enable.
Add check if file is large enough to encode in parallel.
Each record will be passed into a channel where they will be consumed by a couple of goroutines.
Typically encode 3 times faster than normal for large files.

*Maybe we could do a generic goroutine and apply it to all the other file types and databases. This only involves changing the channel type.

eatonphil commented 2 years ago

Closing for now