taraslayshchuk / es2csv

Export from an Elasticsearch into a CSV file
Apache License 2.0
510 stars 191 forks source link

Add JSON export + BigQuery compatibility #63

Open kshmir opened 5 years ago

kshmir commented 5 years ago

I was working with this tool to export the CSV outputs to BigQuery. But the amount of data I managed was over 10GB/day, taking an hour to convert to CSV, so for speeding things up I added a JSON export, since the JSON output with . and @ is not compatible with BigQuery, I added flags to make it compatible.

Usage: es2csv_cli -b -l '_' -j {... normal parameters}

Any feedback or improvement, I can add it.