datafusion-contrib / bdt

Boring Data Tool
Apache License 2.0
210 stars 20 forks source link

"One file output" option added for csv/json. #21

Closed keltia closed 9 months ago

keltia commented 10 months ago

The default output for csv and json files by datafusion is to output a series of files inside a directory named as the output file. I find it quite inconvenient so I have added -o/--one-file option. I have switched to clapa long time ago so I'm not sure how to rename the option (I agree -o is not ideal) so feel free to change it 😁. Thanks a lot for bdt BTW, it beats writing it myself.

kassemz commented 9 months ago

Thanks @andygrove for creating this nice tool! and thanks @keltia you added what I was looking for!

What about -ns/--no-split or -s/--single-file as other options?

keltia commented 9 months ago

I renamed -o into -s and --single-file. I also added support for Zstd. And updated datafusion and tokio.

andygrove commented 9 months ago

@keltia Could you fix the merge conflicts?

keltia commented 9 months ago

Oh sorry, didn't see your request but I see you got it done, thanks for the merge!