axiomhq / cli

The power of Axiom on the command line.
https://axiom.co
MIT License
41 stars 11 forks source link

feat(ingest): support setting fields for csv without header row + csv batching #221

Closed lukasmalkmus closed 2 months ago

lukasmalkmus commented 7 months ago

CSV should be batchable as well!

Also adds the CSV headers feature for CSV files without a header row.

In line with https://github.com/axiomhq/axiom-go/pull/305, I also addressed the batching behaviour here and made it less aggressive in time (flush batch every 5s instead of every 1s) but also introduced batch size based flushing (every 10k), whatever condition is met first. Also files are now batched instead of only stdin data.