search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.69k stars 1.56k forks source link

Benchmark `simd_json` versus `serde_json` for encoding #1931

Open LucioFranco opened 4 years ago

LucioFranco commented 4 years ago

https://github.com/simd-lite/simdjson-rs takes advantage of deeper hardware parallelization techniques to provide a faster serialization/deserialization.

A couple downsides is that it requires us to set some special compile flags since it requires special hardware instructions like avx2 https://github.com/simd-lite/simdjson-rs/blob/master/.cargo/config

ghost commented 4 years ago

Note that if we are ever going to provide pre-built Vector binaries with these AVX instructions, we shouldn't add -C target-cpu=native to the compiler flags, but rather set something like -C target-feature=+avx2.

The reason is that target-cpu=native would pick the CPU model used on the CI machine, which is generally not known to us.

Licenser commented 4 years ago

If you got questions about simd-json I'm glad to lend a hand