Use `simd-json` for faster parsing

fsktom / rusty-endsong-parser

Better, more performant version of https://github.com/fsktom/endsong-parser-python written in Rust

0 stars 0 forks source link

Use `simd-json` for faster parsing #80

Open fsktom opened 3 months ago

fsktom commented 3 months ago

https://purplesyringa.moe/blog/i-sped-up-serde-json-strings-by-20-percent/

See how well it works on ARM-based M1, WSL and Windows

fsktom commented 3 months ago

https://lib.rs/crates/simd-json

For best performance, we highly suggest using mimalloc or jemalloc instead of the system allocator used by default. Another recent allocator that works well (but we have yet to test it in production) is snmalloc.

hmm maybe also a custom allocator? xd probably not

fsktom commented 2 months ago

base currently for 1 .json file

parse                   time:   [16.680 ms 16.727 ms 16.779 ms]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

for 10:

parse                   time:   [167.50 ms 169.41 ms 171.46 ms]
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild

fsktom commented 2 months ago

My first try:

parse                   time:   [20.865 ms 20.928 ms 21.001 ms]
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) high mild
  7 (7.00%) high severe

ok something went wrong xd

let mut file_contents: Vec<u8> = vec![];
File::open(path)?.read_to_end(&mut file_contents)?;
let full_entries: Vec<Entry> = simd_json::serde::from_slice(&mut file_contents)?;

is not the way to go...

fsktom commented 2 months ago

https://github.com/simd-lite/simd-json/issues/375#issuecomment-2137608680

ehh I'd have to use https://github.com/simd-lite/simd-json-derive ? but it doesn't support skip_deserializing attribute yet...?

so I guess I'll have to let it go for a while?