@XrXr pointed out to me that rb_raise isn't quite safe in C++. Long story short rb_raise bypass the C++ destructors which can lead to various leaks etc.
Notably simdjson::dom::parser has a destructor it uses to deallocate the memory it uses. I attached the leak.rb script, if you run it against maser you'll see memory usage go through the root very quickly.
The good new is that simdjson::dom::parser is designed to be re-used. When you call it again it re-use the memory it previously allocated, so by instantiating a single parser once, and then using only that one, we avoid leaks.
The added benefit is that it should be slightly more performant as parse, load, and load_many no longer need to build the parser.
@XrXr pointed out to me that
rb_raise
isn't quite safe in C++. Long story shortrb_raise
bypass the C++ destructors which can lead to various leaks etc.Notably
simdjson::dom::parser
has a destructor it uses to deallocate the memory it uses. I attached theleak.rb
script, if you run it against maser you'll see memory usage go through the root very quickly.The good new is that
simdjson::dom::parser
is designed to be re-used. When you call it again it re-use the memory it previously allocated, so by instantiating a single parser once, and then using only that one, we avoid leaks.The added benefit is that it should be slightly more performant as
parse
,load
, andload_many
no longer need to build the parser.Ref: https://github.com/simdjson/simdjson/blob/master/doc/performance.md#reusing-the-parser-for-maximum-efficiency Also note that this strategy can work because simdjson is thread safe for now (https://github.com/simdjson/simdjson/issues/681), if that change we might need a different strategy.