Open mgirlich opened 2 years ago
Would RcppSimdJson be faster than jsonlite already, perhaps fast enough for most use cases (even if we don't support parsing only subsets)? Could we achieve laziness (and speedup for partial parsing) with ALTREP?
https://cran.r-project.org/web/packages/RcppSimdJson/index.html
I know about RcppSimdJson
and thought resp. think about contributing to it. At least for now I wanted to create a new package because:
simdjson
simdjson
I am currently finishing my work on a first prototype. Would you be interested (and have time) to contribute a little to this project?
Thanks. I'm happy to take a look, I can't commit substantial time to that project though.
On the other hand, I wonder what it takes to bring a green "stable" badge to tibblify. I wonder if e.g. we could reuse the existing tests and infrastructure in tibblify to avoid reinventing substantial parts of the wheel in the new project.
jsonlite
: 1) use a faster JSON parser and 2) use C++ to parse into the correct structure instead ofR
jsonlite
isn't vectorised but pastes input together. When working with JSON columns this is very annoying and slow.NA
:jsonlite
always produces an error message forNA
input. The user should be able to control this (especially in a vectorised version).jsonlite
: this is one of the strengths ofjsonlite
jsonlite
: there is currently no parser that allows the user to specify how the output of the parsing should look like. Allowing the user to do so is more robust and much fasterpurrr
.jsonlite
there is only a hidden option and it is not very robust as sometimes your numbers might all be small enough to actually fit into an integer.jsontools
package: e.g.json_unnest_longer()
json_unnest_wider()
; probably this is by now possible withmutate()
+ user specified structure for parsingjson_hoist()
; probably the same as forjson_unnest_wider()