First I would love to profile q and try to understand where are the pain points, and I have no clue how to do that yet.
Assumption
The biggest operation is reading the file and parsing the JSON if the JSON is huge, this becomes more of a problem.
Exploration
I would expect OCaml multicore to help on the parallelization of parse/compile the user input while parse the JSON, that would speed up things, but I would love to parse the parts of the JSON that q needs, instead of parsing the entire file.
The other idea was improving the parsing of the JSON perse, there an exploration to that, which tries to make the JSON parsing a parallelizable operation, it's called "Stack Monoid" https://raphlinus.github.io/gpu/2020/09/05/stack-monoid.html.
Assumption
query-json's AST is recursive which makes the compilation to be recursive as well and it's tail-recursive, I heard that OCaml compiler optimizes the TL.
while jq, is a stack based described here and they are doing a lot of more work than query-json.
Exploration
See how the OCaml compiler optimizes our TL, and check that the recursion is TL.
Explain better how the jq Stack works.
Assumption
menhir it's a parser generator, which you write your rules in .mly files and OCaml creates the parser for you. It's obvious that having a parser written natively sounds that could be more optional in terms of performance.
Exploration
How much effort would be to turn the Parser.mly into a regular Reason file and how much it will improve performance.
davesnx
commented
4 years ago
First I would love to profile q and try to understand where are the pain points, and I have no clue how to do that yet.
Assumption
The biggest operation is reading the file and parsing the JSON if the JSON is huge, this becomes more of a problem.
Exploration
I would expect OCaml multicore to help on the parallelization of parse/compile the user input while parse the JSON, that would speed up things, but I would love to parse the parts of the JSON that q needs, instead of parsing the entire file.
The other idea was improving the parsing of the JSON perse, there an exploration to that, which tries to make the JSON parsing a parallelizable operation, it's called "Stack Monoid" https://raphlinus.github.io/gpu/2020/09/05/stack-monoid.html.
Assumption
query-json's AST is recursive which makes the compilation to be recursive as well and it's tail-recursive, I heard that OCaml compiler optimizes the TL.
while jq, is a stack based described here and they are doing a lot of more work than query-json.
Exploration
See how the OCaml compiler optimizes our TL, and check that the recursion is TL. Explain better how the jq Stack works.
Assumption
menhir it's a parser generator, which you write your rules in .mly files and OCaml creates the parser for you. It's obvious that having a parser written natively sounds that could be more optional in terms of performance.
Exploration
How much effort would be to turn the Parser.mly into a regular Reason file and how much it will improve performance.