jmespath / jmespath.py

JMESPath is a query language for JSON.
http://jmespath.org
MIT License
2.19k stars 181 forks source link

Perf improvements for sub-expres and index-exprs #72

Closed jamesls closed 9 years ago

jamesls commented 9 years ago

Switch from a binary AST node for sub exprs and index exprs to an array of children nodes. That is, given the expr foo.bar.baz.qux, we go from:

original

To:

after

In terms of perf, we have before:

 python perf/perftest.py
lex_time: 0.01000ms, parse_time: 0.01900ms, search_time: 0.00500ms combined_time: 0.02500ms name: single_expression
lex_time: 0.01800ms, parse_time: 0.03400ms, search_time: 0.00800ms combined_time: 0.04400ms name: single_dot_expression
lex_time: 0.02600ms, parse_time: 0.04700ms, search_time: 0.01000ms combined_time: 0.06000ms name: double_dot_expression
lex_time: 0.03400ms, parse_time: 0.06200ms, search_time: 0.01500ms combined_time: 0.07900ms name: dot_no_match
lex_time: 0.08200ms, parse_time: 0.14700ms, search_time: 0.02500ms combined_time: 0.17400ms name: deep_nesting_10
lex_time: 0.40600ms, parse_time: 0.70600ms, search_time: 0.11300ms combined_time: 0.83500ms name: deep_nesting_50
lex_time: 0.40400ms, parse_time: 0.65900ms, search_time: 0.11300ms combined_time: 0.78800ms name: deep_nesting_50_pipe
lex_time: 0.69700ms, parse_time: 0.95500ms, search_time: 0.12100ms combined_time: 1.08900ms name: deep_nesting_50_index
lex_time: 0.05100ms, parse_time: 0.08300ms, search_time: 0.01900ms combined_time: 0.10400ms name: multi_wildcard_field
lex_time: 0.05300ms, parse_time: 0.08500ms, search_time: 0.02200ms combined_time: 0.10900ms name: wildcard_with_index
lex_time: 0.03000ms, parse_time: 0.05000ms, search_time: 0.01300ms combined_time: 0.06600ms name: wildcard_with_field_match
lex_time: 0.03100ms, parse_time: 0.05100ms, search_time: 0.01300ms combined_time: 0.06600ms name: wildcard_with_field_match2

After:

$ python perf/perftest.py
lex_time: 0.01000ms, parse_time: 0.01900ms, search_time: 0.00500ms combined_time: 0.02600ms name: single_expression
lex_time: 0.01800ms, parse_time: 0.03400ms, search_time: 0.00800ms combined_time: 0.04400ms name: single_dot_expression
lex_time: 0.02600ms, parse_time: 0.04900ms, search_time: 0.01000ms combined_time: 0.06000ms name: double_dot_expression
lex_time: 0.03300ms, parse_time: 0.06300ms, search_time: 0.01200ms combined_time: 0.07700ms name: dot_no_match
lex_time: 0.08000ms, parse_time: 0.14400ms, search_time: 0.01600ms combined_time: 0.16300ms name: deep_nesting_10
lex_time: 0.40100ms, parse_time: 0.69200ms, search_time: 0.05800ms combined_time: 0.75200ms name: deep_nesting_50
lex_time: 0.39600ms, parse_time: 0.64900ms, search_time: 0.11300ms combined_time: 0.78300ms name: deep_nesting_50_pipe
lex_time: 0.69100ms, parse_time: 0.90500ms, search_time: 0.06400ms combined_time: 0.97400ms name: deep_nesting_50_index
lex_time: 0.05100ms, parse_time: 0.08200ms, search_time: 0.01900ms combined_time: 0.10300ms name: multi_wildcard_field
lex_time: 0.05200ms, parse_time: 0.08400ms, search_time: 0.02200ms combined_time: 0.10800ms name: wildcard_with_index
lex_time: 0.03000ms, parse_time: 0.05100ms, search_time: 0.01300ms combined_time: 0.06600ms name: wildcard_with_field_match
lex_time: 0.03100ms, parse_time: 0.05100ms, search_time: 0.01300ms combined_time: 0.06500ms name: wildcard_with_field_match2

Note the deep_nesting_50 and the deep_nesting_50_index differences.

In addition to perf imrovements I also updated the perftest with a few changes:

This should have no functional changes, this is purely for perf concerns.

danielgtaylor commented 9 years ago

Aside from the question above this LGTM. :ship:-it!