Faster Expression parser / Tokenizer with StringScanner

[ggmichaelgo]

Faster Tokenizer & Expression Parser

Inspired by @ianks 's https://github.com/Shopify/liquid/pull/1832 PR, this PR updates the Expression Parser and Tokenizer to use the new StringScanner to improve the performance.

Faster Expression Parser!

Here are the performance improvements by using Expression2	Expression Type	Baseline (iter/s)	Optimized (iter/s)	Improvement (%)
String	1,642,330.4	1,618,752.0	same-ish
Literal	464,185.8	1,439,419.0	210% faster
Variable	170,784.0	1,130,120.6	562% faster
Number	142,954.2	385,356.9	170% faster
Range	57,236.9	570,767.7	897% faster
All	27,806.2	115,904.0	317% faster

Benchmark Improvement

main branch (baseline)

ruby 3.4.0dev (2024-04-23T16:59:11Z v3.4.0-pshopify-pr.. 70b931820e) +YJIT [arm64-darwin23]
Warming up --------------------------------------
              parse:    14.000 i/100ms
             render:    70.000 i/100ms
     parse & render:    11.000 i/100ms
Calculating -------------------------------------
              parse:    142.958 (± 1.4%) i/s    (7.00 ms/i) -      2.870k in  20.081552s
             render:    704.996 (± 1.7%) i/s    (1.42 ms/i) -     14.140k in  20.064078s
     parse & render:    114.978 (± 0.9%) i/s    (8.70 ms/i) -      2.310k in  20.091266s

fast-expression-parse branch

ruby 3.4.0dev (2024-04-23T16:59:11Z v3.4.0-pshopify-pr.. 70b931820e) +YJIT [arm64-darwin23]
Warming up --------------------------------------
           tokenize:   356.000 i/100ms
              parse:    18.000 i/100ms
             render:    70.000 i/100ms
     parse & render:    14.000 i/100ms
Calculating -------------------------------------
           tokenize:      3.522k (± 1.6%) i/s  (283.89 μs/i) -     70.488k in  20.017506s
              parse:    188.257 (± 0.5%) i/s    (5.31 ms/i) -      3.780k in  20.080262s
             render:    708.969 (± 0.7%) i/s    (1.41 ms/i) -     14.210k in  20.044211s
     parse & render:    142.434 (± 2.1%) i/s    (7.02 ms/i) -      2.856k in  20.065080s

This PR improves the parsing speed by ~24%, and parse & render speed by ~19%.

More Benchmark Results!

Here is another benchmarking with a larger and complex theme:

By using the LRU caching, Liquid is creating ~20% less objects, the parsing time is ~18% faster, and render time is ~9% faster.

With production traffic, we are seeing ~14% parsing time improvement:

	Baseline (staging-liquid-2)	Experiment (staging-liquid-1)	Diff
Average	19.9ms	17.4ms	~14% improvement
p99	82ms	71ms	~15% improvement
p95	52ms	45ms	~16% improvement
p90	42ms	36ms	~17% improvement
p75	28ms	24ms	~17% improvement
p50	17ms	15ms	~13% improvement