search

Shopify / liquid-c

Liquid performance extension in C.
MIT License
119 stars 24 forks source link

Make Liquid::C::Tokenizer#shift private so it is only used for testing #140

Closed dylanahsmith closed 3 years ago

dylanahsmith commented 3 years ago

Depends on https://github.com/Shopify/liquid/pull/1386

Problem

The liquid template (de)serialization support that is being added by https://github.com/Shopify/liquid-c/pull/138 won't actually support Liquid::C::Tokenizer#shift being used during parsing. The method would seem to work for the initial parsing, but then it would fail on deserialization where there is no longer a tokenizer.

Solution

Leverage https://github.com/Shopify/liquid/pull/1386 to update the tokenizer monkey patch to instead be on Liquid::ParseContext.new_tokenizer so that we avoid using the Liquid::C::Tokenizer when the liquid-c VM is disabled.

Make Liquid::C::Tokenizer#shift private, like shift_trimmed, and only use it for testing.