make tokenizer shift function to be public

Shopify / liquid-c

Liquid performance extension in C.

MIT License

120 stars 25 forks source link

make tokenizer shift function to be public #209

Closed ggmichaelgo closed 11 months ago

ggmichaelgo commented 11 months ago

This PR is for https://github.com/Shopify/liquid/pull/1755

In the Liquid PR The comment tag requires to consume the token without creating any child nodes. The PR will change the comment tag to not parse its body, and allow incomplete Liquid codes as a comment.