don't allow invalid syntax inside comment tag

[ggmichaelgo]

What are you trying to solve?

https://github.com/Shopify/liquid/pull/1755 introduced a new feature that allows invalid code to be inside a comment tag.

For Liquid version 5.4.0, this is a valid template, and not with our current main branch:

{% comment %}
  {% assign a = 1
  {% endcomment %}
{% endcomment %}

The assign tag is not closed immediately, {% assign a = 1 {% endcomment %} is parsed as the assign tag.

Since the https://github.com/Shopify/liquid/pull/1755 changed the comment tag parser to use the BlockBody::FullTokenPossiblyInvalid regex pattern, it parses {% assign a = 1 {% endcomment %} to be the comment tag delimiter, and it raises a syntax error by the extra {% endcomment %} token.

How are you solving this issue?

In order to remove this breaking change, this PR updates the comment to parse tokens like BlockBody and not like raw tag. However, the comment tag will still allow unclosed tags such as this example:

{% comment %}
  {% if true %}
  {% if  %}
{% endcomment %}