Some kinds of ERB commenting break highlighting

[csuhta]

Prerequisites

• [X] Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

Certain kinds of ERB commenting will break the syntax highlighting flow, such as <% # comment %> and <%-# comment %>

Steps to Reproduce

A poorly colorized file shown below:

<%# This is a ERB comment %>
<%= @object.variable %>

<%-# This is also a ERB comment %> <!-- Styles broken past this point --->
<%= @object.variable %>

<%# This is a comment ERB %>
<%= @object.variable %>

<% # This is a comment %> <!-- Styles broken past this point --->
<%= @object.variable %> 

Expected behavior: Syntax highlighting handles the comment

Actual behavior: Syntax highlighting breaks down for future Ruby blocks after the comment

Edit by @rsese to add screenshot

Reproduces how often: Always

Versions

Edit by @rsese: also reproduced with 1.37.0-nightly12 on macOS 10.12.

❯❯ apm --version
apm  2.1.2
npm  6.2.0
node 8.9.3 x64
atom 1.33.1
python 2.7.15
git 2.20.1

❯❯ atom --version
Atom    : 1.33.1
Electron: 2.0.16
Chrome  : 61.0.3163.100
Node    : 8.9.3

[rsese]

Thanks for the report! Reproduced with 1.37.0-nightly12 on macOS 10.12.6.

[dreamalligator]

confirmed that this is a tree-sitter issue, so I turned this back off and no longer a zillion errors:

> apm --version
apm  2.1.3
npm  6.2.0
node 8.9.3 x64
atom 1.36.0-beta1
python 2.7.15+
git 2.19.1

> atom --version
Atom    : 1.36.0-beta1
Electron: 2.0.18
Chrome  : 61.0.3163.100
Node    : 8.9.3

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.10
Release:    18.10
Codename:   cosmic

[jasonrudolph]

Actual behavior: Syntax highlighting breaks down for future Ruby blocks after the comment

☝️ This is a nice summary of the issue. Thanks, @csuhta.

@maxbrunsfeld and I looked into this issue yesterday. Building on the summary above, the syntax highlighting works correctly when using an ERB-style comment (i.e., <%#), but it breaks when using an ERB statement that includes a Ruby-style comment (i.e., a <%, <%-, or <%= node that contains a Ruby comment ):

We found that this issue also exists for EJS:

It seems like fixing this issue is going to be a rather substantial undertaking, so I don't yet know when we'll be able to address this issue. In the meantime, if you're able to use ERB-style comments exclusively (which I know won't be a viable solution for everyone), then you can work around this bug.

We'll likely open an issue with additional details from our findings, since the underlying issue goes beyond language-ruby and ERB.

[jasonrudolph]

tl;dr: https://github.com/atom/atom/pull/19291 resolves the most common cases where this issue crops us, and the fix is targeted for inclusion in Atom 1.38. 😅

Before and After

Ruby comments

Before

After

ERB comments

Before

After

Remaining edge case and available workarounds

In the fairly uncommon case where you have a Ruby comment in an ERB directive followed by additional ERB directives on the same line, the latter ERB directives are currently styled incorrectly:

We're tracking this bug in https://github.com/tree-sitter/tree-sitter/issues/327. As seen in the screenshot above, there are two workarounds for this bug in the meantime:

• Option 1: Add a newline after any ERB directives that use Ruby comments, or
• Option 2: Use ERB comments instead of Ruby comments

Because the problem identified in the issue body above is now resolved, and because we're tracking the remaining edge case in a separate issue, I'm going to close this issue out.

Thanks again for reporting this, @csuhta. :bow: