Leaks scope when embedded in other grammars

[lierdakil]

Prerequisites

• [X] Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

When embedding in other grammars (most notably Markdown), yaml leaks scope like crazy.

Example:

Also happens with Pandoc-style YAML blocks:

Naked ... should be treated as YAML document end, but it's instead treated as string, for some reason.

Steps to Reproduce

Should be obvious, try the following Markdown documents:

```yaml
example: string
```

scope is leaking here

---
example: string
...

scope is leaking here

Expected behavior: No scope leakage

Actual behavior: Much scope leakage

Reproduces how often: 100%

Versions

I'm pretty sure this started to be an issue since Atom-1.13 or around that.

Additional Information

None

[winstliu]

Yeah, so the end patterns are not designed with any notion of embedding in mind. Looks like they are over-capturing though.

Please create a different issue for the ... tokenization. That one will be much easier to fix.

[lierdakil]

I should probably point out that this:

attribute: value
naked string

is not valid YAML by any stretch of the imagination. naked string is not a string, not any valid token, it's just plain invalid in this context.

So I guess the real problem is that end patterns don't terminate properly, period, not that those "were not designed with embedding in mind".

[winstliu]

Yeah, but if you add a space it is. So the check isn't strict enough.

[winstliu]

Turns out that I had to fix the root embedding issue to make the document end marker work. So expect to see a PR soon.