Markdown Package, "Apparent recursion within a with_prototype action: 25000 context sanity limit hit"

[BourgeoisBear]

Summary

Occasionally get the following error when opening a project:

Error loading syntax file "Packages/Markdown/Markdown.sublime-syntax": Apparent recursion within a with_prototype action: 25000 context sanity limit hit

Expected behavior

To not get that error.

Steps to reproduce

Open an existing Sublime Text 3 project with an open markdown file in the pane.

Environment

• Debian 9 (x64)
• 1080p x 2
• Sublime Text 3, Dev Channel, Build 3175

[keith-hall]

which third party packages do you have installed? related: https://github.com/SublimeTextIssues/Core/issues/1534

[BourgeoisBear]

Alignment Elixir HTMLBeautify Package Control Terminal

[sheriffderek]

I have a fresh computer - with a fresh Sublime install - and I'm getting this:

Error loading syntax file "Packages/Markdown/Markdown.sublime-syntax": Apparent recursion within a with_prototype action: 25000 context sanity limit hit

(anytime I access the .md file)

I've only installed:

Package Control.sublime-package JavaScriptNext - ES6 Syntax.sublime-package MarkdownPreview.sublime-package

(and I'm fairly sure that this alert came before I installed the markdown preview package)

EDIT: someone pointed out that it may be a JSNext issue... https://github.com/jonschlinkert/sublime-markdown-extended/issues/141#issuecomment-405517099 - after removing that / I get the syntax highlighting back - and no more warning.

[hashimaziz1]

Was having the same issue whenever opening .md files or selecting Markdown or MultiMarkdown syntax, and although I didn't have the JavaScriptNext package installed, I had to disable CSS3 markdown to get the error to stop occurring. Far from an ideal scenario, and something that by the looks of it should have been fixed years ago.

[sheriffderek]

Re: @kaos-industries ::: Well, it’s certainly a workaround for those of us who don’t understand the problem... so the next steps would be to determine if the problem is based in core - of if the problem is something in a package - or a combination. I don’t have the skills to find out... but the answer does exists. : )

[deathaxe]

This issue was discussed several times in the sublime forum. ST contains a sanity limit to avoid infinite loops triggered by bugs in syntax definitions and limit the processing time for performance and RAM usage reasons. This limits the complexity a syntax definition is allowed to have - number of contexts and rules being used to express a language.

Both the built-in and the 3rd-party syntaxes became more and more complex during the years. They tend to mature to some kind of static code analysis tool rather than concentrating on providing a base for syntax highlighting. Many meta. ... rules were added to identify class/function bodies and other language details.

Some languages like JavaScript require complex lexer rules to be able to provide a good base for syntax highlighting by nature. Some like CSS3 got too complex because of fetish like love in language details.

Situation gets out of control because modern web development requires many syntaxes to need to work in a single file. HTML/JavaScript/CSS as base extended by some 3rd-party syntax es to make React/.../TypeScript/... and all the none-standard sandbox languages work.

To display such none-standards ST needs to handle a syntax which easily consists of more then 5 already quite complex "default" syntaxes or other 3rd-party syntaxes which include others themselves.

Markdown e.g. includes CSS/JavaScript/JSON/HTML/XML/.... - anything fenced codeblocks can highlight, while HTML itself includes CSS/JavaScript again. ... So the tree of rules used to parse a file grows quickly.

It was discussed to increase that sanity limit, but it won't fix the issue in the long term as syntaxes won't stop becoming more complex.

In short: The reasons and the issue is well known but a solution is hard to find without breaking everything we have.

[BourgeoisBear]

@deathaxe

In that case, going to a markdown file and doing a View → Syntax → Open all with current extension as → Plain Text should get rid of the error message?

Maybe I'll do that for some other formats as well and give Linus Akesson's suggestion a try.

[deathaxe]

Yes, switching to Plain Text would help for such files. The error is displayed each time ST loads a syntax definition, that exceeds the sanity limit.

I still hope the core devs and/or syntax authors will find a proper solution to avoid this issue in near future as it can become really painful.

[sheriffderek]

@deathaxe - That was a very helpful explanation. That all makes sense. So refreshing!

[jrochkind]

I still don't understand what's going on and what should be done about it.

Should I just ignore these errors and click 'OK'? Do these errors mean that Markdown highlighting isn't actually going to be applied -- Markdown syntax highlighting is effectively broken anyway, and I should instead find a way to disable Markdown syntax highlighting, and then not get this error (and not lose anything since markdown syntax highlighting wasn't going to be applied anyway)?

Something else?

[wbond]

@jrochkind The error means that you've got a third party package installed that when combined with the default Markdown syntax is creating a include loop. Markdown syntax highlighting won't work. Sublime Text detected that the number of contexts in the syntax exploded to 25,000. This almost always indicates that there is an include loop. A typical syntax has a few hundred, or maybe a couple of thousand contexts. 25,000 is a huge number of different states in which the syntax definition can be at any given time.

I would uninstall third-party syntax definitions that try to replace/duplicate default syntaxes. A huge amount of work has been done to the default syntaxes over the past couple of years, so for most languages, the default syntaxes are some of the most accurate available, with the best performance.

If you are using the third-party package because there is a bug in the default syntax, please report it. If you just like the third-party syntax better, you, or someone else, will have some work ahead of them to figure out how to prevent the number of contexts exploding to 25,000.

[BourgeoisBear]

To me, not recursing into code blocks to highlight would be an ok performance compromise. Only part of markdown comprehension I miss is being able to jump to headers with CtrlP.

On Wed, Nov 21, 2018, 8:56 AM Jonathan Rochkind <notifications@github.com wrote:

I still don't understand what's going on and what should be done about it.

Should I just ignore these errors and click 'OK'? Do these errors mean that Markdown highlighting isn't actually going to be applied -- Markdown syntax highlighting is effectively broken anyway, and I should instead find a way to disable Markdown syntax highlighting, and then not get this error (and not lose anything since markdown syntax highlighting wasn't going to be applied anyway)?

Something else?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SublimeTextIssues/Core/issues/2444#issuecomment-440692620, or mute the thread https://github.com/notifications/unsubscribe-auth/AINbF2eJ4UPyT5L34VilOA3bteCcW-0Zks5uxWm4gaJpZM4XjGeF .

[jrochkind]

OK thanks @wbond , that is a helpful summary clarifying recommended actions.

Now I just need to figure out what third-party syntaxes I have installed, and delete them all, ha. Whatever I installed, I would not have done it because I thought there was a bug in the default syntax, but because I thought there was no support for the given syntax in the default install.

It looks like some I have installed (via Package Manager/list and looking for ones that look like syntax) are: "Sass", "Better coffeescript", "Pretty JSON", "JavaScriptNext - ES6 Syntax", and "Ruby Slim".

If all of these syntaxes are actually included by default, there's no need for them; I must have ended up with them by following tutorials/instructions with outdated information, or perhaps installed them a long time ago when I still did need them. It can be difficult to navigate the ST plugin landscape and figure out what you really need, it's tempting to just install what a tutorial on the internet suggests will give you a good workspace to start with for a given kind of work.

I will try uninstalling them all, and see if the "25000 context" error goes away -- and if my workspace still functions with syntax highlighting.

[jrochkind]

Actually, what would be REALLY helpful is if the ST error could tell you what other syntax highlighting packages are implicated in the recursion -- the error just says "Markdown", but you suggest this probably isn't really the culprit. If there is a bad-acting syntax highlighting plugin, if the error could identify it (or even suggest a list of which ones it could be), that would help users know what to avoid (or file bugs against) in third-party plugins.

(In my case, I believe it may have been the JavaScriptNext - ES6 Syntax plugin)

[wbond]

"JavaScriptNext - ES6 Syntax" should be removed in favor of the built-in JavaScript syntax. I don't know if it handles JSX, but if that is what you are looking for, you should use https://packagecontrol.io/packages/JSCustom. It is maintained by one of the community members who is the most active contributor to the JavaScript syntax.

We don't know what transitive include may be causing the issue. With Markdown, and other syntaxes we've seen hitting the 25,000 limit the transitive include graph is pretty significant. I will see if I can come up with any more useful info. Right now that message is printed in our compilation phase when adding a new context to the compile syntax. The context being compiled at that moment very likely isn't the direct cause, but I'll see if there is anything else we may be able to infer.

[jrochkind]

Thanks @wbond , this is super helpful.

I ended up re-installing the sass package, because that syntax highlighting did not seem to be included by default without it. May end up re-installing some others as I come accross files in my workflow that are not syntax highlighted, but I want them to be. Which is probably how I wound up with the ones I had before -- at some point for some reason I probably thought I needed a plugin for ES6, I may have been wrong at the time, or I may have been right then but then ST was updated subsequently. And of course, even if I was right (at some time) that I needed a package -- I may have accidentally chosen a poorly implemented one, when a better alternative was available.

I understand that quality control of third-party plugins is pretty much impossible. But the current "UX" for people using ST definitely leads to users kind of installing arbitrary plugins (found in package control list, or recommended on some blog post somewhere), and then there not being a good way to figure out if one of them is misbehaving. I suspect many ST users have a pretty extensive third-party plugin install list, including syntax highlighters.

I recognize that the "25000 limit" error is an effort to at least tell you that you have some misbehaving plugin installed. But it's actionability is limited, without being able to tell which one (and worse when it's text leads one to incorrectly believe it's the (built-in?) Markdown syntax highlighter). Any improvements that could even give you a list of suggested candidate "problem" plugins would be very helpful.

[jrochkind]

I notice that @sheriffderek provided a reproduction from a newly installed ST with very few plugins -- but with "JavaScriptNext - ES6 Syntax" being one of them.

Which matches my list too. And removing "JavaScriptNext - ES6 Syntax" for me is what seems to have resolved it too (I was trying to remove one by one and then restart ST to see when error went away, and that seemed to result in pointing at "JavaScriptNext - ES6 Syntax" as well).

For many people arriving at this ticket, it's probably "JavaScriptNext - ES6 Syntax" that's doing it, if you have that one installed.

I wonder if the maintainers of that plugin should be approached, and/or if it should be removed from Package Control. "JavaScriptNext - ES6 Syntax" is probably recommended on old/outdated blog or forum posts people can find on the internet -- finding other people's lists of recommended packages is a way many people get their ST setup into good shape for their workflow. For instance, this one.

"JavaScriptNext - ES6 Syntax" is in the top 100 packages installed, with 221K installs

[BourgeoisBear]

@wbond, @jrochkind

I was getting this error with Elixir as my only add-on syntax.

[wbond]

@BourgeoisBear I just installed the Elixir package and there was no issue. Most likely you have some out-of-date override, or something else going on.

Follow the directions at https://www.sublimetext.com/docs/3/revert.html and then try installing the Elixir package.

[BourgeoisBear]

When I filed the bug, build 3175 was the latest. Now I'm on 3180 (updated from apt I guess), and after re-enabling markdown syntax processing, the problem is gone--even with the Elixir syntax package.

[FichteFoll]

Although it looks that like people reporting this have solved their issue by now, I'll leave the issue open to catch more people running into this.

---

If you experience this error message and are unable to figure out where it comes from, please include a list of installed third-party packages. The default shipped packages do not cause this problem.

[kevinrue]

• Build 3207
• No update available And running into the issue with *.md files 😢

[FichteFoll]

Since we haven't had a report like this in quite a while, I'm closing the issue.

[laurent22]

That should not be closed as it's still happening. I just had it today in a Markdown file that doesn't contain any code blocks (just some inline code) and I don't know what code exactly or what plugin is causing the issue.

The result is that the Markdown file is rendered as plain text, which makes it hard to read. It's understandable that some error or loop can happen in a third party plugin, but the result should not be no highlighting at all.

I think two things would be needed:

• In case of a loop, cancel rendering for that particular plugin and display the code block as plain text. The rest of the file should be displayed with Markdown highlighting.
• Display what plugin is causing the loop so that we can file a bug on that plugin repo.

[JeremyBernier]

My syntax highlighting package (Lit-Element Syntax Highlighting) also causes this error https://github.com/JeremyBernier/LitElement-Syntax-Highlighting.

To explain how I made the package - I just took Babel's syntax highlighting file and added a couple rules to support Lit-Element template literals inside (html inside html<div>html code</div>, css inside css.class-name { color: red; }. Not sure why this bug is being triggered, but just providing this info in case anyone wants to investigate what might be causing it on Sublime's end.

[ibalpinar]

Hey,

I've no any package about Markdown right now and it seems the problem still continues.

The environment information that I use the applications is as follows;

Sublime Merge: Stable version, build 2020 Sublime Text: v3.2.2, build 3211 macOS: Catalina v10.15.5

And finally, please see attached file for screenshot.

Best, Ilker

[deathaxe]

I've no any package about Markdown

It is enough to have any package installed which provides a syntax definition, which is embedded in the builtin markdown package. As markdown embedds fairly many syntaxes the chance for that is quite high.

[laurent22]

This bug is what finally made me switch to Visual Studio Code. For a commercial application to not being able to load a Markdown file reflects quite poorly on the overall app quality. I can either have a working Markdown package but buggy JS package, or a good JS package (Naomi) and broken Markdown package, there's just no other options.

I feel quality went down over the past few years, and the only good thing left in Sublime is its speed, but I can do without given how many issues there are otherwise.

[laurent22]

Actually I hadn't noticed the bug was still closed. Well problem solved then I guess ¯\_(ツ)_/¯ Sublime can't even render a simple Markdown file but no need to fix.

[sheriffderek]

This bug is what finally made me switch to Visual Studio Code.

It's a slippery slope! If we don't build and maintain and support smaller companies... we'll just be stuck with all Microsoft products again soon. Github, VScode, Atom, NPM etc... it's "free" - but is it? See what they did there? Just kinda used our own sense of "a deal" to move all of our hard work over to a new platform that they own. They bought us.

It doesn't seem like anyone cares...

[wbond]

The issue is caused by a third-party syntax making a loop. The default Markdown syntax works perfectly. We can't prevent third-party authors from writing bad code. We prevent them from using all your memory by stopping context numbers increasing at 25,000.

If you start from a fresh copy of Sublime you'll find Markdown and JS highlighting to be top notch. We'll also have some of the best JSX and TypeScript support soon too.

[laurent22]

The issue is caused by a third-party syntax making a loop. The default Markdown syntax works perfectly. We can't prevent third-party authors from writing bad code. We prevent them from using all your memory by stopping context numbers increasing at 25,000.

A plugin misbehave? Ok close it, but that shouldn't break Sublime's own plugins. Not saying it's an easy problem to solve, but it seems Sublime has just accepted that's the way it is, that plugins can't be isolated from each others. It's a problem that's been solved in other apps so I'm sure it could be solved in Sublime too. The fact that this infinite loop issue happens with many different plugins is also an indication that it's not just the plugin developers who are doing something wrong.

As for the default JS highlighting, it's quite limited and breaks on certain JSX code. So I need to use a different one, which of course I can't because it will break Markdown. Anyway if the default JS highlighting was so great there wouldn't be so many alternatives.

[wbond]

A plugin misbehave? Ok close it, but that shouldn't break Sublime's own plugins.

In this case you installed a plugin that injected itself into the default Markdown syntax, on purpose. So no, we aren't going to stop that. You and the author wanted it that way. However, at run time we detected that the author did it in a way that things will blow up, which we don't want to happen. We tell you things were about to blow up, and we stop them from blowing up.

It's a problem that's been solved in other apps so I'm sure it could be solved in Sublime too.

We did solve the problem - we stopped the third party plugin from blowing up the app by not allowing the Markdown syntax to be loaded since the third party package injected what appears to be an infinite loop. We solved the issue! It is like the Chrome popup that tells you the tab has crashed. They prevented the tab from crashing the whole app. They can't magically make V8 not crash, but they can prevent it from affecting other things.

The fact that this infinite loop issue happens with many different plugins is also an indication that it's not just the plugin developers who are doing something wrong.

Most likely it is an indication that the plugin author is testing one situation, and hasn't tested the situation that you are using - possibly multiple third-party syntaxes with more than one override. Either way, we solved the issue that the combination of syntaxes you have seems to be causing an infinite loop by preventing the infinite loop.

As for the default JS highlighting, it's quite limited and breaks on certain JSX code.

It is true that we don't ship a JSX syntax by default, but the Markdown syntax doesn't include JSX either. So your third-party syntax seems to be injecting itself in an invalid way in place of JS, not just adding JSX support. If you are looking for JSX support, the best place for that is at https://packagecontrol.io/packages/JSCustom.

So I need to use a different one, which of course I can't because it will break Markdown.

These days you can use JSCustom.

Anyway ifl the default JS highlighting was so great there wouldn't be so many alternatives.

The default JavaScript syntax used to be subpar, but now it is probably one of the best. We only currently support JavaScript though, not JSX or TypeScript. For those you have to install a third-party package, but the good news is that JSCustom is an excellent syntax to use.

Another bit of good news is that in the near future you won't need to install a third-party syntax to get JSX, and you can avoid the headaches of poorly tested packages.

[deathaxe]

I feel the need to add one critical comment here as the main reason for this issue was identified to be caused by the with_prototype statement which is known to trigger infinite inclusion loops (see https://github.com/sublimehq/sublime_text/issues/1534). As it is the only way to extend existing syntaxes by injecting custom rules it is for sure to hit that issue when extending a "modern" (or more complex) syntax definition such as JS, CSS etc.

Therefore I'd be careful to blame 3rd party packages causing that issue by arguing about them being poorly designed.

A package nowerdays must literly not use with_prototype to ensure not to hit that issue. Even simple syntaxes with a very good and clear design cause that inclusion loop when extended via with_prototype. Therefore I'd call #1534 a major issue. If this issue was closed than as being a duplicate, IMHO.

It is a major core issue in the end!

[laurent22]

Thanks for the explanation @wbond . I don't know how Sublime is designed, but as a user I'd be fine if a JS package fails and, as a result, JS is not highlighted in the Markdown file. That would make sense to me.

However, what's happening now is that the whole file is no longer highlighted, so whatever the faulty package is doing, it is leaking into another non-faulty package. I guess if the other package is allowed to hook into the Markdown package at a low level then it might be difficult to prevent, but perhaps it's possible to give a more restricted access so that the other package can be switched off, and rendering of Markdown can resume.

[wbond]

In the case of Markdown, Markdown allows for HTML to be interspersed, and HTML has all of the onclick, etc attributes that accept JS. And that is how JS gets in the mix. Since the issue is infinite recursion, it is a hard problem to provide much info about. With the way that with_prototype currently works, the core thing identified in #1534 It is effectively the halting problem.

I appreciate that this is frustrating - it has been frustrating for me too since I do a lot of work related to our syntaxes, and we generally have really good coverage in the defaults. The community has put a ton of work into making trying to make them top-notch, and I've tried to make changes in core and write docs to help support that. I've raised this issue with at least one third-party package known to cause such an issue, and the package did not seem interested in changing the way they authored their syntax.

My hope is that with better coverage in the default we don't have to rely so much of third-party syntaxes for some of the more recently-common syntaxes (JSX and TypeScript come mostly to mind). The other upside of having better coverage is that we can ensure that the new features we release, that work based on syntaxes, will end up working for the majority of users.

[BourgeoisBear]

I'm on build 3211 now, and it eventually exhausts memory no matter how much I give it. I have to run sublime under ulimit to keep it from OOM-locking the system.

Noticed that #1288 relates with_prototype to excessive memory consumption. I may be missing the mechanics of with_prototype here, but why would the recursion depth for nested syntaxes ever need to go higher than 3-5? If some javascript snippet inside of an HTML attribute inside of a markdown code block doesn't get highlighted, that is preferable to what I am dealing with now.

[jrochkind]

I don't know if it's realistic, but the one major improvement would be if it could tell you WHICH plugin was responsible for the infinite loop stack overflow. The you just disable it, and report it to the plugin author.

As it is, when the nature of ST and it's users is that we all have a lot of plugins loaded... it's very difficult to know what to DO when you get that error message, how to make it go away. We all end up just getting it every time we launch ST. We don't know how to stop it. Which is why it gets frustrating.

The architecture might not make it realistic to do so, but if it could just tell you which plugin was the responsible party, it would be sort of self-correcting. The plugin author would get lots of feedback asking for it to be fixed. People could disable that plugin to avoid the error message. Etc.

Jonathan

[andreseduardop]

The issue is caused by a third-party syntax making a loop. The default Markdown syntax works perfectly. We can't prevent third-party authors from writing bad code. We prevent them from using all your memory by stopping context numbers increasing at 25,000.

If you start from a fresh copy of Sublime you'll find Markdown and JS highlighting to be top notch. We'll also have some of the best JSX and TypeScript support soon too.

I have posted an interesting test about it. https://github.com/sublimehq/sublime_text/issues/4061

If you have some time, I'd like you to take a look at it. Thanks.

[zyrup]

I had to remove /Users/user/Library/Application\ Support/Sublime\ Text\ 3/Installed\ Packages/JavaScriptNext\ -\ ES6\ Syntax.sublime-package