Hanging/blocking with Markua code blocks

[abingham]

I'm trying to use mmm-mode for syntax highlighting in the markdown (technically "markua") source for a leanpub book I'm writing. The book has a lot of Python examples, so most of my code blocks are for Python. In some cases, though, the mmm-mode parsing seems to lock Emacs up, making it unreponsive and giving me the wait cursor.

The code blocks looks like this:

{language=python}
~~~~~~~~
python code goes here
~~~~~~~~

I wrote a mmm class for finding these:

(mmm-add-classes
       '((markua-python
          :submode python-mode
          :front "^{.*lang\\(uage\\)?=\"?python\"?.*}[\r\n]+~\\{8,\\}[\r\n]+"
          :back "^~\\{8,\\}$")))
(mmm-add-mode-ext-class 'markdown-mode nil 'markua-python)

In nearly all cases this seems to work well. However, the following snippet throws it into a tail spin:

Note that in Python3 version prior to 3.3, `__path__` was just a
single string, not a list. In this course we're focusing on Python
3.3, but for most purposes the difference is not important.

{language=python}
~~~~~~~~
>>> import sys
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/rope_py3k-0.9.4-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/decorator-3.4.0-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/Baker-1.3-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/beautifulsoup4-4.1.3-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pymongo-2.3-py3.3-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/eagertools-0.3-py3.3.egg', '/Users/abingham/projects/emacs_config/traad/traad', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/bottle-0.11.6-py3.3.egg', '/Users/abingham/projects/see_stats', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/waitress-0.8.5-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pystache-0.5.3-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pyramid_tm-0.7-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pyramid_debugtoolbar-1.0.6-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pyramid-1.4.3-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/transaction-1.4.1-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/Pygments-1.6-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/PasteDeploy-1.5.0-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/translationstring-1.1-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/venusian-1.0a8-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/zope.deprecation-4.0.2-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/zope.interface-4.0.5-py3.3-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/repoze.lru-0.6-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/WebOb-1.2.3-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/Mako-0.8.1-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/Chameleon-2.11-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/MarkupSafe-0.18-py3.3-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pip-1.4.1-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/ipython-1.0.0-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pandas-0.12.0-py3.3-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/setuptools-1.1.6-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/readline-6.2.4.1-py3.3-macosx-10.6-intel.egg', '/Users/abingham/projects/see_stats/distribute-0.6.49-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/nltk-2.0.4-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/PyYAML-3.10-py3.3-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/numpy-1.8.0-py3.3-macosx-10.6-intel.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/grin-1.2.1-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/argparse-1.2.1-py3.3.egg', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python33.zip', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/plat-darwin', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages']
~~~~~~~~

It's some interaction between the various parts of this text, but I'm not sure which. If I leave out the large list (the obvious standout part of the sample), then everything works fine. Likewise, if I leave out the text above the code sample then it works fine.

I assume that the problem ultimately lies in the regex I'm using, but I'm not sure where. Does anyone have any ideas?

[abingham]

This may not bear on the problem, but if I start Emacs from the command line and have it open a file that exhibits this problem, the message area showsnot enabling jit-lock: it does not work in indirect buffer.

Also, when Emacs goes into this bad state it's pegging two of my cores at about 50-60% each.

[dgutov]

Looks like the regexp is not at fault. It's python-mode's font-lock-syntactic-face-function that infloops.

Here's what the backtrace looks like (you can get it by M-x toggle-debug-on-quit and pressing C-g a few times when Emacs freezes):

Debugger entered--Lisp error: (quit)
  python-nav-end-of-statement()
  python-info-end-of-statement-p()
  python-info-end-of-block-p()
  python-nav--forward-sexp(-1 nil nil)
  python-nav-forward-sexp(-1 nil nil)
  python-nav-backward-sexp()
  python-info-docstring-p((1 249 nil t nil nil 0 nil 250 (249)))
  python-font-lock-syntactic-face-function((1 249 nil t nil nil 0 nil 250 (249)))
  font-lock-fontify-syntactically-region(221 4529 nil)
  font-lock-default-fontify-region(221 4529 nil)
  funcall(font-lock-default-fontify-region 221 4529 nil)
  (let ((font-lock-dont-widen t) syntax-ppss-last syntax-ppss-cache) (if (and ovl (not (memq mode mmm-c-derived-modes))) (progn (narrow-to-region beg end))) (funcall func beg end nil))
  (save-restriction (let ((font-lock-dont-widen t) syntax-ppss-last syntax-ppss-cache) (if (and ovl (not (memq mode mmm-c-derived-modes))) (progn (narrow-to-region beg end))) (funcall func beg end nil)))
  (progn (goto-char beg) (mmm-set-current-pair mode ovl) (mmm-set-local-variables (if (eq mmm-previous-submode mode) nil mode) mmm-current-overlay) (save-restriction (let ((font-lock-dont-widen t) syntax-ppss-last syntax-ppss-cache) (if (and ovl (not (memq mode mmm-c-derived-modes))) (progn (narrow-to-region beg end))) (funcall func beg end nil))) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay))
  (let* ((--cl-rest-- reg) (beg (if (= (length --cl-rest--) 3) (car-safe (prog1 --cl-rest-- (setq --cl-rest-- (cdr --cl-rest--)))) (signal (quote wrong-number-of-arguments) (list nil (length --cl-rest--))))) (end (car-safe (prog1 --cl-rest-- (setq --cl-rest-- (cdr --cl-rest--))))) (ovl (car --cl-rest--))) (progn (goto-char beg) (mmm-set-current-pair mode ovl) (mmm-set-local-variables (if (eq mmm-previous-submode mode) nil mode) mmm-current-overlay) (save-restriction (let ((font-lock-dont-widen t) syntax-ppss-last syntax-ppss-cache) (if (and ovl (not (memq mode mmm-c-derived-modes))) (progn (narrow-to-region beg end))) (funcall func beg end nil))) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay)))
  (lambda (reg) (let* ((--cl-rest-- reg) (beg (if (= (length --cl-rest--) 3) (car-safe (prog1 --cl-rest-- (setq --cl-rest-- ...))) (signal (quote wrong-number-of-arguments) (list nil (length --cl-rest--))))) (end (car-safe (prog1 --cl-rest-- (setq --cl-rest-- (cdr --cl-rest--))))) (ovl (car --cl-rest--))) (progn (goto-char beg) (mmm-set-current-pair mode ovl) (mmm-set-local-variables (if (eq mmm-previous-submode mode) nil mode) mmm-current-overlay) (save-restriction (let ((font-lock-dont-widen t) syntax-ppss-last syntax-ppss-cache) (if (and ovl (not ...)) (progn (narrow-to-region beg end))) (funcall func beg end nil))) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay))))((221 4529 #<overlay from 221 to 4529 in asdasdasd>))
  mapc((lambda (reg) (let* ((--cl-rest-- reg) (beg (if (= (length --cl-rest--) 3) (car-safe (prog1 --cl-rest-- (setq --cl-rest-- ...))) (signal (quote wrong-number-of-arguments) (list nil (length --cl-rest--))))) (end (car-safe (prog1 --cl-rest-- (setq --cl-rest-- (cdr --cl-rest--))))) (ovl (car --cl-rest--))) (progn (goto-char beg) (mmm-set-current-pair mode ovl) (mmm-set-local-variables (if (eq mmm-previous-submode mode) nil mode) mmm-current-overlay) (save-restriction (let ((font-lock-dont-widen t) syntax-ppss-last syntax-ppss-cache) (if (and ovl (not ...)) (progn (narrow-to-region beg end))) (funcall func beg end nil))) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay)))) ((221 4529 #<overlay from 221 to 4529 in asdasdasd>)))
  (let ((func (get mode (quote mmm-fontify-region-function))) font-lock-extend-region-functions) (mapc (function (lambda (reg) (let* ((--cl-rest-- reg) (beg (if ... ... ...)) (end (car-safe ...)) (ovl (car --cl-rest--))) (progn (goto-char beg) (mmm-set-current-pair mode ovl) (mmm-set-local-variables (if ... nil mode) mmm-current-overlay) (save-restriction (let ... ... ...)) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay))))) regions))
  (save-excursion (let ((func (get mode (quote mmm-fontify-region-function))) font-lock-extend-region-functions) (mapc (function (lambda (reg) (let* ((--cl-rest-- reg) (beg ...) (end ...) (ovl ...)) (progn (goto-char beg) (mmm-set-current-pair mode ovl) (mmm-set-local-variables ... mmm-current-overlay) (save-restriction ...) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay))))) regions)))
  mmm-fontify-region-list(python-mode ((221 4529 #<overlay from 221 to 4529 in asdasdasd>)))
  (progn (mmm-fontify-region-list (car elt) (cdr elt)))
  (if (get (car elt) (quote mmm-font-lock-mode)) (progn (mmm-fontify-region-list (car elt) (cdr elt))))
  (lambda (elt) (if (get (car elt) (quote mmm-font-lock-mode)) (progn (mmm-fontify-region-list (car elt) (cdr elt)))))((python-mode (221 4529 #<overlay from 221 to 4529 in asdasdasd>)))
  mapc((lambda (elt) (if (get (car elt) (quote mmm-font-lock-mode)) (progn (mmm-fontify-region-list (car elt) (cdr elt))))) ((python-mode (221 4529 #<overlay from 221 to 4529 in asdasdasd>)) (markdown-mode (1 221 nil) (4529 4538 nil))))
  (progn (if loudly (progn (message "Fontifying %s with submode regions..." (buffer-name)))) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay) (mapc (function (lambda (elt) (if (get (car elt) (quote mmm-font-lock-mode)) (progn (mmm-fontify-region-list (car elt) (cdr elt)))))) (mmm-regions-alist start stop)))
  (unwind-protect (progn (if loudly (progn (message "Fontifying %s with submode regions..." (buffer-name)))) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay) (mapc (function (lambda (elt) (if (get (car elt) (quote mmm-font-lock-mode)) (progn (mmm-fontify-region-list ... ...))))) (mmm-regions-alist start stop))) (mmm-set-current-pair saved-mode saved-ovl) (mmm-set-local-variables (or saved-mode mmm-primary-mode) saved-ovl))
  (let ((saved-mode mmm-current-submode) (saved-ovl mmm-current-overlay)) (unwind-protect (progn (if loudly (progn (message "Fontifying %s with submode regions..." (buffer-name)))) (mmm-save-changed-local-variables mmm-current-submode mmm-current-overlay) (mapc (function (lambda (elt) (if (get ... ...) (progn ...)))) (mmm-regions-alist start stop))) (mmm-set-current-pair saved-mode saved-ovl) (mmm-set-local-variables (or saved-mode mmm-primary-mode) saved-ovl)))
  mmm-fontify-region(1 4538 nil)

Not sure how easy that would be to fix.

[abingham]

I'm having trouble getting that stack trace because, for whatever reason, C-g isn't kicking me into the debugger (or doing anything else, for that matter). Did you put a breakpoint somewhere after python-font-lock-syntactic-face-function to verify that execution is actually never leaving that function?

I guess what puzzles me is that Python's font-locking works fine if I remove the text above the code sample. And from what I can tell, python-mode's role in all of this is limited to font-locking a region of text that is handed to it, i.e. it never sees the surrounding context (or, if it does, that seems like a bug). Given all of this, it seems like the blame can't be laid entirely on python-mode; some external factor --- mmm-mode, narrowing, the regex, or something --- is either causing the problem entirely or is somehow contributing to it.

I'll keep poking this as I have time to see if I can figure anything out.

[abingham]

FWIW, in the markdown sample I included originally, I can make the problem go away by removing the apostrophe in "we're". I stumbled onto this because, at one point in debugging, I noticed that font-locking was treating that apostrophe as (what appeared to be) the start of a string literal.

Looking at @dgutov's stack trace, the argument to python-font-lock-syntactic-face-function is interesting. If I'm reading the documentation for syntax-ppss correctly (I think that's where this value comes from), the 4th element should only be "true" if the state is "inside a string". So perhaps the apostrophe in "we're" is tricking some part of the font-locking system into thinking that a string has been started, and this is influencing how Python tries to do font-locking.

So perhaps the real problem here is the general font-lock mechanism, not Python font-locking per se. Or perhaps mmm-mode is using font-locking in a slightly incorrect way. I'm a bit out of my depth here, but this bug sure feels like the confluence of a few interacting factors.

[abingham]

Another data-point in support of python-mode being part of the problem: if I change the :submode from python-mode to something else (I tried about 5 other language modes), the problem doesn't appear. So the submode does seem to be pretty significant.

[dgutov]

I'm having trouble getting that stack trace because, for whatever reason, C-g isn't kicking me into the debugger (or doing anything else, for that matter).

Indeed, I can't trigger the debugger that way anymore. Seems like it was luck the first time.

Still, the below patch kind of fixes the freezing:

diff --git a/mmm-region.el b/mmm-region.el
index 547b351..a98845f 100644
--- a/mmm-region.el
+++ b/mmm-region.el
@@ -804,6 +804,8 @@ of the REGIONS covers START to STOP."
                                            mmm-current-overlay)
                   (save-restriction
                     (let ((font-lock-dont-widen t)
+                          (font-lock-syntactic-face-function
+                           (default-value 'font-lock-syntactic-face-function))
                           syntax-ppss-last syntax-ppss-cache)
                       ;; TODO: Remove this conditional when cc-mode
                       ;; respects submode boundaries.

I say "kind of" because in the resulting buffer any subsequent change to the Python subregion (e.g. press C-d) leads to an infloop of some kind. Those one can be easily broken with C-g, however.

[dgutov]

some external factor --- mmm-mode, narrowing, or something --- is either causing the problem entirely or is somehow contributing to it

Indeed, narrowing (and/or subsequent widening: there are quite a few widen calls in python.el) seems like a possible culprit.

[dgutov]

FWIW, in the markdown sample I included originally, I can make the problem go away by removing the apostrophe in "we're". I stumbled onto this because, at one point in debugging, I noticed that font-locking was treating that apostrophe as (what appeared to be) the start of a string literal.

Yes, I wonder what and when performs that highlighting.

As a wild guess, in that situation some python.el code might be seeing that it's inside a string and it could be trying to get out of that string by traversing toward the beginning of the buffer. If it's limited by a narrowing, however, that could lead to an infloop.

[abingham]

it could be trying to get out of that string by traversing toward the beginning of the buffer

Yeah, I wondered the same thing. The call to python-nav-backward-sexp seems to indicate that some backwards searching is possible.

[dgutov]

This looks relevant: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24856#8

[abingham]

Yep, that sure looks suspicious.