not documented that 1.6 breaks recommonmark

[ratiotile]

It's shown in the documentation that recommonmark may be used to enable markdown support: http://www.sphinx-doc.org/en/stable/markdown.html

However, it seems that Sphinx 1.5.x was the last working version for recommonmark 0.4.0, and that 1.6 causes this error:

Exception occurred:
  File "python3.4/site-packages/recommonmark/states.py", line 134, in run_role
    content=content)
TypeError: 'NoneType' object is not callable

It would help prevent future users from wasting time if the docs were updated with this warning of incompatibility between recommonmark 0.4.0 and Sphinx>=1.6.

Edit: The bug seems to be related to markdown links, see: https://raw.githubusercontent.com/ratiotile/sphinx-1.6-recommonmark-bug-repro/master/source/js_api.md

[jfbu]

On a minimal project following the http://www.sphinx-doc.org/en/stable/markdown.html instructions, I have no issue.

# Sphinx version: 1.6.2+
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6

Can you share project causing the error you report ? We need more details.

[jfbu]

I added to my test project https://daringfireball.net/projects/markdown/syntax.text as markdownsyntax.md, and have no issue.

# Sphinx version: 1.6.1
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6
# Recommonmark version: 0.4.0

[jfbu]

Any update or comment? I wish to close this as I could not confirm. Maybe there is some special setup at your locale.

[lorengordon]

I'm also seeing a problem with markdown and sphinx >= 1.6. this setup:

from recommonmark.parser import CommonMarkParser

source_suffix = ['.md', '.rst']
source_parsers = {
    '.md': CommonMarkParser,
}

results in this traceback:

Traceback (most recent call last):
  File "c:\python35\lib\site-packages\sphinx\cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "c:\python35\lib\site-packages\sphinx\application.py", line 229, in __init__
    self._init_source_parsers()
  File "c:\python35\lib\site-packages\sphinx\application.py", line 266, in _init_source_parsers
    self.add_source_parser(suffix, parser)
  File "c:\python35\lib\site-packages\sphinx\application.py", line 739, in add_source_parser
    self.registry.add_source_parser(suffix, parser)
  File "c:\python35\lib\site-packages\sphinx\registry.py", line 160, in add_source_parser
    raise ExtensionError(_('source_parser for %r is already registered') % suffix)
sphinx.errors.ExtensionError: source_parser for '.md' is already registered

Extension error:
source_parser for '.md' is already registered

And if I remove source_parsers, I get this traceback:

Traceback (most recent call last):
  File "c:\python35\lib\site-packages\sphinx\cmdline.py", line 306, in main
    app.build(opts.force_all, filenames)
  File "c:\python35\lib\site-packages\sphinx\application.py", line 332, in build
    self.builder.build_all()
  File "c:\python35\lib\site-packages\sphinx\builders\__init__.py", line 285, in build_all
    self.build(None, summary='all source files', method='all')
  File "c:\python35\lib\site-packages\sphinx\builders\__init__.py", line 341, in build
    updated_docnames = set(self.env.update(self.config, self.srcdir, self.doctreedir))
  File "c:\python35\lib\site-packages\sphinx\environment\__init__.py", line 584, in update
    self._read_serial(docnames, self.app)
  File "c:\python35\lib\site-packages\sphinx\environment\__init__.py", line 603, in _read_serial
    self.read_doc(docname, app)
  File "c:\python35\lib\site-packages\sphinx\environment\__init__.py", line 718, in read_doc
    pub.publish()
  File "c:\python35\lib\site-packages\docutils\core.py", line 218, in publish
    self.apply_transforms()
  File "c:\python35\lib\site-packages\docutils\core.py", line 199, in apply_transforms
    self.document.transformer.apply_transforms()
  File "c:\python35\lib\site-packages\docutils\transforms\__init__.py", line 171, in apply_transforms
    transform.apply(**kwargs)
  File "c:\python35\lib\site-packages\recommonmark\transform.py", line 325, in apply
    self.traverse(self.document)
  File "c:\python35\lib\site-packages\recommonmark\transform.py", line 297, in traverse
    self.traverse(child)
  File "c:\python35\lib\site-packages\recommonmark\transform.py", line 297, in traverse
    self.traverse(child)
  File "c:\python35\lib\site-packages\recommonmark\transform.py", line 297, in traverse
    self.traverse(child)
  File "c:\python35\lib\site-packages\recommonmark\transform.py", line 287, in traverse
    newnode = self.find_replace(c)
  File "c:\python35\lib\site-packages\recommonmark\transform.py", line 267, in find_replace
    newnode = self.auto_doc_ref(node)
  File "c:\python35\lib\site-packages\recommonmark\transform.py", line 175, in auto_doc_ref
    return self.state_machine.run_role('doc', content=content)
  File "c:\python35\lib\site-packages\recommonmark\states.py", line 134, in run_role
    content=content)
TypeError: 'NoneType' object is not callable

Exception occurred:
  File "c:\python35\lib\site-packages\recommonmark\states.py", line 134, in run_role
    content=content)
TypeError: 'NoneType' object is not callable

[ratiotile]

reproduction at: https://github.com/ratiotile/sphinx-1.6-recommonmark-bug-repro

[jfbu]

Can you try with added quotes:

source_suffix = ['.md', '.rst']
source_parsers = {
    '.md': 'CommonMarkParser',
}

[jfbu]

I actually meant:

source_parsers = {
   '.md': 'recommonmark.parser.CommonMarkParser',
}

source_suffix = ['.rst', '.md']

with an import recommonmark in conf.py. Works for me.

edit: well, sorry it also works for me using

from recommonmark.parser import CommonMarkParser

source_suffix = ['.md', '.rst']
source_parsers = {
    '.md': CommonMarkParser,
}

[lorengordon]

Adding quotes around the parser does not make any difference. Same traceback/error: source_parser for '.md' is already registered.

[ratiotile]

I don't see a difference. I changed accordingly:

+import recommonmark
 from recommonmark.transform import AutoStructify

 # If extensions (or modules to document with autodoc) are in another directory,
@@ -36,10 +37,10 @@ extensions = []
 # Add any paths that contain templates here, relative to this directory.
 templates_path = []

-source_parsers = {'.md': 'recommonmark.parser.CommonMarkParser'}
+source_parsers = {
+   '.md': 'recommonmark.parser.CommonMarkParser',
+}

Still get the error:

Running Sphinx v1.6.2
building [mo]: all of 0 po files
building [html]: all source files
updating environment: 3 added, 0 changed, 0 removed
reading sources... [ 66%] js_api                                                
Exception occurred:
  File "python3.4/site-packages/recommonmark/states.py", line 134, in run_role
    content=content)
TypeError: 'NoneType' object is not callable

Jinja2 (2.9.6) CommonMark (0.5.4) recommonmark (0.4.0) Sphinx (1.6.2) sphinx-rtd-theme (0.2.4) sphinxcontrib-websupport (1.0.1) docutils (0.13.1)

Python 3.4.2

[jfbu]

sorry for my bad counsel and inconvenience.

@ratiotile apart from Python 3.5.3 I have the same versions, but on Mac OS. Tested with Sphinx 1.6.2 and 1.6.1.

# Sphinx version: 1.6.2
# Python version: 3.5.3 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6
# Recommonmark version: 0.4.0
# Sphinx RTD theme version: 0.2.4

Project including the markdownsyntax.md file has no problem for me.

edit actually my sphinxcontrib-websupport==1.0.0

[ratiotile]

@jfbu did you test my project?

The issue seems to be in processing links to another markdown file. Links of the form [x](filename) result in an html link to nonexistant page, while the form [x](filename.md) used to generate the correct link under Sphinx 1.5, but now fails to build.

[jfbu]

@ratiotile Yes I reproduce

Exception occurred:
  File "/opt/miniconda3/lib/python3.5/site-packages/recommonmark/states.py", line 134, in run_role
    content=content)
TypeError: 'NoneType' object is not callable

with your project.

[jfbu]

From #3372. Sorry for inconvenience. Please wait for @tk0miya to look, thanks.

(I don't know if the problem is a breaking change at #3372, or if recommonmark used non-API part of Sphinx...; adding the bug tag nevertheless)

(it seems recommonmark should use sphinx api as 'doc' role is now only in std domain)

[jfbu]

@ratiotile regarding your simple project I can get it working by hacking recommonmark in this horrible way:

diff --git a/transform.py b/transform.py
index 48a93be..15c04b1 100644
--- a/transform.py
+++ b/transform.py
@@ -168,11 +168,13 @@ class AutoStructify(transforms.Transform):
         if title is None:
             return None
         if docpath:
-            content = u'%s <%s>' % (title, docpath)
-            self.state_machine.reset(self.document,
-                                     node.parent,
-                                     self.current_level)
-            return self.state_machine.run_role('doc', content=content)
+            node['refuri'] = uri[1:]
+            return None
+            # content = u'%s <%s>' % (title, docpath)
+            # self.state_machine.reset(self.document,
+            #                          node.parent,
+            #                          self.current_level)
+            # return self.state_machine.run_role('doc', content=content)
         else:
             # inplace modify uri
             node['refuri'] = uri

This is, needless to say, completely bad thing. And no idea how many things it will break in your real project. Just mentioning in passing.

[tk0miya]

Hmm... it seems recommonmark extension uses very hacky way to make document reference. Surely we modified the implementation of :doc: role internally.

As a quick look, sphinx.util.docutils:sphinx_domains() might be help to do that.

Anyway, I think recommonmark should be updated. It uses very fragile way to realize AutoStructify.

[lorengordon]

@jfbu so, at least part of the problem appears to be that recommonmark hasn't released a version in a bit. The code in that diff isn't present in the master branch. It was removed in the last commit that modified that file.

I adjusted my requirements file to build from the recommonmark master branch and was able to get it to work, sort of. Needed to also modify link references from [file](file.md) to [file](file.html) to get the references to point to a page that will actually load, which is kind of awful.

I think I'll just keep pinning to old versions for a while.

[tk0miya]

It seems the linking feature has been dropped from recommonmark. So this might be resolved.

[scottleibrand]

Could anyone clarify what the path forward is for projects experiencing the sphinx 1.6 and recommonmark 0.4.0 incompatibility described in this issue? I do see that this issue was closed, but I'm not sure what @tk0miya meant by "the linking feature has been dropped from recommonmark", and https://github.com/rtfd/recommonmark/issues/73 doesn't really provide any clues. For now, I've done the same as https://github.com/ccpgames/eveonline-third-party-documentation/pull/266 and pinned sphinx==1.5.6 in our requirements.txt, but that doesn't seem like a proper solution. Thanks in advance for any assistance or direction.

[lorengordon]

@scottleibrand Have you tried m2r? We've been able to remove recommonmark entirely, replaced with m2r. Recommonmark issues don't seem to get much attention or support, so probably worth moving on.

[tk0miya]

I'm not sure what @tk0miya meant by "the linking feature has been dropped from recommonmark",

I just read this commit. It says the feature is deprecated. https://github.com/rtfd/recommonmark/commit/8baf238562c2f7c9bea603e352c3a94f51905a3e

I don't know good way, but it seems recommonmark will drops the doc reference in nearly future. So you should find another tool.

[Sieboldianus]

Just saw this and decided to move away from sphinx entirely. mkdocs allows natively parsing Markdown files and also localhost preview with mkdocs serve. There's also a readthedocs-theme!