YapfError: invalid syntax

casperdcl commented 2 years ago

Environment

Pydoc-Markdown Version: v4.6.3
Python Version: v3.10.2
Operating System: Ubuntu 20.04.03 LTS (GitHub Actions)

Describe the bug

working (pydoc-markdown-4.5.1 databind.core-1.5.1 mkdocs-material-8.2.3 pymdown-extensions-9.2)
broken (pydoc-markdown-4.6.3 databind.core-1.5.2 mkdocs-material-8.2.6 pymdown-extensions-9.3)
still broken (as above but replacing node.docstring => node.docstring.content)

There's an odd config warning in all runs.

 [WARNING - pydoc_markdown.main]: Unknown configuration options:
  - $ (ObjectType(pydoc_markdown.PydocMarkdown)) [pydoc-markdown.yml]
  - .renderer (UnionType(pydoc_markdown.interfaces.Renderer))
  - $ (ObjectType(pydoc_markdown.contrib.renderers.mkdocs.MkdocsRenderer))
  - .pages (ListType(ObjectType(pydoc_markdown.util.pages.Page), python_type=Pages))
  - .7 (ObjectType(pydoc_markdown.util.pages.Page))
  - .children (ListType(ObjectType(pydoc_markdown.util.pages.Page)))
  - .0 (ObjectType(pydoc_markdown.util.pages.Page))
  - .directory (UnknownType())

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 119, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=16, value='*', context=(' ', (2, 27))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 125, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=16, value='*', context=(' ', (2, 27))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 183, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 131, in ParseCodeToTree
    raise e
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 129, in ParseCodeToTree
    ast.parse(code)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 2
    async def gather(cls, *fs, *, loop=None, timeout=None, total=None, **tqdm_kwargs): pass
                               ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/bin/pydoc-markdown", line 8, in <module>
    sys.exit(cli())
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/main.py", line 344, in cli
    session.render(pydocmd)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/main.py", line 138, in render
    config.render(modules)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/__init__.py", line 179, in render
    self.renderer.render(modules)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/mkdocs.py", line 168, in render
    item.page.render(filename, modules, self.markdown, self._context.directory)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/util/pages.py", line 179, in render
    renderer.render_single_page(fp, self.filtered_modules(modules), self.title)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 443, in render_single_page
    self._render_recursive(fp, 1, m)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 368, in _render_recursive
    self._render_recursive(fp, level, member)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 368, in _render_recursive
    self._render_recursive(fp, level, member)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 365, in _render_recursive
    self._render_object(fp, level, obj)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 350, in _render_object
    self._render_signature_block(fp, obj)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 329, in _render_signature_block
    code = self._format_function_signature(obj, add_method_bar=self.signature_with_vertical_bar)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 299, in _format_function_signature
    result = self._yapf_code(result + ': pass').rpartition(':')[0].strip()
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/pydoc_markdown/contrib/renderers/markdown.py", line 281, in _yapf_code
    return FormatCode(code, style_config=style)[0]
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 186, in FormatCode
    raise errors.YapfError(errors.FormatErrorMsg(e))
yapf.yapflib.errors.YapfError: <unknown>:2:28: invalid syntax

not sure if related to #246

NiklasRosenstein commented 2 years ago

Hey @casperdcl , it appears you're doing some custom docstring preprocessing. Starting with Pydoc-Markdown 4.6.0, it depends on docspec 2.x, where the ApiObject.docstring is actually a Docstring object rather than a plain string.

While technically this was already the case in later docspec 1.x versions, the Docstring class there was a subclass of str, hoping it would act sufficiently backwards compatible, though actually it caused issues like this: #219

The processor code in Pydoc-Markdown was updated to read and write the documentation string from/to Docstring.content.

Hope this helps 🤞

casperdcl commented 2 years ago

yes updated title & description. Now getting an even weirder error.

NiklasRosenstein commented 2 years ago

Also recently I added back automatic formatting of function signature code blocks using Yapf. The issue appears to be an invalid syntax in the function signature. That may be a bug in docspec_python.format_arglist().

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 183, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 131, in ParseCodeToTree
    raise e
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/site-packages/yapf/yapflib/pytree_utils.py", line 129, in ParseCodeToTree
    ast.parse(code)
  File "/opt/hostedtoolcache/Python/3.10.2/x64/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 2
    async def gather(cls, *fs, *, loop=None, timeout=None, total=None, **tqdm_kwargs): pass
                               ^

NiklasRosenstein commented 2 years ago

I'll have time to look into that later today. In the meantime, you can try setting format_code: false in the MarkdownRenderer.

https://github.com/NiklasRosenstein/pydoc-markdown/blob/b1366696537dd53fc7eb009492d12b0f776826e5/src/pydoc_markdown/contrib/renderers/markdown.py#L206-L207

casperdcl commented 2 years ago

EDIT: actually async def gather(cls, *fs, *, loop=None, ...) is indeed invalid - but I don't know where it comes from. In the codebase it's async def gather(cls, *fs, loop=None, ...)

NiklasRosenstein commented 2 years ago

Hey @casperdcl , if you try again then docspec 2.0.1 should be installed instead of docspec 2.0.0, and in that newer version the arglist formatting should be fixed.

casperdcl commented 2 years ago

Great, confirmed working!

Any ideas about the [WARNING - pydoc_markdown.main]: Unknown configuration options though?

NiklasRosenstein commented 2 years ago

 [WARNING - pydoc_markdown.main]: Unknown configuration options:
  - $ (ObjectType(pydoc_markdown.PydocMarkdown)) [pydoc-markdown.yml]
  - .renderer (UnionType(pydoc_markdown.interfaces.Renderer))
  - $ (ObjectType(pydoc_markdown.contrib.renderers.mkdocs.MkdocsRenderer))
  - .pages (ListType(ObjectType(pydoc_markdown.util.pages.Page), python_type=Pages))
  - .7 (ObjectType(pydoc_markdown.util.pages.Page))
  - .children (ListType(ObjectType(pydoc_markdown.util.pages.Page)))
  - .0 (ObjectType(pydoc_markdown.util.pages.Page))
  - .directory (UnknownType())

Probably refers to this only mention of the "directory" key in your config, and that it is not recognized as a valid key in the configuration.

https://github.com/tqdm/tqdm.github.io/blob/65d79445027266d9c27ba3afe26fd30d03324a01/pydoc-markdown.yml#L106

NiklasRosenstein / pydoc-markdown

YapfError: invalid syntax #255