--dump throws exception when any class has any doc comments

[zeelot]

Environment

• Pydoc-Markdown Version: v4.3.2
• Python Version: v3.9.2
• Operating System: Ubuntu 20.04

Describe the bug Just got started with this tool and trying to generate docs for a small project. I was following the guide and running pydoc-markdown --dump to test my configuration but, if any class has documentation, this command throws an exception when it tries to load that file.

pydoc-markdown --dump   
{"name": "__init__", "location": {"filename": "/home/zeelot/projects/github/owl-games/seagulls-py/src/__init__.py", "lineno": 2}, "docstring": null, "members": []}
Traceback (most recent call last):
  File "/home/zeelot/.local/bin/pydoc-markdown", line 8, in <module>
    sys.exit(cli())
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/pydoc_markdown/main.py", line 339, in cli
    dump_module(module, sys.stdout)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/docspec/__init__.py", line 531, in dump_module
    data = databind.json.dump(module, Module)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/__init__.py", line 117, in dump
    data = (mapper or new_mapper()).serialize(value, type_ or type(value), annotations=annotations, settings=options)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/objectmapper.py", line 102, in serialize
    return self.convert(Direction.serialize, value, type_hint, filename, pos, key, annotations, settings)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/objectmapper.py", line 80, in convert
    return ctx.convert()
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/converter.py", line 116, in convert
    return self.converters.get_converter(self.location.type, self.direction).convert(self)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 102, in convert
    return self._serialize(ctx, ctx.type)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 35, in _serialize
    result[alias] = ctx.push(flat_field.field.type, value, name, flat_field.field).convert()
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/converter.py", line 116, in convert
    return self.converters.get_converter(self.location.type, self.direction).convert(self)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/optional.py", line 12, in convert
    return ctx.push(ctx.type.type, ctx.value, None, ctx.field).convert()
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/converter.py", line 116, in convert
    return self.converters.get_converter(self.location.type, self.direction).convert(self)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 102, in convert
    return self._serialize(ctx, ctx.type)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 17, in _serialize
    raise ctx.type_error(expected=type_.schema.python_type)
databind.core.mapper.converter.ConversionError: [None] ($ ObjectType(docspec.Module)) -> [None] (.docstring OptionalType(ObjectType(docspec.Docstring))) -> [None] ($ ObjectType(docspec.Docstring)): expected `Docstring` to serialize `ObjectType(docspec.Docstring)`, got `str`

Here's an example of a class with a description:

class GameObject(ABC):
    """
    Interface for anything representing an object in the scene.
    """

    @abstractmethod
    def tick(self) -> None:
        pass

    @abstractmethod
    def render(self, surface: Surface) -> None:
        pass

I don't get any exceptions if my class has no comments:

class GameObject(ABC):

    @abstractmethod
    def tick(self) -> None:
        pass

    @abstractmethod
    def render(self, surface: Surface) -> None:
        pass

Can you help me figure out what is going on?

Expected behavior

No exception is thrown.

[zeelot]

If it helps, I seem to get this error when trying to run this command on a fresh checkout of your repo from inside the examples/mkdocs directory:

pydoc-markdown --dump            
Traceback (most recent call last):
  File "/home/zeelot/.local/bin/pydoc-markdown", line 8, in <module>
    sys.exit(cli())
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/pydoc_markdown/main.py", line 339, in cli
    dump_module(module, sys.stdout)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/docspec/__init__.py", line 531, in dump_module
    data = databind.json.dump(module, Module)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/__init__.py", line 117, in dump
    data = (mapper or new_mapper()).serialize(value, type_ or type(value), annotations=annotations, settings=options)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/objectmapper.py", line 102, in serialize
    return self.convert(Direction.serialize, value, type_hint, filename, pos, key, annotations, settings)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/objectmapper.py", line 80, in convert
    return ctx.convert()
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/converter.py", line 116, in convert
    return self.converters.get_converter(self.location.type, self.direction).convert(self)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 102, in convert
    return self._serialize(ctx, ctx.type)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 35, in _serialize
    result[alias] = ctx.push(flat_field.field.type, value, name, flat_field.field).convert()
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/converter.py", line 116, in convert
    return self.converters.get_converter(self.location.type, self.direction).convert(self)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/optional.py", line 12, in convert
    return ctx.push(ctx.type.type, ctx.value, None, ctx.field).convert()
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/core/mapper/converter.py", line 116, in convert
    return self.converters.get_converter(self.location.type, self.direction).convert(self)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 102, in convert
    return self._serialize(ctx, ctx.type)
  File "/home/zeelot/.local/pipx/venvs/pydoc-markdown/lib/python3.9/site-packages/databind/json/modules/object.py", line 17, in _serialize
    raise ctx.type_error(expected=type_.schema.python_type)
databind.core.mapper.converter.ConversionError: [None] ($ ObjectType(docspec.Module)) -> [None] (.docstring OptionalType(ObjectType(docspec.Docstring))) -> [None] ($ ObjectType(docspec.Docstring)): expected `Docstring` to serialize `ObjectType(docspec.Docstring)`, got `str`

[NiklasRosenstein]

Hey @Zeelot ,

Sorry for the long silence. Thanks for reporting this! Seems the change in docspec to replace the type of docstring fields with the Docstring type wasn't as backwards compatible as I thought it would be.

The best solution would be for Pydoc-Markdown to stop treating .docstring fields are just strings and actually understand that they are now instances of docspec.Docstring. However, the easiest solution would probably be to make it so that the serialization of docspec.Dostring accepts a string and coerces it into the right type before serializing (to actually fix the expected backwards compatibility of the Docstring class).

I'll have a closer look into this soon

[NiklasRosenstein]

This will be fixed with #245 because it will upgrade to do spec 2.x, which removed the attempt at "backwards compatibility" that causes this issue.