[BUG] No error information from parsing python source

[jmrgibson]

Describe the bug When parsing a module ::: foo (name changed) using pytkdocs as a plugin for mkdocstrings, I get the following quite-unhelpful error message:

## (other mkdocs log messages above. No errors or warnings, just INFO)
ERROR   -  mkdocstrings.extension: Error while loading JSON:

Traceback (most recent call last):
  File "/home/builduser/.shiv/mkdocs_07b61aea34d5669e5dea996270d4897d6e6a74e36d67697e7a1c82e3bdfd2a77/site-packages/mkdocstrings/handlers/python.py", line 206, in collect
    result = json.loads(stdout)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

ERROR   -  mkdocstrings.extension: Error reading page 'foo.html':
Could not collect 'foo'

This module has a C extension, so the entire source code is not available. Would this be enough to crash the process?

To Reproduce mkdocs config

  - mkdocstrings:
      handlers:
        python:
          rendering:
            show_root_heading: false
            show_signature_annotations: true
            show_if_no_docstring: false
            show_object_full_path: false
            show_root_full_path: true

System (please complete the following information):

• pytkdocs version 0.11.1
• mkdocstrings version 0.15.0
• Python version: 3.8.8
• OS: Linux

Context The module arrangement is like this

foo/
     __init__.py
     bar.so   
    config.py

__init__.py imports functions from both bar.so and config.py

The C extension module can be imported and run without issue (it's part of a test suite that works fine). It's only the documentation step that causes issues.

Diagnostics

Things I've tried so far:

> python3.8
import foo   # works fine

Thanks!

[jmrgibson]

It would appear that building docs works fine from a normal virtual environment, but only when running from a shiv does this issue appear. Does pytkdocs make any assumptions about its operating environment that could cause this?

Specifically, the command I'm running is

export SHIV_ENTRY_POINT=mkdocs.__main__.cli && export PYTHONPATH=/path/to/foo && python3.8 /path/to/shiv/mkdocs.pyz build --config-file /path/to/mkdocs.yml

[pawamoy]

I have zero experience with shiv, could you set up a repo that we could use to reproduce and investigate?

[jmrgibson]

I'm trying to set up an environment to debug/reproduce this, but I'm struggling with poetry.

At the moment I have a brand new project, with the following pyproject.toml

[tool.poetry.dependencies]
python = "^3.8"
mkdocs = "^1.1.2"
mkdocs-material = "^7.1.3"
mkdocstrings = { path = "../mkdocstrings", develop=true }

mkdocstrings is checked out locally, so I can specify pytkdocs as a development dependency in pyproject.toml (also using path=). That dependency change is the only modification, so that I can use the edits from my other PR.

I then run

poetry update
poetry install
poetry run mkdocs build

which gives the following error

Traceback (most recent call last):
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3016, in _dep_map
    return self.__dep_map
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2813, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/mkdocs/__main__.py", line 152, in build_command
    build.build(config.load_config(**kwargs), dirty=not clean)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/mkdocs/config/base.py", line 197, in load_config
    errors, warnings = cfg.validate()
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/mkdocs/config/base.py", line 107, in validate
    run_failed, run_warnings = self._validate()
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/mkdocs/config/base.py", line 62, in _validate
    self[key] = config_option.validate(value)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/mkdocs/config/config_options.py", line 131, in validate
    return self.run_validation(value)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/mkdocs/config/config_options.py", line 609, in run_validation
    plgins[item] = self.load_plugin(item, cfg)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/mkdocs/config/config_options.py", line 617, in load_plugin
    Plugin = self.installed_plugins[name].load()
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in load
    self.require(*args, **kwargs)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2471, in require
    reqs = self.dist.requires(self.extras)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2734, in requires
    dm = self._dep_map
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3018, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3028, in _compute_dependencies
    reqs.extend(parse_requirements(req))
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3084, in parse_requirements
    yield Requirement(line)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3094, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/Users/jgibson/Library/Caches/pypoetry/virtualenvs/pytkdocs-shiv-example-Z64v_VE5-py3.8/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 115, in __init__
    raise InvalidRequirement("Invalid URL: {0}".format(req.url))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid URL: ../pytkdocs

[pawamoy]

I'm not sure to understand what you're trying to achieve. Why do you checkout mkdocstrings and pytkdocs locally? You can specify dependencies as git repositories, even targeting a specific branch.

[pawamoy]

Closing for lack of information, feel free to further comment :slightly_smiling_face: