Unable to include readme.md while using virtualenv and setup.py

I have a package which I want to publish on PyPI so I have created a setup.py file. I am developing it within a virtualenv and currently I am working on the documentation. I have troubles using the pdoc packagename --http localhost:8080 server. It crashes when I try to include a readme.md file in the docstring of my module. I have been debugging it for a few hours already but with no luck. I have created a minimal example to recreate the bug.

Steps to Reproduce

Create a dummy project with this structure:

packagename
└─ .venv
└─ readme.md
└─ setup.py
└─ packagename
└─ __init__.py

__init__.py contains this code:

"""
.. include:: ../readme.md
"""

setup.py contains this code:

from setuptools import setup

setup(
    name='packagename',
    packages=['packagename'],
    install_requires=['pdoc3>=0.9.2'],
)

readme.md contains this code:

# Readme from the root
This is some verbose documentation.

Now run the following commands to start the pdoc server:

virtualenv .venv
source .venv/bin/activate
pip install . 
pdoc packagename --http localhost:8080

In the browser on the address localhost:8080 there will be a correct html output in the default template:

Python module list
packagename    Include: ../readme.md
Generated by pdoc 0.9.2.

Expected Behavior

When I click on the packagename I expect to see a new page with the documentation of said package.

Actual Behavior

I get this error:

Error importing module packagename:

Traceback (most recent call last):
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/html_helpers.py", line 277, in _admonition
    return _ToMarkdown._include_file(indent, value,
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/html_helpers.py", line 336, in _include_file
    with open(os.path.normpath(os.path.join(os.path.dirname(module.obj.__file__), path)),
FileNotFoundError: [Errno 2] No such file or directory: '/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/readme.md'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/cli.py", line 234, in do_GET
    out = self.html()
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/cli.py", line 263, in html
    return pdoc.html(self.import_path_from_req_url,
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/__init__.py", line 176, in html
    mod = Module(import_module(module_name, reload=reload),
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/__init__.py", line 611, in __init__
    super().__init__(module.__name__, self, module)
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/__init__.py", line 481, in __init__
    docstring = _ToMarkdown.admonitions(docstring, self.module, ('include',))
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/html_helpers.py", line 327, in admonitions
    return substitute(substitute(text))
  File "/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/pdoc/html_helpers.py", line 280, in _admonition
    raise RuntimeError('`.. include:: {}` error in module {!r}: {}'
RuntimeError: `.. include:: ../readme.md` error in module 'packagename': [Errno 2] No such file or directory: '/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/readme.md'

How to prevent the bug from occuring

If I do not try to include the readme.md in the __init__.py docstring, all works well.
If I comment the line packages=['packagename'], in the setup.py file, all works well even if I try to include the readme.md.

I suspected some problem with the virtualenv, because the error says it is trying to look for the readme.md file in a wrong location. But I have not clue why it is trying to look there and also I do not understand why commenting that particular line in setup.py helps.

Additional info

Inbetween the changes in code I always deactivate and remove the .venv and packagename/__pycache__/ and I create the .venv and install everything again. I have updated the pip and virtualenv to the newest versions. I really do not see where this can be coming from.

Versions:

Mako        1.1.4
Markdown    3.3.4
MarkupSafe  1.1.1
packagename 0.0.0
pdoc3       0.9.2
pip         21.0.1
setuptools  54.1.2
wheel       0.36.2

OS: Ubuntu 20.04.2 Python version: Python 3.8.5 Virtualenv: virtualenv 20.4.3 from /home/myuser/.local/lib/python3.8/site-packages/virtualenv/__init__.py

Thanks for the comprehensive issue report.

I think it boils down to:

pip install . 
pdoc packagename

with the latter command preferring the installed packagename: https://github.com/pdoc3/pdoc/blob/c2c13200a630f97e5a7b2cba292b0dc9deb58b53/pdoc/__init__.py#L221-L224

I guess you can try eiher:

pip uninstall packagename
pdoc packagename  # Build docs from the directory, without having a version installed
# This is what our CI seems to do.

https://github.com/pdoc3/pdoc/blob/c2c13200a630f97e5a7b2cba292b0dc9deb58b53/doc/build.sh#L22-L25 Or

pdoc ./packagename  # Force interpreting package as a relative directory

Or specify readme.md as package data in setup.py to have it installed somewhere along with your package.

Dear @kernc thank you very much for a quick reply. It was eye-opening. Now it is obvious why I experienced such weird behavior. I have checked what you have suggested and these are my observations after running the pdoc3 server again:

Uninstalling the packagename after pip install . helps as you suggested.
Unfortunately, forcing a relative directory with ./packagename does not help. Although, I would intuitively expect this to work, is this a bug?

Anyways, after some research, I have came up with two variations of yet another approach:

pip install -e . will install the package in editable mode and the --http option works as expected.
Adjusting the setup.py file such that it contains a list of packages necessary for development in extras_require and installing with pip install -e .[dev].
```
from setuptools import setup
```

setup( name='packagename', packages=['packagename'], install_requires=[], extras_require={ 'dev': ['pdoc3>=0.9.2'], } )


I will end up using this variant as I was looking for a solution which would be clear to the future developers and would not bother standard users. In this case, the user can just install `pip install .` and a developer would install `pip install -e .[dev]`

If you have any comments on this I'll be glad to hear it. Thank you.

does not help. Although, I would intuitively expect this to work, is this a bug?

Can't say. I think it should work and grab the local file:

https://github.com/pdoc3/pdoc/blob/c2c13200a630f97e5a7b2cba292b0dc9deb58b53/pdoc/__init__.py#L209-L215

And once it has the correct module, it should resolve the relative include path:

https://github.com/pdoc3/pdoc/blob/c2c13200a630f97e5a7b2cba292b0dc9deb58b53/pdoc/html_helpers.py#L339-L341

On the other hand:

RuntimeError: `.. include:: ../readme.md` error in module 'packagename': 
  [Errno 2] No such file or directory: '/home/myuser/projects/packagename/.venv/lib/python3.8/site-packages/readme.md'

../readme.md is outside your installed package, so obviously it can't work. You can't just have placed files somewhere upwards of site-packages/package as you please. It might work if you move the readme into packagename, like we do.

If you don't want to move the readme as it's also used as front on GitHub, and you don't want to maintain two separate readmes, I think you can include/move the file into proper location by setup.py package_data=.

Thank you for your help.

pdoc3 / pdoc