Allow `pip install --no-deps` to meet the dependencies for Sphinx autodoc

JiaweiZhuang commented 5 years ago

Details

Read the Docs project URL: https://readthedocs.org/projects/xesmf
Build URL (if applicable): https://readthedocs.org/projects/xesmf/builds/8484627/
Read the Docs username (if applicable): zhuangjw

Problem

I am using Sphinx-autodoc to generate API docs for my Python package.

Originally I used Conda to install the full dependencies on readthedocs, but it took too long to build and sometimes even failed (#4695). So, I switched to pure pip (https://github.com/JiaweiZhuang/xESMF/commit/53816d397b3c8f27b195132b27dbb41847e34e0f), which accelerated the build a lot. However, since that switch, Sphinx-autodoc failed to generate API docs. My API page is almost blank:

This is because my package itself has heavy dependencies that cannot be installed by pip. Some C/Fortran extensions must be installed by conda. So, autodoc cannot import my package:

WARNING: autodoc: failed to import module 'xesmf.frontend'; the following exception was raised:
No module named 'xesmf'
WARNING: autodoc: failed to import module 'xesmf.backend'; the following exception was raised:
No module named 'xesmf'
...

Methods I have tried

I am able to generate correct API docs with pip locally, using two hacks:

Force the installation of my package by pip install --no-deps xesmf.
Mock-import the dependencies by adding autodoc_mock_imports = ['numpy', 'xarray', 'scipy', 'ESMF'] in conf.py. Among them, ESMF is an extremely heavy dependency that can only be installed by Conda (and takes a long time to install).

Even with autodoc_mock_imports, I still have to pip-install my package xesmf to provide the docstring for sphinx-autodoc. I have read the Read the Docs Configuration File but cannot find a way to specify the --no-deps option or add custom pip commands.

Right now, I can only think of two highly-unsatisfying solutions to build the API docs correctly online:

Revert to the conda approach to install the full dependencies. This is a huge overkill because building the docs only needs the docstrings in my package, not those C/Fortran extensions.
or, change the requirement in my package's setup.py, so I can pip-install it without the --no-deps option. This is not good for my package itself.

So I am still looking for a better solution. Any suggestions would be appreciated!

Extra information

Here are the full steps to build the correct doc on my laptop:

conda create -n doc python=3.6
conda activate doc
conda install -c conda-forge pandoc
pip install numpydoc ipython nbsphinx sphinx_rtd_theme
pip install --no-deps xesmf
git clone https://github.com/JiaweiZhuang/xesmf
cd xesmf/doc
make html

The correct API page would look like:

stsewd commented 5 years ago

I don't think we are going to add more options for pip, sorry. One solution that comes to my mind is make use of the READTHEDOCS env variable https://docs.readthedocs.io/en/stable/faq.html#how-do-i-change-behavior-for-read-the-docs in your setup.py file.

stsewd commented 5 years ago

And if you need some dependencies, you can just create a custom requirements file only for your docs.

JiaweiZhuang commented 5 years ago

@stsewd Thanks for the suggestion! I tweaked setup.py so that:

on_rtd = os.environ.get('READTHEDOCS') == 'True'
if on_rtd:
    INSTALL_REQUIRES = []
else:
    INSTALL_REQUIRES = ['esmpy', 'xarray', 'numpy', 'scipy']

(https://github.com/JiaweiZhuang/xESMF/commit/5ce3364bfbd3081aad848c5cf4ac54a287ca236e)

Feels a bit hacky but does solve the problem. Thanks again.

readthedocs / readthedocs.org