Unable to keep build from attempting to compile a mocked module

zaneselvans commented 4 years ago

Details

I am attempting to build documentation for a project that depends on packages that include code written in C. For the packages beyond those available via the Read the Docs system_packages setting (numpy, scipy, pandas, and matplotlib) which fail to compile, I am attempting to mock the modules/packages for import, using autodoc_mock_imports in my conf.py. However, there must be something I don't understand about the process because it keeps trying to compile these modules anyway.

The module that is giving me issues right now is the snappy compression library, which requires the libsnappy-dev (Debian) package to be installed. The python package which provides it is called python-snappy (My assumption was that it's the module name and not the package name that needs to be listed in autodoc_mock_imports -- but I've tried both to no effect.

Like the system_packages I have commented python-snappy out in the requirements.txt file associated with the doc build (which is the output from a pip freeze). The build log reports that the requirement is being found in my project, which is named pudl:

Collecting python-snappy (from pudl==0.1.dev398+g5e075d4)

So I assume the build is trying to compile it because it's listed in my setup.py? But I need to leave it there for the real packaging and installation elsewhere.

I have wiped the doc build for this branch (python-packaging) and re-tried, with the same result.

Read the Docs project URL: https://catalyst-cooperative-pudl.readthedocs.io
Build URL (if applicable): https://readthedocs.org/projects/catalyst-cooperative-pudl/builds/9376117/
Read the Docs username (if applicable): zaneselvans

Expected Result

I thought that by adding snappy to the list of modules to be mocked, the build would consider the dependency satisfied and not attempt to compile the package.

Actual Result

The build fails because of a missing C language header file:

Building wheels for collected packages: pudl, python-snappy
  Building wheel for pudl (setup.py): started
  Building wheel for pudl (setup.py): finished with status 'done'
  Stored in directory: /home/docs/checkouts/readthedocs.org/user_builds/catalyst-cooperative-pudl/.cache/pip/wheels/a4/5f/b6/1f8213aeb5876af1c140b54dce3466b845d989ca1101da875a
  Building wheel for python-snappy (setup.py): started
  Building wheel for python-snappy (setup.py): finished with status 'error'
  ERROR: Complete output from command /home/docs/checkouts/readthedocs.org/user_builds/catalyst-cooperative-pudl/envs/python-packaging/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-z6c1cop3/python-snappy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-r54uwk2d --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/snappy
  copying snappy/__init__.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/__main__.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/hadoop_snappy.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy_formats.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy.py -> build/lib.linux-x86_64-3.7/snappy
  copying snappy/snappy_cffi.py -> build/lib.linux-x86_64-3.7/snappy
  running build_ext
  building 'snappy._snappy' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/snappy
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/docs/.pyenv/versions/3.7.3/include/python3.7m -c snappy/snappymodule.cc -o build/temp.linux-x86_64-3.7/snappy/snappymodule.o
  snappy/snappymodule.cc:31:10: fatal error: snappy-c.h: No such file or directory
   #include <snappy-c.h>
            ^~~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for python-snappy
  Running setup.py clean for python-snappy
Successfully built pudl
Failed to build python-snappy

stsewd commented 4 years ago

Yes, that requirement is being installed becuase of https://github.com/catalyst-cooperative/pudl/blob/5e075d40d76fcf70d763b3661d91c864026f8723/.readthedocs.yml#L22

You can use the READTHEDOCS environment variable on your setup.py file

https://docs.readthedocs.io/en/stable/builds.html#build-environment

zaneselvans commented 4 years ago

Hmm. I added that pip install because otherwise the pudl package (which is what I'm trying to document!) wasn't getting installed and everything was failing. It seemed odd that additional action was required to get get my own package to install. Is there a different/better way to get the package you're trying to document installed? Some default I've totally missed that wouldn't create this complication?

Are you suggesting generating different lists of install_requires packages in setup.py depending on the build environment? Like if READTHEDOCS then don't require python-snappy?

stsewd commented 4 years ago

Are you suggesting generating different lists of install_requires packages in setup.py depending on the build environment? Like if READTHEDOCS then don't require python-snappy?

Yes

Is there a different/better way to get the package you're trying to document installed?

The way you are installing the project is correct.

Another way is modifying the sys.path in your conf.py file http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html

zaneselvans commented 4 years ago

Thanks for the explanation! I realized after you explained this why I had moved to installing our package rather than just adding it to sys.path -- I'm using setuptools_scm to automatically generate the package version, and to access it using pkg_resources.get_distribution() I need the package to actually be installed, not just importable. Conditionally including python-snappy in the install_requires based on the READTHEDOCS environment variable worked, so this is solved now.

I also posted the question (and more details on the answer) over on Stack Overflow in case anyone else has this issue.

readthedocs / readthedocs.org