sphinx.ext.viewcode could not work as expected on parallel build

[cheekyshibe]

Describe the bug I'm using sphinx to build the document for the project MegEngine.

In Docs repo, most APIs are organized manually for some reasons:

.. autosummary::
   :toctree: api
   :nosignatures:

   cos
   sin
   tan
   acos
   asin
   atan
   atan2
   cosh
   sinh
   tanh
   acosh
   asinh
   atanh

As you can see, several functions are implemented here:

Everything works for cosh():

But it doesn't work for some other APIs like asin() acos() ...

Seems viewcode-back class is not generated for every API.

To Reproduce

I'm sure that sphinx.ext.viewcode not works as expected. You can do it with these steps to reproduce the behavior with a GitHub workflow file: https://github.com/MegEngine/Tutorials/blob/main/.github/workflows/sphinx-build.yml

Expected behavior copied Class/Function source codes HTML pages can be highlighted and add links to documents correctly.

Your project MegEngine / Docs

Environment info

• OS: Ubuntu 18.04
• Python version: 3.6/3.7
• Sphinx version: sphinx-build 3.5.1
• Sphinx extensions: sphinx.ext.viewcode
• Extra tools: -

[tk0miya]

Now I checked your public document on https://megengine.org.cn/api/latest/_modules/megengine/functional/elemwise.html#cosh, but it seems working fine. Does this trouble happen in your local environment?

[cheekyshibe]

@tk0miya Thanks for your reply. Now our doc website is an old version(all APIs are generated by sphinx.ext.autodoc automodule) and it will be replaced soon. If you want to reproduce the problem locally, here is the source code: https://github.com/MegEngine/Tutorials/tree/docs-transfer

And I can provide a Tutorial to build and deploy the document locally. But it was written in Chinese so I think you can use Google translation or look at this file:

jobs:
  Build:
    name: Build html
    runs-on: ubuntu-latest
    steps:
      - name: Checkout document repp
        uses: actions/checkout@v2

      - name: Checkout theme repo
        uses: actions/checkout@v2
        with: 
          repository: MegEngine/pydata-sphinx-theme
          ref: dev
          fetch-depth: 1
          path: theme

      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.6

      - name: Cache Python packages
        id: cache-python
        uses: actions/cache@v2
        with:
          path: ${{ env.pythonLocation }}
          key: ${{ env.pythonLocation }}$-${{ hashFiles('requirements.txt') }}

      - name: Install MegEngine, Sphinx and other Python dependencies
        if: steps.cache-python.outputs.cache-hit != 'true'
        run: |
          python3 -m pip install --upgrade pip
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
          pip install megengine
      - name: Install Pydata Sphinx theme
        run: python3 -m pip install ./theme

      - name: Install pandoc and other dependencies
        run: sudo apt-get install -y pandoc

      - name: Generate HTML files
        run: make html -W --keep-going

It's in the testing status for MegEngine 1.3 (not release now) so there will be lots of API missing warning if you build it with MegEngine v1.2.

[tk0miya]

I found the reason. It's a bug of the viewcode extension on parallel build mode. #8935 will resolve the error.

[cheekyshibe]

So efficient. 👍

[astrojuanlu]

@MegChai Hi! I see that you closed MegEngine/Documentation#49 but, from what I understand, MegEngine has Sphinx 3.5.4 pinned. We are investigating whether there is still a bug with sphinx.ext.viewcode accidentally introduced by #8935 (see #9210). Could you please try to build MegEngine docs with Sphinx 4.0 and report back the results? Thanks in advance!

[cheekyshibe]

@astrojuanlu Cool. sphinx.ext.viewcode works as expected now. Thanks for your work.

I fix the Sphinx version because of other unadapted extensions and have a workaround later. And I will start using Sphinx 4.0 in the next version of MegEngine documentation (see WIP PR).