Add capability to automatically test code in documentation

[willzeng]

@nathanshammah I recall that you were looking into something for this? Did you have suggestions in mind?

[andreamari]

For the main documentation, I don't know.

However if we want a list of tutorials/examples/implementations which are automatically executed this is great: https://sphinx-gallery.github.io/stable/index.html For example this is a typical result: https://pytorch.org/tutorials/

The source of each example is just python script with comments. The rest is automatically built by sphinx-gallery.

[nathanshammah]

@willzeng In the comment to #72 I meant that I think it is already done, as we have doctest as extension in the conf.py file. But building it and playing with the python block, I do not see a clear error message. So I will investigate this further.

@andreamari I really like sphinx gallery. We are currently adding it to QuTiP. I did not include it here as extension as it was giving some error in the conf.py but I think it would be amazing to include it, will look into it and ping you if I get stuck.

[nathanshammah]

Ok, so there are basically three tools that help with this: mainly doctest (python core) and then pytest, which that does testing in general, and sphinx, which builds the docs. It took some time to disentangle this, and I am not completely sure of the findings, but at least happy of what's somewhat clearer now:

• For the code base, one can use doctest to test docstrings in modules. This can be called in various ways. Best would be to set up an automated script in a test file. pytest+doctest is also possible. In any case it would be nice to implement this inside CI workflow in the .github file.
• For the documentation: Running from bash make doctest just like make html and make latexpdf, runs the tests of both modules (as it includes them in the autodoc) and of the guide, and gives a report. This is thanks to the sphinx-doctest extension inserted in the conf.py file.

Currently, I added a dummy example in factories.py on a private branch, to check what was happening, below is copied the quite long error message. Basically the modules need to be implicitly imported for doctest to properly run the script.

(mitdev) Nathans-MacBook-Pro:docs nathanshammah$ make doctest
Running Sphinx v2.4.4
loading translations [Python]... not available for built-in messages
making output directory... done
WARNING: html_static_path entry '_static' does not exist
building [mo]: targets for 0 po files that are out of date
building [doctest]: targets for 8 source files that are out of date
updating environment: [new config] 8 added, 0 changed, 0 removed
WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1057e6220>: Failed to establish a new connection: [Errno 61] Connection refused')': /
WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105815a00>: Failed to establish a new connection: [Errno 61] Connection refused')': /
WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105828cd0>: Failed to establish a new connection: [Errno 61] Connection refused')': /
WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105881df0>: Failed to establish a new connection: [Errno 61] Connection refused')': /
WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105881fa0>: Failed to establish a new connection: [Errno 61] Connection refused')': /
WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1058891c0>: Failed to establish a new connection: [Errno 61] Connection refused')': /
/miniconda3/envs/mitdev/lib/python3.8/site-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
  warn("Container node skipped: type={0}".format(mdnode.t))

/Users/nathanshammah/github/mitiq/mitiq/zne.py:docstring of mitiq.zne.mitigate_executor:4: WARNING: Unexpected indentation.
WARNING: autodoc: failed to import module 'pyquil_utils' from module 'mitiq.pyquil'; the following exception was raised:
Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
    conn = connection.create_connection(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1230, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1276, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1225, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1004, in _send_output
    self.send(msg)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 944, in send
    self.connect()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x105828670>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105828670>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 492, in connect
    version_dict = self.get_version_info()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 504, in get_version_info
    return cast(str, self.connection._qvm_get_version_info())
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_base_connection.py", line 625, in _qvm_get_version_info
    response = post_json(self.session, self.sync_endpoint, {"type": "version"})
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_base_connection.py", line 63, in post_json
    res = session.post(url, json=json)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_base_connection.py", line 469, in request
    response = super().request(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105828670>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/sphinx/ext/autodoc/importer.py", line 32, in import_module
    return importlib.import_module(modname)
  File "/miniconda3/envs/mitdev/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/nathanshammah/github/mitiq/mitiq/pyquil/pyquil_utils.py", line 11, in <module>
    QVM = get_qc("1q-qvm")
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 868, in get_qc
    return _get_unrestricted_qvm(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 736, in _get_unrestricted_qvm
    return _get_qvm_with_topology(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 680, in _get_qvm_with_topology
    return _get_qvm_qc(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 636, in _get_qvm_qc
    qam=_get_qvm_or_pyqvm(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 598, in _get_qvm_or_pyqvm
    return QVM(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 488, in __init__
    self.connect()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 495, in connect
    raise QVMNotRunning(f"No QVM server running at {self.connection.sync_endpoint}")
pyquil.api._qvm.QVMNotRunning: No QVM server running at http://127.0.0.1:5000

WARNING: autodoc: failed to import module 'test_zne' from module 'mitiq.pyquil.tests'; the following exception was raised:
Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connection.py", line 156, in _new_conn
    conn = connection.create_connection(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1230, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1276, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1225, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 1004, in _send_output
    self.send(msg)
  File "/miniconda3/envs/mitdev/lib/python3.8/http/client.py", line 944, in send
    self.connect()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connection.py", line 168, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x105889370>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 747, in urlopen
    return self.urlopen(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105889370>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 492, in connect
    version_dict = self.get_version_info()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 504, in get_version_info
    return cast(str, self.connection._qvm_get_version_info())
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_base_connection.py", line 625, in _qvm_get_version_info
    response = post_json(self.session, self.sync_endpoint, {"type": "version"})
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_base_connection.py", line 63, in post_json
    res = session.post(url, json=json)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_base_connection.py", line 469, in request
    response = super().request(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x105889370>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/sphinx/ext/autodoc/importer.py", line 32, in import_module
    return importlib.import_module(modname)
  File "/miniconda3/envs/mitdev/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/nathanshammah/github/mitiq/mitiq/pyquil/tests/test_zne.py", line 12, in <module>
    from mitiq.pyquil.pyquil_utils import (
  File "/Users/nathanshammah/github/mitiq/mitiq/pyquil/pyquil_utils.py", line 11, in <module>
    QVM = get_qc("1q-qvm")
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 868, in get_qc
    return _get_unrestricted_qvm(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 736, in _get_unrestricted_qvm
    return _get_qvm_with_topology(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 680, in _get_qvm_with_topology
    return _get_qvm_qc(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 636, in _get_qvm_qc
    qam=_get_qvm_or_pyqvm(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_quantum_computer.py", line 598, in _get_qvm_or_pyqvm
    return QVM(
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_error_reporting.py", line 251, in wrapper
    val = func(*args, **kwargs)
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 488, in __init__
    self.connect()
  File "/miniconda3/envs/mitdev/lib/python3.8/site-packages/pyquil/api/_qvm.py", line 495, in connect
    raise QVMNotRunning(f"No QVM server running at {self.connection.sync_endpoint}")
pyquil.api._qvm.QVMNotRunning: No QVM server running at http://127.0.0.1:5000

WARNING: autodoc: failed to import module 'tests.test_zne' from module 'mitiq.qiskit.tests'; the following exception was raised:
No module named 'mitiq.qiskit.tests.tests'
/Users/nathanshammah/github/mitiq/docs/source/guide/guide-getting-started.rst:5: WARNING: Duplicate explicit target name: "here".
/Users/nathanshammah/github/mitiq/docs/source/guide/guide-getting-started.rst:118: WARNING: Footnote [1] is not referenced.
/Users/nathanshammah/github/mitiq/docs/source/guide/guide-overview.rst:28: WARNING: Footnote [1] is not referenced.
/Users/nathanshammah/github/mitiq/docs/source/guide/guide-overview.rst:29: WARNING: Footnote [2] is not referenced.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
running tests...

Document: guide/guide-factories
-------------------------------
**********************************************************************
File "guide/guide-factories.rst", line 95, in default
Failed example:
    print(f"The zero-noise extrapolation is: {zn_limit:.3}")
Exception raised:
    Traceback (most recent call last):
      File "/miniconda3/envs/mitdev/lib/python3.8/doctest.py", line 1329, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest default[0]>", line 1, in <module>
        print(f"The zero-noise extrapolation is: {zn_limit:.3}")
    NameError: name 'zn_limit' is not defined
**********************************************************************
1 items had failures:
   1 of   1 in default
1 tests in 1 items.
0 passed and 1 failed.
***Test Failed*** 1 failures.

Document: apidoc
----------------
**********************************************************************
File "../../mitiq/factories.py", line ?, in default
Failed example:
    fact = Factory()
Exception raised:
    Traceback (most recent call last):
      File "/miniconda3/envs/mitdev/lib/python3.8/doctest.py", line 1329, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest default[0]>", line 1, in <module>
        fact = Factory()
    NameError: name 'Factory' is not defined
**********************************************************************
1 items had failures:
   1 of   1 in default
1 tests in 1 items.
0 passed and 1 failed.
***Test Failed*** 1 failures.

Doctest summary
===============
    2 tests
    2 failures in tests
    0 failures in setup code
    0 failures in cleanup code
build finished with problems, 9 warnings.
make: *** [doctest] Error 1

This is just an update, while I keep working on the solutions.

[nathanshammah]

Closed by #105.