PyQuil examples not rendering on RTD

crazy4pi314 commented 3 years ago

Pre-Report Checklist

[x] I am running the latest version of mitiq
[x] I checked to make sure that this bug has not already been reported

Issue Description

We have two places in the docs where we execute notebooks that use PyQuil backends, and they are not showing the execution output:

Looking at the RTD logs (below), this is because the evaluation environment on their server cannot start the necessary Docker images needed to act as the server/simulator. They don't support specifying these containers, and when we run the docs build on GH we can (logs).

There are two ways to approach solving this:

See if the newer PyQuil packages come bundled with the servers/compilers bundled natively with the pip packages
(Drastic) Move to hosting our own docs via GH pages

Logs

Running Sphinx v3.5.4
loading translations [en]... done
making output directory... done
checking for /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/refs.bib in bibtex cache... not found
parsing bibtex file /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/refs.bib... parsed 29 entries
myst v0.13.7: MdParserConfig(renderer='sphinx', commonmark_only=False, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, update_mathjax=False, enable_extensions=['amsmath', 'colon_fence', 'deflist', 'dollarmath', 'html_image', 'smartquotes'], disable_syntax=[], url_schemes=('http', 'https', 'mailto'), heading_anchors=None, html_meta=[], footnote_transition=True, substitutions=[], sub_delimiters=['{', '}'])
loading intersphinx inventory from https://docs.python.org/3.7/objects.inv...
loading intersphinx inventory from https://numpy.org/doc/stable/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/scipy/reference/objects.inv...
loading intersphinx inventory from https://pyquil-docs.rigetti.com/en/stable/objects.inv...
loading intersphinx inventory from https://qiskit.org/documentation/objects.inv...
loading intersphinx inventory from https://qutip.org/docs/latest/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 28 source files that are out of date
updating environment: executing outdated notebooks... Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/ibmq-backends.myst
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/ibmq-backends.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/vqe-pyquil-demo.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/hamiltonians.myst
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/hamiltonians.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/template.myst
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/template.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/simple_landscape.myst
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/simple_landscape.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/pyquil_demo.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/cdr_api.myst
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/cdr_api.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/maxcut-demo.myst
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/maxcut-demo.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/pec-tutorial.myst
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/pec-tutorial.myst
Executing: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/mitiq-paper/mitiq-paper-codeblocks.md
Execution Succeeded: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/mitiq-paper/mitiq-paper-codeblocks.md
done
[new config] 28 added, 0 changed, 0 removed
reading sources... [  3%] apidoc
reading sources... [  7%] changelog
reading sources... [ 10%] code_of_conduct
reading sources... [ 14%] contributing
reading sources... [ 17%] contributing_docs
reading sources... [ 21%] examples/cdr_api
reading sources... [ 25%] examples/examples
reading sources... [ 28%] examples/hamiltonians
reading sources... [ 32%] examples/ibmq-backends
reading sources... [ 35%] examples/maxcut-demo
reading sources... [ 39%] examples/mitiq-paper/mitiq-paper-codeblocks
reading sources... [ 42%] examples/pec-tutorial
reading sources... [ 46%] examples/pyquil_demo
reading sources... [ 50%] examples/simple_landscape
reading sources... [ 53%] examples/template
reading sources... [ 57%] examples/vqe-pyquil-demo
reading sources... [ 60%] guide/guide
reading sources... [ 64%] guide/guide-error-mitigation
reading sources... [ 67%] guide/guide-executors
reading sources... [ 71%] guide/guide-getting-started
reading sources... [ 75%] guide/guide-overview
reading sources... [ 78%] guide/guide-pec
reading sources... [ 82%] guide/guide-zne
reading sources... [ 85%] index
reading sources... [ 89%] readme
reading sources... [ 92%] release
reading sources... [ 96%] toc_contributing
reading sources... [100%] zz_bibliography

WARNING: Execution Failed: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/vqe-pyquil-demo.myst
WARNING: Execution Failed: /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/examples/pyquil_demo.myst
looking for now-outdated files... WARNING: Couldn't find cache key for notebook file examples/pyquil_demo.myst. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/_build/html/reports/pyquil_demo.log
WARNING: Couldn't find cache key for notebook file examples/vqe-pyquil-demo.myst. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/docs/checkouts/readthedocs.org/user_builds/mitiq/checkouts/708/docs/source/_build/html/reports/vqe-pyquil-demo.log
none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [  3%] apidoc
writing output... [  7%] changelog
writing output... [ 10%] code_of_conduct
writing output... [ 14%] contributing
writing output... [ 17%] contributing_docs
writing output... [ 21%] examples/cdr_api
writing output... [ 25%] examples/examples
writing output... [ 28%] examples/hamiltonians
writing output... [ 32%] examples/ibmq-backends
writing output... [ 35%] examples/maxcut-demo
writing output... [ 39%] examples/mitiq-paper/mitiq-paper-codeblocks
writing output... [ 42%] examples/pec-tutorial
writing output... [ 46%] examples/pyquil_demo
writing output... [ 50%] examples/simple_landscape
writing output... [ 53%] examples/template
writing output... [ 57%] examples/vqe-pyquil-demo
writing output... [ 60%] guide/guide
writing output... [ 64%] guide/guide-error-mitigation
writing output... [ 67%] guide/guide-executors
writing output... [ 71%] guide/guide-getting-started
writing output... [ 75%] guide/guide-overview
writing output... [ 78%] guide/guide-pec
writing output... [ 82%] guide/guide-zne
writing output... [ 85%] index
writing output... [ 89%] readme
writing output... [ 92%] release
writing output... [ 96%] toc_contributing
writing output... [100%] zz_bibliography

generating indices... genindex py-modindex done
highlighting module code... [  4%] mitiq.benchmarks.maxcut
highlighting module code... [  9%] mitiq.benchmarks.random_circuits
highlighting module code... [ 14%] mitiq.benchmarks.randomized_benchmarking
highlighting module code... [ 19%] mitiq.benchmarks.utils
highlighting module code... [ 23%] mitiq.cdr.clifford_training_data
highlighting module code... [ 28%] mitiq.interface.mitiq_braket.conversions
highlighting module code... [ 33%] mitiq.interface.mitiq_cirq.cirq_utils
highlighting module code... [ 38%] mitiq.interface.mitiq_pyquil.conversions
highlighting module code... [ 42%] mitiq.interface.mitiq_qiskit.conversions
highlighting module code... [ 47%] mitiq.interface.mitiq_qiskit.qiskit_utils
highlighting module code... [ 52%] mitiq.pec.channels
highlighting module code... [ 57%] mitiq.pec.pec
highlighting module code... [ 61%] mitiq.pec.representations.damping
highlighting module code... [ 66%] mitiq.pec.representations.depolarizing
highlighting module code... [ 71%] mitiq.pec.representations.optimal
highlighting module code... [ 76%] mitiq.pec.sampling
highlighting module code... [ 80%] mitiq.pec.types.types
highlighting module code... [ 85%] mitiq.zne.inference
highlighting module code... [ 90%] mitiq.zne.scaling.folding
highlighting module code... [ 95%] mitiq.zne.scaling.parameter
highlighting module code... [100%] mitiq.zne.zne

writing additional pages... search done
copying images... [  9%] img/vqe-cirq-pauli-sum-mitigation-plot.png
copying images... [ 18%] _build/jupyter_execute/examples/maxcut-demo_48_1.png
copying images... [ 27%] _build/jupyter_execute/examples/maxcut-demo_51_1.png
copying images... [ 36%] _build/jupyter_execute/examples/maxcut-demo_55_1.png
copying images... [ 45%] _build/jupyter_execute/examples/pec-tutorial_62_0.png
copying images... [ 54%] _build/jupyter_execute/examples/simple_landscape_13_0.png
copying images... [ 63%] _build/jupyter_execute/examples/simple_landscape_17_0.png
copying images... [ 72%] _build/jupyter_execute/examples/simple_landscape_23_0.png
copying images... [ 81%] img/unitary_fund_logo.png
copying images... [ 90%] _build/jupyter_execute/examples/template_9_1.png
copying images... [100%] img/factory-plot_fit.png

copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 4 warnings.

The HTML pages are in _build/html.
Updating searchtools for Read the Docs search...
Command time: 482s Return: 0

crazy4pi314 commented 3 years ago

@Misty-W and @rmlarose got this filed from our investigations yesterday :)

rmlarose commented 3 years ago

Thanks Sarah for the clear description.

There are two ways to approach solving this:

See if the newer PyQuil packages come bundled with the servers/compilers bundled natively with the pip packages

(Drastic) Move to hosting our own docs via GH pages

Is there any notion of saving output in a .myst file like in an .ipynb file? If so this could be an alternative.

I would also add "leave as is" as an option. It's certainly not ideal but not terrible to add a note saying "you'll have to run this yourself to see output." People using pyQuil will be familiar with the reason why.

Misty-W commented 3 years ago

thanks Sarah for filing the issue :)

@rmlarose I like the idea of saving the output in a file to solve the immediate problem, although I'm not familiar with RTD so I don't know how to make it work on that end.

crazy4pi314 commented 3 years ago

Re: saving output, that could be really hard as it would have to be checked into the repo, and we would have to update it manually. The point of having executable docs was to avoid precisely this problem. I will look into updating the PyQuil package, as the drastic option (preferable for other reasons) is gonna be a lot more time.

crazy4pi314 commented 3 years ago

https://pyquil-docs.rigetti.com/en/stable/migration.html what I am looking at

Misty-W commented 3 years ago

@crazy4pi314, fyi my examples will need to be updated to work on pyQuil 3.0. Should be easy to fix though.

andreamari commented 3 years ago

What about if we define a mock PyQuil qvm or, more simply, a mock PyQuil executor? They could be used as a second option if the QVM is not available when building the docs.

Qiskit uses a similar trick (https://qiskit.org/documentation/tutorials/simulators/2_device_noise_simulation.html#Terra-Mock-Backends).

Under the hood a mock qvm could:

convert a pyQuil program to a Cirq circuit
execute it with Criq simulator
return measurements counts possibly in the same format of a PyQuil qvm.

Alternatively, a mock PyQuil executor could:

convert a PyQuil program to Cirq
call one of the built-in Cirq executors.

nathanshammah commented 3 years ago

@andreamari +1 for your suggestion.

crazy4pi314 commented 3 years ago

Imo, the effort doing this would be better spend moving docs generation to our control on GitHub as it addresses other issues/features that we have wanted (custom domain, jupyter book support, etc.)

astrojuanlu commented 2 years ago

FYI, Jupyter Book projects can now be hosted on Read the Docs, see https://jupyterbook.org/publish/readthedocs.html (still to be released).

nathanshammah commented 2 years ago

This some wonderful news!

github-actions[bot] commented 2 years ago

This issue had no activity for 2 months, and will be closed in one week unless there is new activity. Cheers!

Misty-W commented 2 years ago

This issue had no activity for 2 months, and will be closed in one week unless there is new activity. Cheers!

@crazy4pi314, @nathanshammah, I haven't really been following this issue lately- are there any plans to solve it?

crazy4pi314 commented 2 years ago

AFAIK there are 2 ways to handle this:

have the results from running this pre-cached via GH actions
Move docs generation to GH pages instead of RTD

We could add this to the agenda for friday, but so far I think 1 has been preferred, but with no implementation of yet.

astrojuanlu commented 2 years ago

There's a contorted but effective workaround for this use case: https://github.com/dfm/rtds-action/

nathanshammah commented 2 years ago

Now that RTD supports Jupyter Book, I'd much rather implement that rather than moving out. With jupyter book we can cache parts that is not essential to continuously rebuild.

Misty-W commented 2 years ago

From @crazy4pi314's comment on PR #1095:

The one thing I am not sure what to do with here is the conf setting which asks the os for the GITHUB_TOKEN. Now this is well defined in the CI runner, but takes setting up api keys for users on local machines doing doc testing. This isn't the worst, but it could be friction we don't want users to deal with.

crazy4pi314 commented 2 years ago

So an update here, I with ~10 lines got our current docs builds in CI to publish to GH pages. You can see that the pyquill examples are executed correctly (#1135):

RTD: https://mitiq.readthedocs.io/en/stable/examples/pyquil_demo.html GH pages: https://unitaryfund.github.io/mitiq/examples/pyquil_demo.html

I know this is not the best solution, but it at least currently works. What I would suggest is that we use a custom domain (we own some for mitiq) and have that re-direct to GH Pages. For posterity, we can probably re-direct RTD to the new link.

Misty-W commented 2 years ago

Looks awesome, thank you @crazy4pi314! 🎉

nathanshammah commented 2 years ago

Very nice @crazy4pi314. Maybe we could start testing this. Having versions as in RTD and docs preview in PR would be great. Maybe there are ways to keep these features also on GH pages.

crazy4pi314 commented 2 years ago

https://holzhaus.github.io/sphinx-multiversion/master/index.html <- This sphinx plugin would give us version selection in the TOC on the side, drafts for open PRs would be def possible but not straight forward.

I motion that this is closed for now, and other issues opened for adding the versioning to the GH pages test doc site, feel free to re-open if needed.

unitaryfund / mitiq