Add `doctest` support to run docstring example automatically

[antalszava]

Feature details

It would be good to add doctest support. See https://github.com/PennyLaneAI/pennylane/pull/705.

Implementation

No response

How important would you say this feature is?

1: Not important. Would be nice to have.

Additional information

No response

[timmysilv]

note that I ran doctest with today's master (and a config that just does import pennylane as qml) and saw ~5000 failures, so I don't believe that simply enabling doctest will be an option.

That said, I will leave this issue open for discoverability/discussion

[josh146]

@timmysilv out of curiosity do you know what the main failure reason was? Something that shouldn't work (e.g., docstrings with external hardware device examples) or things that should work?

[timmysilv]

I saw a couple hundred failing because numpy wasn't imported, but there didn't seem to be one main reason. It was almost always a NameError because the example used a variable that wasn't previously defined. Perhaps because we'd have some code, then a line or two talking, then a continuation of the code. We'd also often drop the qml. prefix in docstrings, such as just calling Exp instead of qml.ops.Exp because it's obvious from context (though of course, doctest begs to differ!)

[timmysilv]

some heavy-hitting NameErrors (the number is the number of occurrences):

  83     NameError: name 'qscript' is not defined
  95     NameError: name 'tape' is not defined
  98     NameError: name 'torch' is not defined
 110     NameError: name 'dev' is not defined
 160     NameError: name 'op2' is not defined
 160     NameError: name 'op3' is not defined
 182     NameError: name 'U' is not defined
 213     NameError: name 'circuit' is not defined
 590     NameError: name 'np' is not defined

I also saw 104 instances of TypeError: 'module' object is not callable

[josh146]

Thanks! Definitely not priority for now, but from what I recall this is something we can configure in doctest ('common imports'), but I could also be completely wrong.

[timmysilv]

alternate idea: make it opt-in with .. ipython directives instead. you can configure imports using ipython_execlines in doc/conf.py, then you can change code blocks to use it with the IPython Sphinx Directive. The main con of this IMO is that it shows IPython-style output which can be kinda ugly (see the linked doc for examples).

There might be something similar for pycon (eg. sphinx-autorun, though idk how maintained it is) which would ofc be great.