[BUG] Inconsistent behaviour with non-commuting observables

[timmysilv]

Expected behavior

Originally found by @albi3ro

When running a circuit that returns non-commuting observables, I expect it to either fail every time, or succeed every time. This should likely be decided by the device that executes the circuit, but the details can be determined as a result of this bug report.

Actual behavior

In the source code, only circuit2 fails. circuit2 fails because it does the following:

1. checks if all ops are supported by its device, default.qubit. It sees that Permute is not supported
2. because there is an unsupported operator, it calls circuit.expand (which calls qml.tape.tape.expand_tape(circuit))
3. expand_tape tries to diagonalize the circuit observables but fails because they are non-commuting

circuit1 succeeds because all its operators are supported, so it never does the expansion in step 2 above.

Additional information

Lots of possible results here. I'll propose a few:

1. tape expansion and trying to diagonalize measurements become tightly coupled
2. only diagonalize if expand_measurements=True
3. try to diagonalize measurements (or simply check that measurements commute) even if all ops are supported
4. give this decision to the device and have it factored into the circuit expansion workflow

The fourth option seems most sensible but might require a bit more effort.

Source code

@qml.qnode(qml.device('default.qubit', wires=2))
def circuit1():
    qml.RX(1.0, wires=0)
    return qml.expval(2.0 * qml.PauliY(0) + qml.PauliX(0)),  qml.expval(qml.PauliX(0))

@qml.qnode(qml.device('default.qubit', wires=3))
def circuit2():
    qml.RX(1.0, wires=0)
    qml.Permute([0,1,2], wires=(2,1,0))
    return qml.expval(2.0 * qml.PauliY(0) + qml.PauliX(0)),  qml.expval(qml.PauliX(0))

circuit1()  # passes
circuit2()  # raises an error

Tracebacks

Traceback (most recent call last):
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/tape/tape.py", line 87, in rotations_and_diagonal_measurements
    rotations, diag_obs = qml.pauli.diagonalize_qwc_pauli_words(tape._obs_sharing_wires)
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/pauli/utils.py", line 1228, in diagonalize_qwc_pauli_words
    raise ValueError("The list of Pauli words are not qubit-wise commuting.")
ValueError: The list of Pauli words are not qubit-wise commuting.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/qnode.py", line 867, in __call__
    res = qml.execute(
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/interfaces/execution.py", line 407, in execute
    qml.interfaces.cache_execute(
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/interfaces/execution.py", line 204, in wrapper
    res = fn(execution_tapes.values(), **kwargs)
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/interfaces/execution.py", line 129, in fn
    tapes = [expand_fn(tape) for tape in tapes]
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/interfaces/execution.py", line 129, in <listcomp>
    tapes = [expand_fn(tape) for tape in tapes]
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/interfaces/execution.py", line 388, in <lambda>
    expand_fn = lambda tape: device.expand_fn(tape, max_expansion=max_expansion)
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/_device.py", line 696, in expand_fn
    return self.default_expand_fn(circuit, max_expansion=max_expansion)
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/_device.py", line 670, in default_expand_fn
    circuit = circuit.expand(depth=max_expansion, stop_at=self.stopping_condition)
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/tape/qscript.py", line 1090, in expand
    new_script = qml.tape.tape.expand_tape(
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/tape/tape.py", line 181, in expand_tape
    diagonalizing_gates, diagonal_measurements = rotations_and_diagonal_measurements(tape)
  File "/Users/matthews/src/github.com/PennyLaneAI/pennylane/pennylane/tape/tape.py", line 98, in rotations_and_diagonal_measurements
    raise qml.QuantumFunctionError(_err_msg_for_some_meas_not_qwc(tape.measurements)) from e
pennylane.QuantumFunctionError: Only observables that are qubit-wise commuting Pauli words can be returned on the same wire, some of the following measurements do not commute:
[  (1.0) [X0]
+ (2.0) [Y0], expval(PauliX(wires=[0]))]

System information

Name: PennyLane
Version: 0.30.0.dev0
Summary: PennyLane is a Python quantum machine learning library by Xanadu Inc.
Home-page: https://github.com/XanaduAI/pennylane
Author: 
Author-email: 
License: Apache License 2.0
Location: /Users/matthews/.pyenv/versions/3.9.13/envs/pl/lib/python3.9/site-packages
Editable project location: /Users/matthews/src/github.com/PennyLaneAI/pennylane
Requires: appdirs, autograd, autoray, cachetools, networkx, numpy, pennylane-lightning, requests, rustworkx, scipy, semantic-version, toml
Required-by: PennyLane-Lightning

Platform info:           macOS-13.3.1-arm64-arm-64bit
Python version:          3.9.13
Numpy version:           1.23.5
Scipy version:           1.10.0
Installed devices:
- default.gaussian (PennyLane-0.30.0.dev0)
- default.mixed (PennyLane-0.30.0.dev0)
- default.qubit (PennyLane-0.30.0.dev0)
- default.qubit.autograd (PennyLane-0.30.0.dev0)
- default.qubit.jax (PennyLane-0.30.0.dev0)
- default.qubit.tf (PennyLane-0.30.0.dev0)
- default.qubit.torch (PennyLane-0.30.0.dev0)
- default.qutrit (PennyLane-0.30.0.dev0)
- null.qubit (PennyLane-0.30.0.dev0)
- lightning.qubit (PennyLane-Lightning-0.30.0.dev5)

Existing GitHub issues

• [X] I have searched existing GitHub issues to make sure the issue does not already exist.

[josh146]

I feel like I'm forgetting something, but why isn't qml.transforms.split_non_commuting called before tape expansion?

[timmysilv]

In this particular case, it's because there's a Hamiltonian measurement. If there wasn't, we'd have called it

[josh146]

Ahhhhhhhh. Interesting. cc @trbromley, because this is definitely something we could support if we want to. That is, split_non_commuting could be updated to work with op_sum/Hamiltonians. I feel like this is the very high-level 'feature issue' underlying this bug if we want to address it.

[dwierichs]

Another case where this behaviour becomes relevant is when creating and executing tapes manually. In this case, no error will be raised when reusing wires in non-qwc measurements, but the diagonalizing gates of one measurement affect the outcome of other measurements:

with qml.queuing.AnnotatedQueue() as q:
    qml.expval(qml.PauliZ(0) @ qml.PauliX(1))
    qml.probs(wires=[0, 1])

tape = qml.tape.QuantumScript.from_queue(q)
>>> dev.execute(tape)
(array(0.), array([0.5, 0.5, 0. , 0. ]))

The state in which we measure the probabilities should be [1, 0, 0, 0] but the diagonalizing gates of qml.PauliZ(0)@qml.Paulix(1), which are applied to the state to compute its expectation value, alter it into [1, 1, 0, 0]/np.sqrt(2), yielding a wrong result.

[dwierichs]

This bug seems to be resolved at the latest by #5424. :) Good to close @timmysilv @josh146 ?

[timmysilv]

confirmed the example passes for both now! lgtm