QAOA: remember intermediate results and choose the best of all known eigenstates

[RaoulHeese]

What is the expected enhancement?

When using result = MinimumEigenOptimizer(QAOA(...)).solve(qubo), the intermediate results of QAOA are not considered in the final outcome. Only the eigenstates from the final parameter values are sampled in https://github.com/Qiskit/qiskit-terra/blob/23acf363e52b6436998afc3150e117625cc6b7d6/qiskit/algorithms/minimum_eigen_solvers/vqe.py#L576 and are returned to MinimumEigenOptimizer._solve_internal in qiskit-optimization to choose the best from. This is highly counter-intuitive since valuable results might get lost during the optimization. Or, in other words, the final result might be worse than an intermediate result.

To fix this issue without much effort, the optimizer function https://github.com/Qiskit/qiskit-terra/blob/23acf363e52b6436998afc3150e117625cc6b7d6/qiskit/algorithms/minimum_eigen_solvers/vqe.py#L598 can be extended to store state_fn from self._circuit_sampler.convert (as in _get_eigenstate) over the course of the optimization. With this improvement, all eigenstates of the optimization history can be returned instead of just one from the best parameter set.

(Naturally, this enhancement request also applies to VQE in general.)

[woodsp-ibm]

I will just note that when an optimizer is computing a finite diff gradient that the objective function here will get called for that gradient computation as well which would leads to a much larger set of values than just the points traversed.

Also the minimum_eigenvalue/state is a result of the minimum_eigen_solver and may not be solved variationally where intermediate states are available. Hence the availability of such intermediate results to a higher level algorithm would have to be treated conditionally upon how things are solved.

When using QAOA (or VQE) directly one can use the callback to capture all the intermediate parameters and given the anstaz get the states afterwards so you can do this already at this level of the algorithms with whats there. This request seems perhaps to be asking to make this more readily available for consumption, particularly for the Qiskit Optimization level. As such perhaps a corresponding Issue should be opened there referencing this one in order to have that discussion at that level too.

[RaoulHeese]

Thanks @woodsp-ibm, I have a follow-up question regarding your hint for the callback: If I understand correctly, the callback only takes the parameter values as an argument and therefore, the measurement results (i.e., counts) can not be retrieved in this way. Consequently, another run has to be performed to get (new) counts, which is an unsuited approach for real hardware in my opinion and also not ideal for simulators.

[woodsp-ibm]

Yes, the callback only has the parameters - the ansatz would need to be executed again. Its what is done with VQE at the end to get the final state. If you want all the intermediate states then indeed, as you say, its less than ideal to say the least. I recall having discussions in the past in regards of the storing the data, I think the callback was chosen since we can end up with tens of thousands of iterations for chemistry programs; and each iteration with VQE can be many circuits for the operator. I will note that VQE and QAOA are likely to be refactored over to use the new Estimator primitives later. So any change in this regard of storing history and making it available to applications I doubt would be considered at all ahead of that.

[woodsp-ibm]

VQE and QAOA changed when they were re-written to primitives. As this is about the former algorithms, which are now deprecated and soon to be removed, I am going to close this off.