QSVM classification tutorial fails with Scipy 1.6.0

[mtreinish]

Information

• Qiskit Aqua version: 0.8.1
• Python version: 3.8
• Operating system: Linux

What is the current behavior?

When running the qsvm tutorial with the latest scipy (1.6.0) the tutorial fails in the qp_solver module trying to run cvxpy to solve the quadratic programming problem with a mismatched dimension error:

ValueError                                Traceback (most recent call last)
<ipython-input-3-3863988dcf61> in <module>
      5 quantum_instance = QuantumInstance(backend, shots=1024, seed_simulator=seed, seed_transpiler=seed)
      6 
----> 7 result = qsvm.run(quantum_instance)
      8 
      9 print(f'Testing success ratio: {result["testing_accuracy"]}')

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/qiskit/aqua/algorithms/quantum_algorithm.py in run(self, quantum_instance, **kwargs)
     69                 self.quantum_instance = quantum_instance
     70 
---> 71         return self._run()
     72 
     73     @abstractmethod

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/qiskit/aqua/algorithms/classifiers/qsvm/qsvm.py in _run(self)
    470 
    471     def _run(self):
--> 472         return self.instance.run()
    473 
    474     @property

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/qiskit/aqua/algorithms/classifiers/qsvm/_qsvm_binary.py in run(self)
    133     def run(self):
    134         """Put the train, test, predict together."""
--> 135         self.train(self._qalgo.training_dataset[0], self._qalgo.training_dataset[1])
    136         if self._qalgo.test_dataset is not None:
    137             self.test(self._qalgo.test_dataset[0], self._qalgo.test_dataset[1])

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/qiskit/aqua/algorithms/classifiers/qsvm/_qsvm_binary.py in train(self, data, labels)
     80         labels = labels * 2 - 1  # map label from 0 --> -1 and 1 --> 1
     81         labels = labels.astype(np.float)
---> 82         [alpha, b, support] = optimize_svm(kernel_matrix, labels, scaling=scaling, lambda2=lambda2)
     83         support_index = np.where(support)
     84         alphas = alpha[support_index]

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/qiskit/aqua/utils/qp_solver.py in optimize_svm(kernel_matrix, y, scaling, maxiter, show_progress, max_iters, lambda2)
     93         [G@x <= h,
     94          A@x == b])
---> 95     prob.solve(verbose=show_progress, qcp=True)
     96     result = np.asarray(x.value).reshape((n, 1))
     97     alpha = result * scaling

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/cvxpy/problems/problem.py in solve(self, *args, **kwargs)
    288         else:
    289             solve_func = Problem._solve
--> 290         return solve_func(self, *args, **kwargs)
    291 
    292     @classmethod

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/cvxpy/problems/problem.py in _solve(self, solver, warm_start, verbose, parallel, gp, qcp, **kwargs)
    567 
    568         self._construct_chains(solver=solver, gp=gp)
--> 569         data, solving_inverse_data = self._solving_chain.apply(
    570             self._intermediate_problem)
    571         solution = self._solving_chain.solve_via_data(

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/cvxpy/reductions/chain.py in apply(self, problem)
     63         inverse_data = []
     64         for r in self.reductions:
---> 65             problem, inv = r.apply(problem)
     66             inverse_data.append(inv)
     67         return problem, inverse_data

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/cvxpy/reductions/matrix_stuffing.py in apply(self, problem)
     84         # Form the constraints
     85         extractor = CoeffExtractor(inverse_data)
---> 86         new_obj, new_var, r = self.stuffed_objective(problem, extractor)
     87         inverse_data.r = r
     88         # Lower equality and inequality to Zero and NonPos.

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/cvxpy/reductions/qp2quad_form/qp_matrix_stuffing.py in stuffed_objective(self, problem, extractor)
     47         # extract to x.T * P * x + q.T * x + r
     48         expr = problem.objective.expr.copy()
---> 49         P, q, r = extractor.quad_form(expr)
     50 
     51         # concatenate all variables in one vector

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/cvxpy/utilities/coeff_extractor.py in quad_form(self, expr)
    156         for var_id, offset in offsets:
    157             if var_id in coeffs:
--> 158                 P = sp.block_diag([P, coeffs[var_id]['P']])
    159                 q = np.concatenate([q, coeffs[var_id]['q']])
    160             else:

~/git/qiskit/qiskit-tutorial/test_notebooks/lib/python3.8/site-packages/scipy/sparse/construct.py in block_diag(mats, format, dtype)
    691     row = np.concatenate(row)
    692     col = np.concatenate(col)
--> 693     data = np.concatenate(data)
    694     return coo_matrix((data, (row, col)),
    695                       shape=(r_idx, c_idx),

<__array_function__ internals> in concatenate(*args, **kwargs)

ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 1 dimension(s) and the array at index 1 has 2 dimension(s)

Steps to reproduce the problem

Run https://github.com/Qiskit/qiskit-tutorials/blob/master/tutorials/machine_learning/01_qsvm_classification.ipynb with scipy 1.6.0 installed.

What is the expected behavior?

The tutorial works without error

Suggested solutions

It looks like this is being caused by a mismatched dimension between P we pass into cvxpy as part of the quadratic programming solver and when cvxpy tries to make a sparse block diagonal array with the input it errors because the dimensions differ, probably (n, 1) != (n,). We should either fix the issue in qp_solver, or if it's an issue in cvxpy's internals report this upstream and try to get a fix.

[manoelmarques]

We run that same tutorial on the Aqua CI and it passes on python 3.8. For a recent run: https://github.com/Qiskit/qiskit-aqua/runs/1824072555?check_suite_focus=true

Also, our CI saves the tutorials as artifacts that can be downloaded. I downloaded the artifact tutorials3.8.zip and looked at the notebook in question and it seemed fine to me. Our tutorial CI is using scipy 1.6.0: Requirement already satisfied: scipy>=1.4 in /opt/hostedtoolcache/Python/3.8.7/x64/lib/python3.8/site-packages (from qiskit-aqua==0.9.0) (1.6.0)

There is a problem with cvxpy 1.1.8 installing on python 3.7 and we pinned this to !=1.1.8

[mtreinish]

The difference in the qiskit/qiskit CI is that it runs on released aqua and not master (since they are run to verify that the versions in the metapackage are compatible and don't regress/are backwards compatible) Looking at the git log I'm wondering if https://github.com/Qiskit/qiskit-aqua/commit/739e3302942a66070566ad28e3708a5f7be93550#diff-373b76176a94d6c7d7584735fbb83cfde7d8b4f678431361d7ae85958f5b323a fixed this issue. I know we're in a weird place being in the middle of a transfer but maybe we want to push that out as part of a 0.8.2

[mtreinish]

I'm also working on trying to fix the CI jobs in qiskit/qiskit, the numpy 1.20.0 release and cvxpy 1.1.8 release broken things have it stuck pretty good. But, once it's working again I can try removing the constraints pin and seeing if the job works

[woodsp-ibm]

@mtreinish Did you get a chance to check as per above?

[woodsp-ibm]

No longer relevant