search

qiskit-community / qiskit-machine-learning

Quantum Machine Learning
https://qiskit-community.github.io/qiskit-machine-learning/
Apache License 2.0
686 stars 326 forks source link

Memory Error while using VQC #837

Open Ilyaant opened 1 month ago

Ilyaant commented 1 month ago

Hello! While fitting VQC on a (3599, 20) training dataset I faced the following exception:

MemoryError: Unable to allocate 1.64 GiB for an array with shape (1048576,) and data type <U420

Could you please explain the error to me? I'm working on a machine with 120 GB of RAM and don't understand how the program cannot allocate 1.64 GB. Is there a way solve the problem?

The idea I have is to divide the dataset into smaller parts and fit the VQC on them. But here's another question: can I call the fit() method multiple times on different data chunks, or will the model be training from scratch each time?

Still, I don't like this idea, I'd want to do something with the memory usage limitation. Thank you!

edoaltamura commented 1 month ago

Hi @Ilyaant, thanks for your question. Could you please share the part of your code that triggers the allocation? This sometimes happens when trying to accidentally allocate eg 2**1048576 rather than 1048576 bits, but somehow NumPy recognises it as of reasonable size.

Ilyaant commented 1 month ago

Hello @edoaltamura, thank you very much for your answer. The code snippet that I run is:

num_features = X_train.shape[1]

feature_map = ZZFeatureMap(feature_dimension=num_features, reps=1)
ansatz = RealAmplitudes(num_qubits=num_features, reps=3)
optimizer = COBYLA(maxiter=100)

vqc = VQC(
    feature_map=feature_map,
    ansatz=ansatz,
    optimizer=optimizer
)

vqc.fit(X_train, y_train.to_numpy())

The vqc.fit(X_train, y_train.to_numpy()) is the exact row that causes the exception.

The full description of the exception:

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\neural_networks\sampler_qnn.py:393, in SamplerQNN._forward(self, input_data, weights)
    [392](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:392) try:
--> [393](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:393)     results = job.result()
    [394](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:394) except Exception as exc:

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit\primitives\primitive_job.py:51, in PrimitiveJob.result(self)
     [50](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/primitive_job.py:50) self._check_submitted()
---> [51](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/primitive_job.py:51) return self._future.result()

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py:456, in Future.result(self, timeout)
    [455](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/_base.py:455) elif self._state == FINISHED:
--> [456](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/_base.py:456)     return self.__get_result()
    [457](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/_base.py:457) else:

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py:401, in Future.__get_result(self)
    [400](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/_base.py:400) try:
--> [401](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/_base.py:401)     raise self._exception
    [402](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/_base.py:402) finally:
    [403](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/_base.py:403)     # Break a reference cycle with the exception in self._exception

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\thread.py:58, in _WorkItem.run(self)
     [57](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/thread.py:57) try:
---> [58](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/thread.py:58)     result = self.fn(*self.args, **self.kwargs)
     [59](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/concurrent/futures/thread.py:59) except BaseException as exc:

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit\primitives\sampler.py:109, in Sampler._call(self, circuits, parameter_values, **run_options)
    [107](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:107)     qargs_list.append(self._qargs_list[i])
    [108](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:108) probabilities = [
--> [109](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:109)     Statevector(bound_circuit_to_instruction(circ)).probabilities_dict(
    [110](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:110)         qargs=qargs, decimals=16
    [111](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:111)     )
    [112](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:112)     for circ, qargs in zip(bound_circuits, qargs_list)
    [113](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:113) ]
    [114](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/primitives/sampler.py:114) if shots is not None:

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit\quantum_info\states\quantum_state.py:240, in QuantumState.probabilities_dict(self, qargs, decimals)
    [221](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:221) """Return the subsystem measurement probability dictionary.
    [222](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:222) 
    [223](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:223) Measurement probabilities are with respect to measurement in the
   (...)
    [238](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:238)     dict: The measurement probabilities in dict (ket) form.
    [239](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:239) """
--> [240](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:240) return self._vector_to_dict(
    [241](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:241)     self.probabilities(qargs=qargs, decimals=decimals),
    [242](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:242)     self.dims(qargs),
    [243](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:243)     string_labels=True,
    [244](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:244) )

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit\quantum_info\states\quantum_state.py:399, in QuantumState._vector_to_dict(vec, dims, decimals, string_labels)
    [398](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:398) # Convert to ket tuple based on subsystem dimensions
--> [399](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:399) kets = QuantumState._index_to_ket_array(inds, dims, string_labels=string_labels)
    [401](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:401) # Make dict of tuples

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit\quantum_info\states\quantum_state.py:371, in QuantumState._index_to_ket_array(inds, dims, string_labels)
    [370](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:370)         str_kets = np.char.add(",", str_kets)
--> [371](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:371)     str_kets = np.char.add(row, str_kets)
    [372](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit/quantum_info/states/quantum_state.py:372) return str_kets.T

MemoryError: Unable to allocate 1.64 GiB for an array with shape (1048576,) and data type <U420

The above exception was the direct cause of the following exception:

QiskitMachineLearningError                Traceback (most recent call last)
Cell In[12], [line 14](vscode-notebook-cell:?execution_count=12&line=14)
      [7](vscode-notebook-cell:?execution_count=12&line=7) vqc = VQC(
      [8](vscode-notebook-cell:?execution_count=12&line=8)     feature_map=feature_map,
      [9](vscode-notebook-cell:?execution_count=12&line=9)     ansatz=ansatz,
     [10](vscode-notebook-cell:?execution_count=12&line=10)     optimizer=optimizer
     [11](vscode-notebook-cell:?execution_count=12&line=11) )
     [13](vscode-notebook-cell:?execution_count=12&line=13) # vqc.fit(X_train.to_numpy(), y_train.to_numpy())
---> [14](vscode-notebook-cell:?execution_count=12&line=14) vqc.fit(X_train, y_train.to_numpy())

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\algorithms\trainable_model.py:199, in TrainableModel.fit(self, X, y)
    [196](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:196) if not self._warm_start:
    [197](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:197)     self._fit_result = None
--> [199](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:199) self._fit_result = self._fit_internal(X, y)
    [200](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:200) return self

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\algorithms\classifiers\vqc.py:167, in VQC._fit_internal(self, X, y)
    [164](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/classifiers/vqc.py:164)     self._neural_network.set_interpret(self._get_interpret(num_classes), num_classes)
    [166](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/classifiers/vqc.py:166) function = self._create_objective(X, y)
--> [167](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/classifiers/vqc.py:167) return self._minimize(function)

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\algorithms\trainable_model.py:295, in TrainableModel._minimize(self, function)
    [291](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:291)     optimizer_result = self._optimizer(
    [292](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:292)         fun=objective, x0=initial_point, jac=function.gradient
    [293](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:293)     )
    [294](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:294) else:
--> [295](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:295)     optimizer_result = self._optimizer.minimize(
    [296](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:296)         fun=objective,
    [297](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:297)         x0=initial_point,
    [298](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:298)         jac=function.gradient,
    [299](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:299)     )
    [300](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/trainable_model.py:300) return optimizer_result

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_algorithms\optimizers\scipy_optimizer.py:148, in SciPyOptimizer.minimize(self, fun, x0, jac, bounds)
    [145](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:145)     swapped_deprecated_args = True
    [146](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:146)     self._options["maxfun"] = self._options.pop("maxiter")
--> [148](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:148) raw_result = minimize(
    [149](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:149)     fun=fun,
    [150](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:150)     x0=x0,
    [151](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:151)     method=self._method,
    [152](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:152)     jac=jac,
    [153](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:153)     bounds=bounds,
    [154](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:154)     options=self._options,
    [155](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:155)     **self._kwargs,
    [156](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:156) )
    [157](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:157) if swapped_deprecated_args:
    [158](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_algorithms/optimizers/scipy_optimizer.py:158)     self._options["maxiter"] = self._options.pop("maxfun")

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\optimize\_minimize.py:737, in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
    [734](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:734)     res = _minimize_tnc(fun, x0, args, jac, bounds, callback=callback,
    [735](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:735)                         **options)
    [736](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:736) elif meth == 'cobyla':
--> [737](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:737)     res = _minimize_cobyla(fun, x0, args, constraints, callback=callback,
    [738](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:738)                            bounds=bounds, **options)
    [739](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:739) elif meth == 'cobyqa':
    [740](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:740)     res = _minimize_cobyqa(fun, x0, args, bounds, constraints, callback,
    [741](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_minimize.py:741)                            **options)

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\optimize\_cobyla_py.py:35, in synchronized.<locals>.wrapper(*args, **kwargs)
     [32](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:32) @functools.wraps(func)
     [33](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:33) def wrapper(*args, **kwargs):
     [34](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:34)     with _module_lock:
---> [35](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:35)         return func(*args, **kwargs)

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\optimize\_cobyla_py.py:278, in _minimize_cobyla(fun, x0, args, constraints, rhobeg, tol, maxiter, disp, catol, callback, bounds, **unknown_options)
    [275](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:275) def _jac(x, *args):
    [276](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:276)     return None
--> [278](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:278) sf = _prepare_scalar_function(fun, x0, args=args, jac=_jac)
    [280](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:280) def calcfc(x, con):
    [281](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_cobyla_py.py:281)     f = sf.fun(x)

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\optimize\_optimize.py:288, in _prepare_scalar_function(fun, x0, jac, args, bounds, epsilon, finite_diff_rel_step, hess)
    [284](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_optimize.py:284)     bounds = (-np.inf, np.inf)
    [286](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_optimize.py:286) # ScalarFunction caches. Reuse of fun(x) during grad
    [287](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_optimize.py:287) # calculation reduces overall function evaluations.
--> [288](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_optimize.py:288) sf = ScalarFunction(fun, x0, args, grad, hess,
    [289](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_optimize.py:289)                     finite_diff_rel_step, bounds, epsilon=epsilon)
    [291](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_optimize.py:291) return sf

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\optimize\_differentiable_functions.py:222, in ScalarFunction.__init__(self, fun, x0, args, grad, hess, finite_diff_rel_step, finite_diff_bounds, epsilon)
    [219](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:219)     finite_diff_options["as_linear_operator"] = True
    [221](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:221) # Initial function evaluation
--> [222](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:222) self._update_fun()
    [224](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:224) # Initial gradient evaluation
    [225](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:225) self._wrapped_grad, self._ngev = _wrapper_grad(
    [226](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:226)     grad,
    [227](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:227)     fun=self._wrapped_fun,
    [228](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:228)     args=args,
    [229](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:229)     finite_diff_options=finite_diff_options
    [230](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:230) )

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\optimize\_differentiable_functions.py:294, in ScalarFunction._update_fun(self)
    [292](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:292) def _update_fun(self):
    [293](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:293)     if not self.f_updated:
--> [294](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:294)         fx = self._wrapped_fun(self.x)
    [295](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:295)         if fx < self._lowest_f:
    [296](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:296)             self._lowest_x = self.x

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\scipy\optimize\_differentiable_functions.py:20, in _wrapper_fun.<locals>.wrapped(x)
     [16](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:16) ncalls[0] += 1
     [17](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:17) # Send a copy because the user may overwrite it.
     [18](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:18) # Overwriting results in undefined behaviour because
     [19](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:19) # fun(self.x) will change self.x, with the two no longer linked.
---> [20](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:20) fx = fun(np.copy(x), *args)
     [21](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:21) # Make sure the function returns a true scalar
     [22](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/scipy/optimize/_differentiable_functions.py:22) if not np.isscalar(fx):

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\algorithms\objective_functions.py:191, in OneHotObjectiveFunction.objective(self, weights)
    [189](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:189) def objective(self, weights: np.ndarray) -> float:
    [190](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:190)     # probabilities is of shape (N, num_outputs)
--> [191](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:191)     probs = self._neural_network_forward(weights)
    [192](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:192)     # float(...) is for mypy compliance
    [193](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:193)     value = float(np.sum(self._loss(probs, self._y)) / self._num_samples)

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\algorithms\objective_functions.py:102, in ObjectiveFunction._neural_network_forward(self, weights)
     [97](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:97) # if we get the same weights, we don't compute the forward pass again.
     [98](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:98) if self._last_forward_weights is None or (
     [99](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:99)     not np.all(np.isclose(weights, self._last_forward_weights))
    [100](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:100) ):
    [101](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:101)     # compute forward and cache the results for re-use in backward
--> [102](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:102)     self._last_forward = self._neural_network.forward(self._X, weights)
    [103](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:103)     # a copy avoids keeping a reference to the same array, so we are sure we have
    [104](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:104)     # different arrays on the next iteration.
    [105](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/algorithms/objective_functions.py:105)     self._last_forward_weights = np.copy(weights)

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\neural_networks\neural_network.py:228, in NeuralNetwork.forward(self, input_data, weights)
    [226](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/neural_network.py:226) input_, shape = self._validate_input(input_data)
    [227](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/neural_network.py:227) weights_ = self._validate_weights(weights)
--> [228](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/neural_network.py:228) output_data = self._forward(input_, weights_)
    [229](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/neural_network.py:229) return self._validate_forward_output(output_data, shape)

File c:\Users\iantonov\AppData\Local\Programs\Python\Python312\Lib\site-packages\qiskit_machine_learning\neural_networks\sampler_qnn.py:395, in SamplerQNN._forward(self, input_data, weights)
    [393](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:393)     results = job.result()
    [394](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:394) except Exception as exc:
--> [395](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:395)     raise QiskitMachineLearningError("Sampler job failed.") from exc
    [396](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:396) result = self._postprocess(num_samples, results)
    [398](file:///C:/Users/iantonov/AppData/Local/Programs/Python/Python312/Lib/site-packages/qiskit_machine_learning/neural_networks/sampler_qnn.py:398) return result

QiskitMachineLearningError: 'Sampler job failed.'
Ilyaant commented 4 weeks ago

@edoaltamura For now, it seems to me that the issue is inside the fit() method because the command y_train.to_numpy() itself runs without problems. Please, take a look when you have time. Thank you

oscar-wallis commented 14 hours ago

Hi @Ilyaant , so I am not 100% sure why this is going wrong but it is worth noting that log2(1048576) = 20 exactly, so i am assuming that something is going wrong with your array size of 20. Try checking the shape of the arrays heading into the network / what happens to them when you wrong .to_numpy(). Otherwise, it is challenging for us to investigate this bug without access to the full script. I'd also try running VQC.score() with some of your data before fitting. This is essentially checking if the problem is VQC or the optimiser COBYLA.