q-optimize / c3

Toolset for control, calibration and characterization of physical systems
https://c3-toolset.readthedocs.io/
Apache License 2.0
66 stars 36 forks source link

Removing tensorflow bottlenecks #177

Closed nwittler closed 2 years ago

nwittler commented 2 years ago

What

Fixes the control flow of the simulation to allow for efficient evaluation as compiled Tensorflow function.

Why

This removes performance bottlenecks, allowing for better scaling and parallelization. Closes #74

How

Introducing one @tf.function decorator at the goal function level in the optimizer class. And several fixes along the way to make the code compatible with graph compilation, e.g. supported operations and function names, avoiding state dependent logic that would require retracing.

Remarks

This is a proof of concept and base for discussion. Solutions are currently bolted onto the Experiment class and should probably move to a new Propagation class. Depending on the method, a number of pre-computations might be beneficial to make the pure math portion be as lean and efficient as possible.

Checklist

Please include and complete the following checklist. Your Pull Request is (in most cases) not ready for review until the following have been completed. You can create a draft PR while you are still completing the checklist. Check the Contribution Guidelines for more details. You can mark an item as complete with the - [x] prefix

lazyoracle commented 2 years ago

The formatting and linting test should be fixed by merging (or preferably rebasing on) latest dev and then either amending your latest commit if you have pre-commit installed or running black c3/ if you have black installed in your virtual environment.

nwittler commented 2 years ago

The formatting and linting test should be fixed by merging (or preferably rebasing on) latest dev and then either amending your latest commit if you have pre-commit installed or running black c3/ if you have black installed in your virtual environment.

Weird, I actually ran black manually and I have the pre-commits in the same setup I'm always using. But yes, there's a lot of rough stuff still left.

lazyoracle commented 2 years ago

The formatting and linting test should be fixed by merging (or preferably rebasing on) latest dev and then either amending your latest commit if you have pre-commit installed or running black c3/ if you have black installed in your virtual environment.

Weird, I actually ran black manually and I have the pre-commits in the same setup I'm always using. But yes, there's a lot of rough stuff still left.

You probably have an older version of black. Maybe try pip install -U black and then black c3/?

lazyoracle commented 2 years ago

test_qiskit_physics fails with an assertion error on the final readout populations. This indicates some possible change in the API of the compute_propagators(), evaluate() or process() methods since the qiskit backend (mostly) only uses the public methods without relying on any specific implementation quirks.

git bisect indicates that this failing test was first introduced in fb3be66d4035c28b8bf61905d33a027f3331b4dc , quite a lot of changes in that commit - so not sure how to pinpoint what caused the failing test.

Update 1 - test_model_learning.py was initially giving an assertion error on not converging to the given tolerance bound but subsequently started throwing an obscure numpy clipping error - also from the same commit as above.

Update 2 - Unsupported TF Graph operations seem to be the reason for a lot of the still failing tests (test_optim() and test_c1_robust()) - OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed and the tests pass when you remove the @tf.function decorator for optimal control.

Update 3 - test_noise_devices seems to be failing due to a type mismatch in what process expects as an input signal (List[Dict[str,Any]]) and what it gets. A similar error was fixed in e20d2d5 where the signal being provided as an input in the test code was not of the required type. However, in test_noise_devices, the signal is not directly provided in the testing code and comes through the experiment -> propagation -> generator call stack. Needs investigation.

lgtm-com[bot] commented 2 years ago

This pull request introduces 1 alert when merging 41db44bc4f72be0b8abae96cccd564aca1708df8 into 7801dc2daf21499f3837ec334063b8a8c3cb4951 - view on LGTM.com

new alerts:

codecov[bot] commented 2 years ago

Codecov Report

Merging #177 (2815cb4) into dev (0c2a80a) will increase coverage by 0.58%. The diff coverage is 92.23%.

@@            Coverage Diff             @@
##              dev     #177      +/-   ##
==========================================
+ Coverage   72.65%   73.24%   +0.58%     
==========================================
  Files          37       37              
  Lines        5559     5561       +2     
==========================================
+ Hits         4039     4073      +34     
+ Misses       1520     1488      -32     
Impacted Files Coverage Δ
c3/optimizers/modellearning.py 89.82% <ø> (-0.27%) :arrow_down:
c3/qiskit/c3_backend.py 92.92% <ø> (ø)
c3/generator/devices.py 67.99% <79.59%> (-0.05%) :arrow_down:
c3/experiment.py 77.04% <86.95%> (+1.69%) :arrow_up:
c3/c3objs.py 84.95% <91.66%> (-1.60%) :arrow_down:
c3/parametermap.py 93.16% <95.65%> (ø)
c3/signal/pulse.py 81.17% <96.00%> (+2.00%) :arrow_up:
c3/signal/gates.py 90.59% <96.42%> (-0.12%) :arrow_down:
c3/generator/generator.py 91.08% <100.00%> (-0.18%) :arrow_down:
c3/libraries/algorithms.py 55.00% <100.00%> (+0.17%) :arrow_up:
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0c2a80a...2815cb4. Read the comment docs.

lazyoracle commented 2 years ago

The failing notebook - two_qubit_entangling_gate.ipynb is due to a dead kernel which might indicate scalability/compute/memory issues. I checked by running it locally and ran into the same issue. I also tried reducing the maxfun for optimization to 10 instead of 25 and the kernel continues to crash.

lazyoracle commented 2 years ago

There seems to be some memory leak issues (or is this a feature of the new TF graph mode computation?) since the two_qubit_entangling_gate.ipynb now takes 12 GB (6x) more RAM and double the time compared to the version in dev. Additionally you get the following error:

Error Trace (Click to expand) ```python jupyter nbconvert --to notebook --inplace --execute examples/two_qubit_entangling_gate.ipynb [NbConvertApp] Converting notebook examples/two_qubit_entangling_gate.ipynb to notebook Traceback (most recent call last): File "/home/users/anurag/miniconda3/envs/c3-py39/bin/jupyter-nbconvert", line 8, in sys.exit(main()) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance app.start() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 369, in start self.convert_notebooks() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 541, in convert_notebooks self.convert_single_notebook(notebook_filename) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 506, in convert_single_notebook output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 435, in export_single_notebook output, resources = self.exporter.from_filename(notebook_filename, resources=resources) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 190, in from_filename return self.from_file(f, resources=resources, **kw) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 208, in from_file return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node nb_copy, resources = super().from_notebook_node(nb, resources, **kw) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 147, in from_notebook_node nb_copy, resources = self._preprocess(nb_copy, resources) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 334, in _preprocess nbc, resc = preprocessor(nbc, resc) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__ return self.preprocess(nb, resources) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 83, in preprocess self.preprocess_cell(cell, resources, index) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 104, in preprocess_cell cell = self.execute_cell(cell, index, store_history=True) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete return future.result() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/client.py", line 965, in async_execute_cell await self._check_raise_for_error(cell, cell_index, exec_reply) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/nbclient/client.py", line 862, in _check_raise_for_error raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content) nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell: ------------------ parameter_map.print_parameters() print(opt.current_best_goal) ------------------ --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Input In [29], in () ----> 1 parameter_map.print_parameters() 2 print(opt.current_best_goal) File ~/dev/c3/c3/parametermap.py:463, in ParameterMap.print_parameters(self, opt_map) 459 """ 460 Print current parameters to stdout. 461 """ 462 opt_map = self.get_opt_map(opt_map) --> 463 print(self.str_parameters(opt_map)) File ~/dev/c3/c3/parametermap.py:450, in ParameterMap.str_parameters(self, opt_map) 448 key = par_id 449 par = self._pars[key] --> 450 ret.append(f"{key:38}: {par}\n") 451 if len(equiv_ids) > 1: 452 for eid in equiv_ids[1:]: File ~/dev/c3/c3/c3objs.py:225, in Quantity.__str__(self) 224 def __str__(self): --> 225 val = self.numpy() 226 ret = "" 227 for entry in np.nditer(val): File ~/dev/c3/c3/c3objs.py:245, in Quantity.numpy(self) 241 """ 242 Return the value of this quantity as numpy. 243 """ 244 # TODO should be removed to be consistent with get_value --> 245 return self.get_value().numpy() / self.pref File ~/dev/c3/c3/c3objs.py:256, in Quantity.get_value(self) 247 def get_value(self) -> tf.Tensor: 248 """ 249 Return the value of this quantity as tensorflow. 250 (...) 254 dtype: tf.dtypes 255 """ --> 256 return self.scale * (self.value + 1) / 2 + self.offset File ~/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:153, in filter_traceback..error_handler(*args, **kwargs) 151 except Exception as e: 152 filtered_tb = _process_traceback_frames(e.__traceback__) --> 153 raise e.with_traceback(filtered_tb) from None 154 finally: 155 del filtered_tb File ~/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 52 try: 53 ctx.ensure_initialized() ---> 54 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 55 inputs, attrs, num_outputs) 56 except core._NotOkStatusException as e: 57 if name is not None: TypeError: is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it. Please see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information. was defined here: File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance app.start() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 677, in start self.io_loop.start() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start self.asyncio_loop.run_forever() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 596, in run_forever self._run_once() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once handle._run() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 473, in dispatch_queue await self.process_one() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 462, in process_one await dispatch(*args) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 369, in dispatch_shell await result File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 664, in execute_request reply_content = await reply_content File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 355, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 532, in run_cell return super().run_cell(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2863, in run_cell result = self._run_cell( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2909, in _run_cell return runner(coro) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3106, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3309, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3369, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_3537389/4248971210.py", line 3, in opt.optimize_controls() File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 147, in optimize_controls self.algorithm( File "/home/users/anurag/dev/c3/c3/libraries/algorithms.py", line 418, in lbfgs return minimize( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_minimize.py", line 681, in minimize res = _minimize_lbfgsb(fun, x0, args, jac, bounds, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_lbfgsb_py.py", line 308, in _minimize_lbfgsb sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_optimize.py", line 263, in _prepare_scalar_function sf = ScalarFunction(fun, x0, args, grad, hess, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in __init__ self._update_fun() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun self._update_fun_impl() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun self.f = fun_wrapped(self.x) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped fx = fun(np.copy(x), *args) File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 286, in fct_to_min_autograd goal, grad = self.goal_run_with_grad(current_params) File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 208, in goal_run_with_grad goal = self.goal_run(current_params) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 915, in __call__ result = self._call(*args, **kwds) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 963, in _call self._initialize(args, kwds, add_initializers_to=initializers) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 785, in _initialize self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 2983, in _get_concrete_function_internal_garbage_collected graph_function, _ = self._maybe_define_function(args, kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3292, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3130, in _create_graph_function func_graph_module.func_graph_from_py_func( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1161, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 677, in wrapped_fn out = weak_wrapped_fn().__wrapped__(*args, **kwds) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3831, in bound_method_wrapper return wrapped_fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1136, in autograph_handler return autograph.converted_call( File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 174, in goal_run self.pmap.set_parameters_scaled(current_params) File "/home/users/anurag/dev/c3/c3/parametermap.py", line 364, in set_parameters_scaled for equiv_ids in opt_map: File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt _py_for_stmt(iter_, extra_test, body, None, None) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt body(target) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body original_body(protected_iter) File "/home/users/anurag/dev/c3/c3/parametermap.py", line 367, in set_parameters_scaled for par_id in equiv_ids: File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt _py_for_stmt(iter_, extra_test, body, None, None) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt body(target) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body original_body(protected_iter) File "/home/users/anurag/dev/c3/c3/parametermap.py", line 371, in set_parameters_scaled par.set_opt_value(values[val_indx : val_indx + par_len]) File "/home/users/anurag/dev/c3/c3/c3objs.py", line 318, in set_opt_value self.value = tf.acos(bound_val) / np.pi * 2 - 1 File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 1406, in binary_op_wrapper return func(x, y, name=name) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 548, in subtract return gen_math_ops.sub(x, y, name) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/gen_math_ops.py", line 11139, in sub _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 693, in _create_op_internal return super(FuncGraph, self)._create_op_internal( # pylint: disable=protected-access File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal ret = Operation( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in __init__ self._traceback = tf_stack.extract_stack_for_node(self._c_op) The tensor cannot be accessed from here, because it was defined in FuncGraph(name=goal_run, id=140493482171936), which is out of scope. TypeError: is out of scope and cannot be used here. Use return values, explicit Python locals or TensorFlow collections to access it. Please see https://www.tensorflow.org/guide/function#all_outputs_of_a_tffunction_must_be_return_values for more information. was defined here: File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel_launcher.py", line 16, in app.launch_new_instance() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance app.start() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 677, in start self.io_loop.start() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start self.asyncio_loop.run_forever() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 596, in run_forever self._run_once() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once handle._run() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 473, in dispatch_queue await self.process_one() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 462, in process_one await dispatch(*args) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 369, in dispatch_shell await result File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 664, in execute_request reply_content = await reply_content File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 355, in do_execute res = shell.run_cell(code, store_history=store_history, silent=silent) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 532, in run_cell return super().run_cell(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2863, in run_cell result = self._run_cell( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2909, in _run_cell return runner(coro) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner coro.send(None) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3106, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3309, in run_ast_nodes if await self.run_code(code, result, async_=asy): File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3369, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "/tmp/ipykernel_3537389/4248971210.py", line 3, in opt.optimize_controls() File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 147, in optimize_controls self.algorithm( File "/home/users/anurag/dev/c3/c3/libraries/algorithms.py", line 418, in lbfgs return minimize( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_minimize.py", line 681, in minimize res = _minimize_lbfgsb(fun, x0, args, jac, bounds, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_lbfgsb_py.py", line 308, in _minimize_lbfgsb sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_optimize.py", line 263, in _prepare_scalar_function sf = ScalarFunction(fun, x0, args, grad, hess, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in __init__ self._update_fun() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun self._update_fun_impl() File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun self.f = fun_wrapped(self.x) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped fx = fun(np.copy(x), *args) File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 286, in fct_to_min_autograd goal, grad = self.goal_run_with_grad(current_params) File "/home/users/anurag/dev/c3/c3/optimizers/optimizer.py", line 208, in goal_run_with_grad goal = self.goal_run(current_params) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 915, in __call__ result = self._call(*args, **kwds) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 963, in _call self._initialize(args, kwds, add_initializers_to=initializers) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 785, in _initialize self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 2983, in _get_concrete_function_internal_garbage_collected graph_function, _ = self._maybe_define_function(args, kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3292, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3130, in _create_graph_function func_graph_module.func_graph_from_py_func( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1161, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 677, in wrapped_fn out = weak_wrapped_fn().__wrapped__(*args, **kwds) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3831, in bound_method_wrapper return wrapped_fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1136, in autograph_handler return autograph.converted_call( File "/home/users/anurag/dev/c3/c3/optimizers/optimalcontrol.py", line 174, in goal_run self.pmap.set_parameters_scaled(current_params) File "/home/users/anurag/dev/c3/c3/parametermap.py", line 364, in set_parameters_scaled for equiv_ids in opt_map: File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt _py_for_stmt(iter_, extra_test, body, None, None) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt body(target) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body original_body(protected_iter) File "/home/users/anurag/dev/c3/c3/parametermap.py", line 367, in set_parameters_scaled for par_id in equiv_ids: File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 440, in for_stmt _py_for_stmt(iter_, extra_test, body, None, None) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 489, in _py_for_stmt body(target) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 455, in protected_body original_body(protected_iter) File "/home/users/anurag/dev/c3/c3/parametermap.py", line 371, in set_parameters_scaled par.set_opt_value(values[val_indx : val_indx + par_len]) File "/home/users/anurag/dev/c3/c3/c3objs.py", line 318, in set_opt_value self.value = tf.acos(bound_val) / np.pi * 2 - 1 File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 1406, in binary_op_wrapper return func(x, y, name=name) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler return fn(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py", line 548, in subtract return gen_math_ops.sub(x, y, name) File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/ops/gen_math_ops.py", line 11139, in sub _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 693, in _create_op_internal return super(FuncGraph, self)._create_op_internal( # pylint: disable=protected-access File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal ret = Operation( File "/home/users/anurag/miniconda3/envs/c3-py39/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in __init__ self._traceback = tf_stack.extract_stack_for_node(self._c_op) The tensor cannot be accessed from here, because it was defined in FuncGraph(name=goal_run, id=140493482171936), which is out of scope. jupyter nbconvert --to notebook --inplace --execute 835.31s user 11.63s system 99% cpu 14:10.59 total ```
nwittler commented 2 years ago

Some increased memory requirement is expected but I don't have the experience on how much is reasonable here. I suspect the response function module to be a major cause of trouble. Removing it from the example chains might improve things. I'll check.

lgtm-com[bot] commented 2 years ago

This pull request introduces 1 alert when merging 2815cb4669b5251a5d1c36a20009607869153e3a into 52d0a12d568b6f12e24d87434459a37550fa901b - view on LGTM.com

new alerts: