search

keras-team / autokeras

AutoML library for deep learning
http://autokeras.com/
Apache License 2.0
9.15k stars 1.4k forks source link

WARNING:tensorflow:Large unrolled loop detected. #1604

Open talhaanwarch opened 3 years ago

talhaanwarch commented 3 years ago

I am getting this weird log 


WARNING:tensorflow:Large unrolled loop detected. Did you mean to use a TF loop? The following ops were created after iteration 3002: (<tf.Operation 'multi_category_encoding/AsString_3000' type=AsString>, <tf.Operation 'multi_category_encoding/string_lookup_3000/None_lookup_table_find/LookupTableFindV2/table_handle' type=Placeholder>, <tf.Operation 'multi_category_encoding/string_lookup_3000/None_lookup_table_find/LookupTableFindV2/default_value' type=Const>, <tf.Operation 'multi_category_encoding/string_lookup_3000/None_lookup_table_find/LookupTableFindV2' type=LookupTableFindV2>, <tf.Operation 'multi_category_encoding/Cast_3000' type=Cast>)
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/reference/common_errors.md#warning-large-unrolled-loop-detected
Location:
  File "C:\Anaconda3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,

  File "C:\Anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)

  File "d:\env\paindata\lib\site-packages\ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()

  File "d:\env\paindata\lib\site-packages\traitlets\config\application.py", line 845, in launch_instance
    app.start()

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelapp.py", line 667, in start
    self.io_loop.start()

  File "d:\env\paindata\lib\site-packages\tornado\platform\asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()

  File "C:\Anaconda3\lib\asyncio\base_events.py", line 570, in run_forever
    self._run_once()

  File "C:\Anaconda3\lib\asyncio\base_events.py", line 1859, in _run_once
    handle._run()

  File "C:\Anaconda3\lib\asyncio\events.py", line 81, in _run
    self._context.run(self._callback, *self._args)

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 456, in dispatch_queue
    await self.process_one()

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 445, in process_one
    await dispatch(*args)

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 352, in dispatch_shell
    await result

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 647, in execute_request
    reply_content = await reply_content

  File "d:\env\paindata\lib\site-packages\ipykernel\ipkernel.py", line 345, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)

  File "d:\env\paindata\lib\site-packages\ipykernel\zmqshell.py", line 532, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 2898, in run_cell
    result = self._run_cell(

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 2944, in _run_cell
    return runner(coro)

  File "d:\env\paindata\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 3169, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 3361, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "C:\Users\TAC\AppData\Local\Temp/ipykernel_17400/814657760.py", line 2, in <module>
    clf.fit(train.astype('float32'), y_train,verbose=2, epochs=1)

  File "d:\env\paindata\lib\site-packages\autokeras\tasks\structured_data.py", line 315, in fit
    super().fit(

  File "d:\env\paindata\lib\site-packages\autokeras\tasks\structured_data.py", line 134, in fit
    super().fit(

  File "d:\env\paindata\lib\site-packages\autokeras\auto_model.py", line 278, in fit
    self.tuner.search(

  File "d:\env\paindata\lib\site-packages\autokeras\engine\tuner.py", line 185, in search
    self.hypermodel.build(hp)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 127, in build
    model = self.hypermodel.build(hp)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\autokeras\graph.py", line 250, in build
    outputs = block.build(hp, inputs=temp_inputs)

  File "d:\env\paindata\lib\site-packages\autokeras\engine\block.py", line 38, in _build_wrapper
    return super()._build_wrapper(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\autokeras\blocks\wrapper.py", line 249, in build
    output_node = block.build(hp, output_node)

  File "d:\env\paindata\lib\site-packages\autokeras\engine\block.py", line 38, in _build_wrapper
    return super()._build_wrapper(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\autokeras\blocks\preprocessing.py", line 283, in build
    return keras_layers.MultiCategoryEncoding(encoding)(input_node)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 925, in __call__
    return self._functional_construction_call(inputs, args, kwargs,

  File "d:\env\paindata\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1117, in _functional_construction_call
    outputs = call_fn(cast_inputs, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 255, in wrapper
    return converted_call(f, args, kwargs, options=options)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 596, in converted_call
    result = converted_f(*effective_args, **kwargs)

  File "C:\Users\TAC\AppData\Local\Temp\tmpb74g7o_n.py", line 41, in tf__call
    ag__.for_stmt(ag__.converted_call(ag__.ld(zip), (ag__.ld(split_inputs), ag__.ld(self).encoding_layers), None, fscope), None, loop_body, get_state_1, set_state_1, (), {'iterate_names': '(input_node, encoding_layer)'})

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 368, in for_stmt
    _py_for_stmt(iter_, extra_test, body, None, None)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 397, in _py_for_stmt
    body(target)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 384, in protected_body
    after_iteration()

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 834, in after_iteration
    did_warn = self._verify_inefficient_unroll()

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 817, in _verify_inefficient_unroll
    '', self.iterations, new_ops, '\n'.join(traceback.format_stack()))

WARNING: Large unrolled loop detected. Did you mean to use a TF loop? The following ops were created after iteration 3002: (<tf.Operation 'multi_category_encoding/AsString_3000' type=AsString>, <tf.Operation 'multi_category_encoding/string_lookup_3000/None_lookup_table_find/LookupTableFindV2/table_handle' type=Placeholder>, <tf.Operation 'multi_category_encoding/string_lookup_3000/None_lookup_table_find/LookupTableFindV2/default_value' type=Const>, <tf.Operation 'multi_category_encoding/string_lookup_3000/None_lookup_table_find/LookupTableFindV2' type=LookupTableFindV2>, <tf.Operation 'multi_category_encoding/Cast_3000' type=Cast>)
See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/reference/common_errors.md#warning-large-unrolled-loop-detected
Location:
  File "C:\Anaconda3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,

  File "C:\Anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)

  File "d:\env\paindata\lib\site-packages\ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()

  File "d:\env\paindata\lib\site-packages\traitlets\config\application.py", line 845, in launch_instance
    app.start()

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelapp.py", line 667, in start
    self.io_loop.start()

  File "d:\env\paindata\lib\site-packages\tornado\platform\asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()

  File "C:\Anaconda3\lib\asyncio\base_events.py", line 570, in run_forever
    self._run_once()

  File "C:\Anaconda3\lib\asyncio\base_events.py", line 1859, in _run_once
    handle._run()

  File "C:\Anaconda3\lib\asyncio\events.py", line 81, in _run
    self._context.run(self._callback, *self._args)

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 456, in dispatch_queue
    await self.process_one()

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 445, in process_one
    await dispatch(*args)

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 352, in dispatch_shell
    await result

  File "d:\env\paindata\lib\site-packages\ipykernel\kernelbase.py", line 647, in execute_request
    reply_content = await reply_content

  File "d:\env\paindata\lib\site-packages\ipykernel\ipkernel.py", line 345, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)

  File "d:\env\paindata\lib\site-packages\ipykernel\zmqshell.py", line 532, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 2898, in run_cell
    result = self._run_cell(

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 2944, in _run_cell
    return runner(coro)

  File "d:\env\paindata\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 3169, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 3361, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):

  File "d:\env\paindata\lib\site-packages\IPython\core\interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "C:\Users\TAC\AppData\Local\Temp/ipykernel_17400/814657760.py", line 2, in <module>
    clf.fit(train.astype('float32'), y_train,verbose=2, epochs=1)

  File "d:\env\paindata\lib\site-packages\autokeras\tasks\structured_data.py", line 315, in fit
    super().fit(

  File "d:\env\paindata\lib\site-packages\autokeras\tasks\structured_data.py", line 134, in fit
    super().fit(

  File "d:\env\paindata\lib\site-packages\autokeras\auto_model.py", line 278, in fit
    self.tuner.search(

  File "d:\env\paindata\lib\site-packages\autokeras\engine\tuner.py", line 185, in search
    self.hypermodel.build(hp)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 127, in build
    model = self.hypermodel.build(hp)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\autokeras\graph.py", line 250, in build
    outputs = block.build(hp, inputs=temp_inputs)

  File "d:\env\paindata\lib\site-packages\autokeras\engine\block.py", line 38, in _build_wrapper
    return super()._build_wrapper(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\autokeras\blocks\wrapper.py", line 249, in build
    output_node = block.build(hp, output_node)

  File "d:\env\paindata\lib\site-packages\autokeras\engine\block.py", line 38, in _build_wrapper
    return super()._build_wrapper(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\keras_tuner\engine\hypermodel.py", line 84, in _build_wrapper
    return self._build(hp, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\autokeras\blocks\preprocessing.py", line 283, in build
    return keras_layers.MultiCategoryEncoding(encoding)(input_node)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 925, in __call__
    return self._functional_construction_call(inputs, args, kwargs,

  File "d:\env\paindata\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1117, in _functional_construction_call
    outputs = call_fn(cast_inputs, *args, **kwargs)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 255, in wrapper
    return converted_call(f, args, kwargs, options=options)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 596, in converted_call
    result = converted_f(*effective_args, **kwargs)

  File "C:\Users\TAC\AppData\Local\Temp\tmpb74g7o_n.py", line 41, in tf__call
    ag__.for_stmt(ag__.converted_call(ag__.ld(zip), (ag__.ld(split_inputs), ag__.ld(self).encoding_layers), None, fscope), None, loop_body, get_state_1, set_state_1, (), {'iterate_names': '(input_node, encoding_layer)'})

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 368, in for_stmt
    _py_for_stmt(iter_, extra_test, body, None, None)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 397, in _py_for_stmt
    body(target)

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 384, in protected_body
    after_iteration()

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 834, in after_iteration
    did_warn = self._verify_inefficient_unroll()

  File "d:\env\paindata\lib\site-packages\tensorflow\python\autograph\operators\control_flow.py", line 817, in _verify_inefficient_unroll
    '', self.iterations, new_ops, '\n'.join(traceback.format_stack()))

Search: Running Trial #1

Hyperparameter    |Value             |Best Value So Far 
structured_data...|True              |?                 
structured_data...|2                 |?                 
structured_data...|False             |?                 
structured_data...|0                 |?                 
structured_data...|32                |?                 
structured_data...|32                |?                 
classification_...|0                 |?                 
optimizer         |adam              |?                 
learning_rate     |0.001             |?
JonasH22 commented 3 years ago

I'm receiving the same warning, when using the fit function of the StructuredDataClassifier on rather high-dimensional, sparse tabular data (e.g. of shape 5,000x3,500). Autokeras takes a lot of time before starting to train a model in those instances. Is there a way to better process or load sparse & high-dimensional data with Autokeras?

rwst commented 1 year ago

Same here with a random forest and 25k input one-hot tensors of dimension (>130,000, 2). Yes, this needs a 128GB RAM machine.