search

NVlabs / sionna

Sionna: An Open-Source Library for Next-Generation Physical Layer Research
https://nvlabs.github.io/sionna
Other
748 stars 214 forks source link

InvalidArgumentError: Exception encountered when calling layer 'model' (type Model). #255

Closed mahmoud-atwi closed 10 months ago

mahmoud-atwi commented 11 months ago

running "MIMO_OFDM_Transmissions_over_CDL.ipynb" notebook is returning error at the cdl_model for loop

python: 3.10.12 sionna: 0.15.1 TF: 2.14.0

for cdl_model in UL_SIMS["cdl_model"]:

    model = Model(domain=UL_SIMS["domain"],
                  direction=UL_SIMS["direction"],
                  cdl_model=cdl_model,
                  delay_spread=UL_SIMS["delay_spread"],
                  perfect_csi=UL_SIMS["perfect_csi"],
                  speed=UL_SIMS["speed"],
                  cyclic_prefix_length=UL_SIMS["cyclic_prefix_length"],
                  pilot_ofdm_symbol_indices=UL_SIMS["pilot_ofdm_symbol_indices"])

    ber, bler = sim_ber(model,
                        UL_SIMS["ebno_db"],
                        batch_size=256,
                        max_mc_iter=100,
                        num_target_block_errors=1000)

    UL_SIMS["ber"].append(list(ber.numpy()))
    UL_SIMS["bler"].append(list(bler.numpy()))

UL_SIMS["duration"] = time.time() - start

Error:

----------------------------------------------------------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
[<ipython-input-31-a353aeee9ff7>](https://xj9dxa1piq-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231106-060121_RC00_579805961#) in <cell line: 18>()
     27                   pilot_ofdm_symbol_indices=UL_SIMS["pilot_ofdm_symbol_indices"])
     28 
---> 29     ber, bler = sim_ber(model,
     30                         UL_SIMS["ebno_db"],
     31                         batch_size=256,

2 frames
[/usr/local/lib/python3.10/dist-packages/sionna/utils/misc.py](https://xj9dxa1piq-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231106-060121_RC00_579805961#) in sim_ber(mc_fun, ebno_dbs, batch_size, max_mc_iter, soft_estimates, num_target_bit_errors, num_target_block_errors, early_stop, graph_mode, verbose, forward_keyboard_interrupt, dtype)
    631                 iter_count += 1
    632 
--> 633                 outputs = mc_fun(batch_size=batch_size, ebno_db=ebno_dbs[i])
    634 
    635                 # assume first and second return value is b and b_hat

[/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py](https://xj9dxa1piq-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231106-060121_RC00_579805961#) in error_handler(*args, **kwargs)
     68             # To get the full stack trace, call:
     69             # `tf.debugging.disable_traceback_filtering()`
---> 70             raise e.with_traceback(filtered_tb) from None
     71         finally:
     72             del filtered_tb

[/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/execute.py](https://xj9dxa1piq-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20231106-060121_RC00_579805961#) in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58         for t in inputs
     59     ]
---> 60     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     61                                         inputs, attrs, num_outputs)
     62   except core._NotOkStatusException as e:

InvalidArgumentError: Exception encountered when calling layer 'model' (type Model).

Detected unsupported operations when trying to compile graph ldpc5g_decoder_1_while_body_116508_const_0[] on XLA_GPU_JIT: RaggedRange (No registered 'RaggedRange' OpKernel for XLA_GPU_JIT devices compatible with node {{node ldpc5g_decoder_1/while/RaggedWhere/RaggedRange}}){{node ldpc5g_decoder_1/while/RaggedWhere/RaggedRange}}
The op is created at: 
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
File "/usr/local/lib/python3.10/dist-packages/colab_kernel_launcher.py", line 37, in <module>
File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 992, in launch_instance
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelapp.py", line 619, in start
File "/usr/local/lib/python3.10/dist-packages/tornado/platform/asyncio.py", line 195, in start
File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 685, in <lambda>
File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 738, in _run_callback
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 825, in inner
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 786, in run
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 377, in dispatch_queue
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 250, in wrapper
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 748, in __init__
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 786, in run
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 361, in process_one
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 261, in dispatch_shell
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 539, in execute_request
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
File "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py", line 302, in do_execute
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 539, in run_cell
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 2975, in run_cell
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3030, in _run_cell
File "/usr/local/lib/python3.10/dist-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3257, in run_cell_async
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3473, in run_ast_nodes
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3553, in run_code
File "<ipython-input-31-a353aeee9ff7>", line 29, in <cell line: 18>
File "/usr/local/lib/python3.10/dist-packages/sionna/utils/misc.py", line 633, in sim_ber
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py", line 589, in __call__
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/base_layer.py", line 1149, in __call__
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
File "<ipython-input-30-31a1628a2ed7>", line 228, in call
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/base_layer.py", line 1149, in __call__
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 1382, in call
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 1383, in call
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 1002, in call
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 980, in dec_iter
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 654, in _cn_update_phi
    tf2xla conversion failed while converting ldpc5g_decoder_1_while_body_116508_const_0[]. Run with TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to obtain a dump of the compiled functions.

Stack trace for op definition: 
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
File "/usr/local/lib/python3.10/dist-packages/colab_kernel_launcher.py", line 37, in <module>
File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 992, in launch_instance
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelapp.py", line 619, in start
File "/usr/local/lib/python3.10/dist-packages/tornado/platform/asyncio.py", line 195, in start
File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 685, in <lambda>
File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 738, in _run_callback
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 825, in inner
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 786, in run
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 377, in dispatch_queue
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 250, in wrapper
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 748, in __init__
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 786, in run
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 361, in process_one
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 261, in dispatch_shell
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 539, in execute_request
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
File "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py", line 302, in do_execute
File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 539, in run_cell
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 2975, in run_cell
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3030, in _run_cell
File "/usr/local/lib/python3.10/dist-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3257, in run_cell_async
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3473, in run_ast_nodes
File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3553, in run_code
File "<ipython-input-31-a353aeee9ff7>", line 29, in <cell line: 18>
File "/usr/local/lib/python3.10/dist-packages/sionna/utils/misc.py", line 633, in sim_ber
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py", line 589, in __call__
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/base_layer.py", line 1149, in __call__
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
File "<ipython-input-30-31a1628a2ed7>", line 228, in call
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
File "/usr/local/lib/python3.10/dist-packages/keras/src/engine/base_layer.py", line 1149, in __call__
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 1382, in call
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 1383, in call
File "/usr/local/lib/python3.10/dist-packages/sionna/fec/ldpc/decoding.py", line 1002, in call

     [[ldpc5g_decoder_1/while]]
    tf2xla conversion failed while converting __inference_call_118062[_XlaMustCompile=true,config_proto=8589078909834744431,executor_type=11160318154034397263]. Run with TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to obtain a dump of the compiled functions. [Op:__inference_call_118062]

Call arguments received by layer 'model' (type Model):
  • batch_size=tf.Tensor(shape=(), dtype=int32)
  • ebno_db=tf.Tensor(shape=(), dtype=float32)
faycalaa commented 11 months ago

Hi,

I cannot reproduce the error with Python 3.10.12 and TF 2.13.0. I haven't tested with TF 2.14 yet. It appears that the error may come from the use of XLA. Could you try running the code in graph mode without XLA?

SebastianCa commented 10 months ago

Hi, just a quick update: we can confirm an issue with the LDPC decoder in XLA for TF 2.14. A workaround is to downgrade to TF 2.13 or to disable XLA for the simulations.

faycalaa commented 10 months ago

Fixed by v0.16.