UnknownError: {{function_node __wrapped__FloorMod_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:FloorMod]

[nurullahsevim]

Every time I try to get a coverage map, I get the following error: UnknownError: {{function_node wrappedFloorModdevice/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:FloorMod] I am on Windows Native, using TF 2.10.1, Sionna 0.14.

Things work ok when I try to compute paths. I only observed this problem with coverage_map() function.

And this is the full error tree:

---------------------------------------------------------------------------
UnknownError                              Traceback (most recent call last)
Cell In[1], line 28
     25 tx.look_at([40,80,1.5])
     27 # Compute coverage map
---> 28 cm = scene.coverage_map(max_depth=8)
     30 # Show coverage map
     31 cm.show()

File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\scene.py:908, in Scene.coverage_map(self, rx_orientation, max_depth, cm_center, cm_orientation, cm_size, cm_cell_size, combining_vec, precoding_vec, num_samples, seed)
    903     seed = tf.random.uniform(shape=(), maxval=0x7FFFFFFF,
    904                              dtype=tf.int32)
    906 # Compute the coverage map using the solver
    907 # [num_sources, num_cells_x, num_cells_y]
--> 908 output = self._solver_cm(max_depth=max_depth,
    909                          rx_orientation=rx_orientation,
    910                          cm_center=cm_center,
    911                          cm_orientation=cm_orientation,
    912                          cm_size=cm_size,
    913                          cm_cell_size=cm_cell_size,
    914                          combining_vec=combining_vec,
    915                          precoding_vec=precoding_vec,
    916                          num_samples=num_samples,
    917                          seed=seed)
    919 return output

File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\solver_cm.py:138, in SolverCoverageMap.__call__(self, max_depth, rx_orientation, cm_center, cm_orientation, cm_size, cm_cell_size, combining_vec, precoding_vec, num_samples, seed)
    129 # Shoot and bounce
    130 # hit : [max_depth+1, num_tx, samples_per_tx]
    131 # hit_point : [max_depth+1, num_tx, samples_per_tx, 3]
    132 # objects : [max_depth, num_tx, samples_per_tx]
    133 # int_point : [max_depth, num_tx, samples_per_tx, 3]
    134 rays = self._shoot_and_bounce(cm_center, cm_orientation, cm_size,
    135                               max_depth,
    136                               num_samples, seed)
--> 138 cm = self._rays_2_coverage_map(
    139                     cm_center, cm_orientation, cm_size, cm_cell_size,
    140                     rx_orientation,
    141                     combining_vec, precoding_vec,
    142                     *rays,
    143                     etas)
    144 return cm

File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\solver_cm.py:975, in SolverCoverageMap._rays_2_coverage_map(self, cm_center, cm_orientation, cm_size, cm_cell_size, rx_orientation, combining_vec, precoding_vec, hit, hit_point, objects, int_point, normals, relative_permittivity)
    973     theta_t_0, phi_t_0 = theta_t, phi_t
    974 # [num_tx, samples_per_tx]
--> 975 ray_contrib = self._ray_contribution(mat_t_hit,
    976                                      k_i_0, theta_t_0, phi_t_0,
    977                                      last_step_norm, theta_r, phi_r,
    978                                      rx_orientation,
    979                                      combining_vec, precoding_vec)
    980 # Zeros the contribution of rays that didn't hit the coverage map
    981 # [num_tx, samples_per_tx]
    982 ray_contrib = tf.where(hit_, ray_contrib,
    983                        tf.zeros_like(ray_contrib))

File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\solver_cm.py:636, in SolverCoverageMap._ray_contribution(self, mat_t, k_t, theta_t, phi_t, k_r, theta_r, phi_r, rx_orientation, combining_vec, precoding_vec)
    633 tx_ant_fields_hat = []
    634 for pattern in tx_patterns:
    635     # [num_tx, samples_per_tx, 2]
--> 636     tx_ant_f = tf.stack(pattern(theta_prime_t, phi_prime_t), axis=-1)
    637     tx_ant_fields_hat.append(tx_ant_f)
    639 rx_ant_fields_hat = []

File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\antenna.py:122, in Antenna.pattern_with_slant_angle.<locals>.<lambda>(theta, phi)
    120 def pattern_with_slant_angle(self, pattern, slant_angle):
    121     """Applies slant angle to antenna pattern"""
--> 122     return lambda theta, phi: pattern(theta, phi, slant_angle, self._dtype)

File ~\anaconda3\envs\RLHF\lib\site-packages\sionna\rt\antenna.py:500, in tr38901_pattern(theta, phi, slant_angle, dtype)
    497 phi = tf.cast(phi, rdtype)
    499 # Wrap phi to [-PI,PI]
--> 500 phi = tf.math.floormod(phi+PI, 2*PI)-PI
    502 if not theta.shape==phi.shape:
    503     raise ValueError("theta and phi must have the same shape.")

File ~\anaconda3\envs\RLHF\lib\site-packages\tensorflow\python\ops\gen_math_ops.py:4002, in floor_mod(x, y, name)
   4000   return _result
   4001 except _core._NotOkStatusException as e:
-> 4002   _ops.raise_from_not_ok_status(e, name)
   4003 except _core._FallbackException:
   4004   pass

File ~\anaconda3\envs\RLHF\lib\site-packages\tensorflow\python\framework\ops.py:7209, in raise_from_not_ok_status(e, name)
   7207 def raise_from_not_ok_status(e, name):
   7208   e.message += (" name: " + name if name is not None else "")
-> 7209   raise core._status_to_exception(e) from None

UnknownError: {{function_node __wrapped__FloorMod_device_/job:localhost/replica:0/task:0/device:GPU:0}} JIT compilation failed. [Op:FloorMod]

[faycalaa]

Hi,

Are you using XLA? If so, it would explain why you get this error, as Scene.coverage_map() does not work with XLA.

[nurullahsevim]

Thanks for the response. I did not explicitly use XLA. Maybe it is activated in Sionna library? How can I disable it?

[faycalaa]

Sionna does not activate XLA. Could you try with the latest version of Sionna (0.16.0)?

[nurullahsevim]

Upgraded Sionna to 0.16 but I got the same error. I tried to put sionna.Config.xla_compat = False before coverage map function but it did not solve the issue.

[faycalaa]

Could you share a minimal code snippet that reproduces the issue?

[nurullahsevim]

Sure. I get the error after executing the last line. The execution runs for a while but outputs an error eventually.

import sionna
import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import time

from sionna.rt import load_scene, Transmitter, Receiver, PlanarArray, Camera

from sionna.channel import cir_to_ofdm_channel, subcarrier_frequencies, OFDMChannel, ApplyOFDMChannel, CIRDataset
from sionna.nr import PUSCHConfig, PUSCHTransmitter, PUSCHReceiver
from sionna.utils import compute_ber, ebnodb2no, PlotBER
from sionna.ofdm import KBestDetector, LinearDetector
from sionna.mimo import StreamManagement

gpus = tf.config.list_physical_devices('GPU')
if gpus:
    try:
        tf.config.experimental.set_memory_growth(gpus[0], True)
    except RuntimeError as e:
        print(e)

tf.get_logger().setLevel('ERROR')
tf.random.set_seed(1)  # Set global random seed for reproducibility

scene = load_scene(sionna.rt.scene.munich)
scene.preview()

scene.tx_array = PlanarArray(num_rows=8,
                          num_cols=2,
                          vertical_spacing=0.7,
                          horizontal_spacing=0.5,
                          pattern="tr38901",
                          polarization="VH")

scene.rx_array = PlanarArray(num_rows=1,
                          num_cols=1,
                          vertical_spacing=0.5,
                          horizontal_spacing=0.5,
                          pattern="dipole",
                          polarization="cross")

tx = Transmitter(name="tx",
              position=[8.5,21,27],
              orientation=[0,0,0])
scene.add(tx)

rx = Receiver(name="rx",
           position=[45,90,1.5],
           orientation=[0,0,0])
scene.add(rx)

tx.look_at(rx)

print(scene.transmitters)
print(scene.receivers)

paths = scene.compute_paths(max_depth=5,
                            num_samples=1e6)

[jhoydis]

Hi, I cannot reproduce this error. Could you describe your full system configuration (OS, GPU, TF, Sionna, Python, etc.)? There is no coverage computation in this code snippet. You get the error in the compute_path call?

Could you upgrade to TF 2.13?

[nurullahsevim]

Yeah, actually it was also giving the same error in compute_path call. However, I somehow solved the issue. When I copied the "nvvm" folder from CUDA/vx.x to my working directory, I stopped getting the error. But, I have still no idea what the problem is. I am sure that I have cuda directory in the Path variables. If you have any idea on that (like what causes the problem) and a better solution than my work-around I will be glad. Otherwise I can close the issue.

[SebastianCa]

Closing as it appears to be an issue with the local CUDA installation.