NVlabs / sionna

Sionna: An Open-Source Library for Next-Generation Physical Layer Research
https://nvlabs.github.io/sionna
Other
819 stars 232 forks source link

Kernel died on CoverageMap with 1080Ti GPU ad Docker #650

Open lillidith opened 2 weeks ago

lillidith commented 2 weeks ago

Hi, I set up the Sionna environment with docker using the associated makefile. The Notebook Sionna_Ray_Tracing_Coverage_Map.ipynb con load the libraries, and the GPU is correctly configured. The scene is loaded successfully, but the kernel died when I computed:

cm = scene.coverage_map(max_depth=2,           # Maximum number of ray scene interactions
                        num_samples=int(2e6), # If you increase: less noise, but more memory required
                        cm_cell_size=(5, 5),   # Resolution of the coverage map
                        cm_center=[0, 0, 0],   # Center of the coverage map
                        cm_size=[400, 400],    # Total size of the coverage map
                        cm_orientation=[0, 0, 0]) # Orientation of the coverage map, e.g., could be also vertical

The same configuration works fine and smooth on Server with RTX A5000 and A100 GPUs.

Server configuration

sudo apt-get update
sudo apt-get install -y cuda-drivers-560
sudo apt-get -y install cuda-toolkit-12-6
sudo usermod -aG docker myuser
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
merlinND commented 2 weeks ago

Hello @lillidith,

What error message was shown in the terminal when the kernel died? The most likely cause is running out of memory. You could try reducing num_samples and trying again.

Fedomer commented 2 weeks ago

Same issue on a workstation with 1080Ti 11Gb:

scene = load_scene() # Load empty scene

scene.tx_array = PlanarArray(num_rows=1,
                             num_cols=1,
                             vertical_spacing=0.5,  # relative to wavelength
                             horizontal_spacing=0.5,  # relative to wavelength
                             pattern="iso",
                             polarization="V")
scene.rx_array = scene.tx_array

tx0 = Transmitter(name='tx0',
                  position=[15, -10, 20],
                  orientation=[np.pi*5/6, 0, 0],
                  power_dbm=10)
scene.add(tx0)

cm = scene.coverage_map(max_depth=5,           # Maximum number of ray scene interactions
                        num_samples=int(100), # If you increase: less noise, but more memory required
                        cm_cell_size=(5, 5),   # Resolution of the coverage map
                        cm_center=[0, 0, 0],   # Center of the coverage map
                        cm_size=[50, 50],    # Total size of the coverage map
                        cm_orientation=[0, 0, 0])

.... of corse all seem perfect in workstation with A100 40Gb but it's not a memory pb ( _numsamples=int(100) )! Compute capability? Or incompatible GPU ? I dont find a minimun requirement on hardware in the Sionna Doc site.

merlinND commented 2 weeks ago

What error message was shown in the terminal when the kernel died?

merlinND commented 2 weeks ago

By the way, as a workaround you can set the environment variable CUDA_VISIBLE_DEVICE="" before launching the Jupyter server to fall back on the CPU backend. Depending on your hardware, it could be a viable alternative.

Fedomer commented 21 hours ago

it's seem that with 2080Ti with 11 Gb it works so the the 1080Ti 11Gb card it's to old for sionna.