Segmentation fault in WSL

[nnn911]

Running the OVITO Pro (v3.10.3) VisRTX renderer inside an Ubuntu WSL we get a Segmentation Fault in the VisRTX module crashing the application. While we don't expect / require VisRTX to work inside WSL it would be nice to catch the error and prevent the crash.

Should we test for VisRTX requirements before rendering thereby preventing VisRTX from starting or should this be handled inside VisRTX?

OVITO Pro Build Info

• VisRTX commit: b8e8ab80a939186dd032b7cad9c68071b84da7ac
• OptiX 7.6
• Cuda Toolkit v12.2

System Information

• Windows 11
• WSL:

  Default Distribution: Ubuntu-22.04
  Default Version: 2

• WSL Ubuntu:

  Ubuntu 22.04.4 LTS (GNU/Linux 5.15.133.1-microsoft-standard-WSL2 x86_64)

[jeffamstutz]

This was definitely a bug in VisRTX -- initialization wasn't actually being stopped if optixInit() failed, which caused all kinds of UB. This is addressed now in aa2181 on the next_release branch. I tested this by removing libnvidia-rtcore from the environment, which causes optixInit() to fail, where it will report a fatal error and then stop initialization. Note that if ANARI_SEVERITY_FATAL_ERROR occurs, then the device should be immediately released and not used further. However, I think it's reasonable just to check this after the device is initialized and doesn't have to be done after every ANARI call in the app.

Please reopen if you continue to have issues, thanks!