Incorrect Target Architecture

[TwiggyDaniels]

I'm fairly certain my issue is due to the end of TensorFlow GPU support on native Windows, but maybe not and someone can help me out?

First, a quick error dump for you:

Critical Dr.Jit compiler failure: jit_optix_compile(): optixModuleGetCompilationState() indicates that the compilation did not complete succesfully. The module's compilation state is: 0x2363
Please see the PTX assembly listing and error message below:

.version 7.4
.target sm_87
.address_size 64

.const .align 8 .b8 params[168];

.entry __raygen__ee6bb8655c92deb30f65d833244aa862() {
    .reg.b8   %b <258>; .reg.b16 %w<258>; .reg.b32 %r<258>;
    .reg.b64  %rd<258>; .reg.f32 %f<258>; .reg.f64 %d<258>;
    .reg.pred %p <258>;

    call (%r0), _optix_get_launch_index_x, ();
    ld.const.u32 %r1, [params + 4];
    add.u32 %r0, %r0, %r1;

**...etc...**

COMPILE ERROR: Invalid PTX input: ptx2llvm-module-001: error: Failed to parse input PTX string
ptx2llvm-module-001, line 2; fatal   : Unsupported .target 'sm_87'
Cannot parse input PTX string

I am on Windows 10 (native) using an Nvidia 4090 with CUDA 11.8 and TensorFlow 2.10.1 with Sionna. I'm a bit clueless with anything CUDA but I believe it should be targeting sm_89 rather than sm_87.

I figure this might be from TensorFlow 2.10.1 not supporting CUDA 11.8 (but 11.7 seemed to work just fine before upgrading from a 3090). I would opt to use WSL2 but it seems that OptiX doesn't support it. I would go the WSL2 route but I am unsure if it is supported by OptiX yet?

[njroussel]

Hi @TwiggyDaniels

I think you need to update your GPU driver. You will find a list of valid (PTX version, CUDA version / driver version, target architectures) triplets here: https://docs.nvidia.com/cuda/parallel-thread-execution/#release-notes

Indeed the 4090 could target the sm_89 architecture, but PTX is forwards compatible (w.r.t GPU architecture) and should therefore not be an issue. FWIW, I'm also using a 4090 and have not run into this issue.

One important note: Dr.Jit does not use the CUDA runtime API, but instead uses the driver API. Hence why I think this is related to your driver version. In your setup the CUDA runtime installation is only needed by Tensorflow.

I indeed believe that WSL/WSL2 is still not supported by OptiX. I'd recommend you double-check this on their forums/documentaton before going down this route.

[njroussel]

An update:

I've ran into this same issue myself twice, after upgrading my drivers from v530 to v535. As far as I can tell, it seems like OptiX specifically doesn't support the sm_87 architecture anymore. Launching the same PTX through CUDA (without OptiX) is supported. For example:

import mitsuba as mi
import drjit as dr

dr.set_flag(dr.JitFlag.PrintIR, True) # Check that sm_87 is targeted with PTX ISA version 7.4
dr.set_flag(dr.JitFlag.ForceOptiX, True) # Launch kernel through OptiX rather than CUDA

mi.set_variant('cuda_ad_rgb')

a = dr.opaque(mi.Float, 3)
b = dr.opaque(mi.Float, 3)
print(a + b) # Launches a kernel

In addition, sm_87 only describes a single Jetson platform which doesn't seem to be supported by OptiX anyway. Finally, targeting older architectures like like sm_86, sm_50 works just fine.

I've pushed a fix to drjit-core: https://github.com/mitsuba-renderer/drjit-core/commit/25dd7a5cb96ee58d65cc1499f47de76f6140ff36 and propagated it to drjit. Until the next release, you will need to manually build the project to have this fix.

For completeness, @TwiggyDaniels could you give your GPU driver version?

[TwiggyDaniels]

Hi @njroussel,

I tried using v536 and v535, after rolling back to v528 from the triplets you linked and things are working again. Thanks for getting back to me and being thorough!

As an aside to anyone using Sionna that might find this issue: Setting the flags as recommended to run without OptiX doesn't seem to work, I figure Sionna is resetting them internally somewhere. Best to change driver versions or build from source.