[l4t-base] Building CUDA code in a Jetson Container on an x86 workstation

[trilorez]

It doesn't currently appear to be possible to compile CUDA code in a Jetson Container running on an x86 workstation.

For example, if I go through this example but attempt to compile samples/5_Simulations/nbody instead of samples/1_Utilities/deviceQuery, I get the error: /bin/sh: 1: cicc: not found.

Comparing containers, I see that on my device /usr/local/cuda contains many files not present on my workstation, including nvvm.

Is this a known issue and are there plans to make this possible?

[RenaudWasTaken]

Hello!

Sorry for the long delay. Jetson support is still pretty experimental and suject to a lot of change. Bbetter CUDA images is on the roadmap, unfortunately the team is currently focused on our GPU Operator work. We are more likely to handle this in the second part of the year.

Keeping open until we resolve this.

[trn84]

If I understand correctly there are merely the CUDA runtime libs in the docker image...If you copy the /usr/lib/cuda-10.0 folder from your native Jetson into your image to the same folder it should be possible to build everything on your x86 machine.

[IceTDrinker]

I have a use case where I need to do such a thing, does the copy of cuda from a jetson into the l4t-base based container work ?

[IceTDrinker]

My guess is it would not if there is something driver dependant as hinted in the documentation ?

[IceTDrinker]

Quick update, it seems copying the jetson cuda directory to my x86 PC and mounting it as a volume in the l4t-base container in /usr/local/cuda-10.0 works for the project I'm working on, basically compiling stuff does not seem to be an issue for my current project.

Obviously YMMV.

Cheers

[fh-igd-mueller-roemer]

We are more likely to handle this in the second part of the year.

Has there been any progress? Building via a Docker container and QEMU would be a lot more comfortable, but without cicc most programs cannot compile (and other missing files such as nvvm/libdevice, which CMake checks for to detect CUDA in the first place)

[trn84]

We are more likely to handle this in the second part of the year.

Has there been any progress? Building via a Docker container and QEMU would be a lot more comfortable, but without cicc most programs cannot compile (and other missing files such as nvvm/libdevice, which CMake checks for to detect CUDA in the first place)

Warning: shameless self-promoting!

Maybe my guide for this could help you?

https://trn84.medium.com/creating-a-cross-compilation-like-process-for-jetson-tx2-aarch64-using-qemu-with-docker-5cb38f6a65bc

It shows how to use docker together with QEMU to build for the Jetson TX2 on your x86 machine...

BR

[fh-igd-mueller-roemer]

@trn84 I can't find cicc/nvvm on my Jetson either, so I don't see how building an image from the MMC helps?

[trn84]

So you are not able to compile CUDA natively on the Jetson in the first place?

[fh-igd-mueller-roemer]

@trn84 Building directly on the board isn't really viable for me due to the limited MMC space anyway, so I have only used cross compiles so far. There is a CUDA toolkit apt package for the jetson board available via https://repo.download.nvidia.com/jetson/common so maybe I'll just try including that repo and installing from it on the docker image....

[fh-igd-mueller-roemer]

Adding

RUN cd /etc/apt/trusted.gpg.d/ \
 && wget https://repo.download.nvidia.com/jetson/jetson-ota-public.asc \
 && echo "deb https://repo.download.nvidia.com/jetson/common r32 main" > /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

RUN apt-get update \
 && apt-get install -y cuda-toolkit-10-0 \
 && rm -rf /var/lib/apt/lists/*

to my Dockerfile works. Obviously the image is now very large, so it should really only be used as a build environment.

[otamajakusi]

Can this article help you? https://medium.com/@hiroyuki.obinata/3-times-faster-than-native-build-nvidia-jetson-12fbb3d30450

[fh-igd-mueller-roemer]

Can this article help you? https://medium.com/@hiroyuki.obinata/3-times-faster-than-native-build-nvidia-jetson-12fbb3d30450

Including the compiler and libraries as volumes from a local JetPack install (not an option for me) seems overly complicated and fragile. The solution I described in https://github.com/NVIDIA/nvidia-docker/issues/1179#issuecomment-797562463 works well and is simpler.