python3.8 + pytorch + cuda + opencv

[SkalskiP]

Hello, @dusty-nv 👋! I absolutely love your work! I've done some experiments with Jetson Nano and Xavier in the past put I must say I'm stuck... I need to deploy computer vision application that requires python3.8 or higher to run. Additionally we need pytorch and opencv with cuda support. Application was originally deployed on x86 machines but now we want to bring it to arm. I tried few different approaches but with no luck.

1. I used nvcr.io/nvidia/l4t-pytorch:r32.6.1-pth1.9-py3 as base image for my Dockerfile. I managed to get pytorch and opencv with cuda but not on python3.8.
2. I used nvcr.io/nvidia/l4t-pytorch:r34.1.0-pth1.12-py3 as base image for my Dockerfile. I get python3.8 and pytorch for "free", but there is no cuda support and I did not managed to setup opencv.

Do you have any advice for me? Does it make sense to pursue senario 2? Is it possible to install opencv and have cuda support on r34.1.0? Or is it not possible and it is just easier to downgrade the app to python3.6?

[CourchesneA]

Hi @SkalskiP , for option 2, I can verify that it works properly out of the box for me (python3.8, opencv, torch & cuda). You would have to make sure that you have matching version on you host since CUDA is mounted through nvidia-docker. For ex. my setup on a Jetson NX:

 cat /etc/nv_tegra_release 
# R34 (release), REVISION: 1.1, GCID: 30414990, BOARD: t186ref, EABI: aarch64, DATE: Tue May 17 04:20:55 UTC 2022

If you have an older L4T with jetpack 4.5, CUDA would not work in r34.1.x containers

[SkalskiP]

Hi, @CourchesneA 👋, thanks for the response. What do you mean by "out of the box"? Is there any prebuild docker image that supports that configuration or do you still need to do some installation manually?

Over the weekend I put some work and build all I need on top of the old nvcr.io/nvidia/l4t-base:r32.6.1 image. I'm still lacking opencv, but I have some parts already figured out, although the build time is killing me... 6h+.

FROM nvcr.io/nvidia/l4t-base:r32.6.1

ENV DEBIAN_FRONTEND=noninteractive
ENV VIRTUAL_ENV=/opt/venv
ENV CUDA_HOME=/usr/local/cuda
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
ENV PATH=$PATH:$CUDA_HOME/bin

# 1. Install system dependencies

RUN apt-get update -qq \
    && apt-get -y install --no-install-recommends \
    curl \
    wget \
    git \
    libgdm-dev \
    libnss3-dev \
    libssl-dev \
    libsqlite3-dev \
    libreadline-dev \
    libbz2-dev \
    libdb-dev \
    libdb++-dev \
    libgdbm-dev \
    libgdbm-dev \
    libffi-dev \
    cmake \
    ca-certificates \
    build-essential

# 2. Install python 3.8

RUN apt-get -y install --no-install-recommends \
    python3.8 \
    python3.8-dev \
    python3.8-venv \
    && curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py \
    && python3.8 get-pip.py \
    && python3.8 -m venv $VIRTUAL_ENV \
    && . $VIRTUAL_ENV/bin/activate \
    && pip install wheel setuptools

# 3. Install pytorch 1.7

RUN apt-get -y install --no-install-recommends \
    libopenblas-base \
    libopenmpi-dev \
    libomp-dev \
    && . $VIRTUAL_ENV/bin/activate \
    && pip install Cython \
    && pip install numpy \
    && git clone --recursive --branch 1.7 http://github.com/pytorch/pytorch \
    && cd pytorch \
    && pip install -r requirements.txt \
    && python setup.py build --cmake \
    && python setup.py install \
    && python setup.py develop

# 4. Install torchvision 0.8

RUN apt-get -y install --no-install-recommends \
    libjpeg-dev \
    zlib1g-dev \
    && . $VIRTUAL_ENV/bin/activate \
    && git clone https://github.com/pytorch/vision.git \
    && cd vision \
    && git checkout v0.8.1-rc1 \
    && python setup.py install

[CourchesneA]

Yes, the option number two that you provided in your question does support the environment that you are asking for. You would need to have matching cuda version on your host, which comes with L4T 34.1.1 (jetpack 5). I assume that is the part you are missing, you would need to reflash your host with the new jetpack.

I think the manual upgrades you are working on should work. However you are manually rebuilding PyTorch, which will inevitably take a while. I would suggest taking the wheels from the jetson zoo instead, which will save you 95% of your build time

[SkalskiP]

@CourchesneA Are there any prebuild PyTorch wheels that would work with python 3.8 and L4T 32.6.1? I thought only python3.6 was supported for jetpack <=5.0.

[CourchesneA]

Oops you are right that version is not available in the archive. I don't have a version to share, but once you build it once you could save the .whl file and then just save it somewhere

...
RUN cd /build/pytorch/src && mkdir dist && python3 setup.py bdist_wheel -d /build/pytorch | tee /build/pytorch_build.log
RUN pip3 install /build/pytorch/torch-1.10.0-cp38-cp38-linux_aarch64.whl

[SkalskiP]

So you are suggesting splitting my Dockerfile into two - one to build and extract wheel and one to install it?

[CourchesneA]

Yes, that would be a good way to do it. For our project we have found that we don't need to rebuild once we have a wheel so we just host it as a static artifact

[SkalskiP]

This is great, one question tho about Linux packages... does the production image the one that is simply installing the wheel needs to have all Linux packages installed? I guess some of them are only installed so that you can build the wheel and are not required when you install it. That could make the production image slimmer. Did you do it? And if so how did you figured it out?

[CourchesneA]

Yea anything that you used only to build the wheel won't be required in the production image. You can also remove the .whl itself once it's installed

[dreammonkey]

@SkalskiP Did you get this up and running (on the nano)? Would love to see your final docker file !

[SkalskiP]

@dreammonkey I managed to have it running on Xavier, but in docker so (performance aside) it should be the same. I question first: do you want to run it with an older JetPack (like 4.6 for example?) or the new one (like 5.1)?

[dreammonkey]

Hi thanks for your answer, I'm trying to run on a jetson nano dev board, i believe it only supports jetpack up to 4.6.

Do you believe performance is better running inference on the host instead of in a docker container?

[javadan]

I'm also trying to get my hands on a pre-built Python 3.7 or 3.8 built version of 4.6/.1 Subscribing to the thread in case someone replies.

[SkalskiP]

Hi @javadan, I actually convinced CEO of the company I work for to open source our Docker image as well as some demo app to show you how we did it. I'm not really sure when that will happen, but I'll make sure to throw some info into this issue.

[javadan]

That would be cool. I don't mind leaving it building if the Dockerfile's tested. I just don't want to get stuck into the inevitable debugging if I can avoid it.

I'm getting super slow inference running RealSense through TF2 MobileNetSSDv2, streaming as JPEG over HTTP, on the Nano, (~2 FPS locally, ~0.25 FPS over the LAN) and all the instructions I've found for converting TF2 models to TensorRT require ONNX libraries that need 3.7.

Trying other options in the meantime, specifically, switching from JPEG/HTTP to GStreamer/RTSP, since the network is clearly slowing it down. If I can't get that working, I'll try ROS2 again. First prize would be deploying a TensorRT model though, as that should give me a few more FPS.

(PS.1: Tried GStreamer/RTSP - worse results than Jpeg/http. )

[javadan]

@SkalskiP just following up... Need to get python 3.7+ on a Nano (4.6.x) now, for pytorch with PIL. Any quick solution yet? Or just leave it building? Thanks

[dusty-nv]

@javadan if you can use Python 3.8, there is a PyTorch 1.7 wheel for Python 3.8 and Jetson Nano here: https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048/972

[javadan]

Oh boy, so I worked out how to use that wheel, but now it seems I need at least torch 1.8, as

from torchvision import transforms

leads to

File "/usr/local/lib/python3.8/dist-packages/torchvision/ops/drop_block.py", line 105, in <module>
    torch.fx.wrap("drop_block2d")
AttributeError: module 'torch.fx' has no attribute 'wrap'

It's always something. Anyway, i'll try install pytorch 1.8 manually using @SkalskiP 's example for 1.7, next. Thanks

[SkalskiP]

@javadan I left my old company last week. So I won't be able to share the exact Dockerfile with you. But if you'd share what you have, I should be able to workout a solution with you :)

[javadan]

Seems they stopped updating torch github branches past 1.7... and the official instructions requires conda, and archiconda doesn't have torch. And the torch 1.8 wheels in the Jetson Zoo are python 3.6. So actually not sure what to do, at this point.

[dusty-nv]

Seems they stopped updating torch github branches past 1.7... and the official instructions requires conda, and archiconda doesn't have torch. And the torch 1.8 wheels in the Jetson Zoo are python 3.6. So actually not sure what to do, at this point.

Hi @javadan, you would need to build PyTorch 1.8 from source for Python 3.8 on JetPack 4.x - here's a post about doing that: https://forums.developer.nvidia.com/t/install-pytorch-with-python-3-8-on-jetpack-4-4-1/160060/3

There are also some patches included in the Build from Source section of this post if you require them: https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048

[XiangkunFang]

I'm in the same situation. If I use torch 1.10 with python3.6, then there will be an error that needs a higher version of python. If I use torch 1.7 with python3.6, then it gives me 'AttributeError: module 'torch.fx' has no attribute 'wrap''. So annoying.