TegraJPEG support for recent L4T versions (e.g. 28.3.1)

[dbruning]

I'm using a Jetson Nano and trying to find out how quickly it can take frames from the Kinect v2 camera.

Out of the box, it's very slow with the RBG frames, so I'm trying to enable TegraJPEG support. It seems that cmake_modules/FindTegraJPEG.cmake is responsible for downloading and compiling TegraJPEG library, but there are a couple of problems with it:

1. it tries to find l4t version by parsing /etc/nv_tegra_release, but that file isn't there anymore

2. I tried a workaround of guessing the URL of the BSP source, based on the previous examples in that .cmake file. My guess was http://developer.download.nvidia.com/embedded/L4T/r28_Release_v3.1/BSP/source_release.tbz2 (because 28.3.1 seems to be the latest release of L4t) but that link isn't correct. Perhaps the location of those sources changed recently.

Does anyone know the current location of the BSP sources for the Nano?

Disclaimer: I'm new to C and CMake so apologies if I've made an obvious errors.

[xlz]

The CMake file is trying many ways to find Nvidia's custom jpeglib.h. In a recent installation it looks like this file is installed as part of "Tegra Multimedia API" in /usr/src/tegra... but I'm not sure all people include it during installation.

Anyway, that CMake file is only supposed to produce the INCLUDE PATH of jpeglib.h and LINKING -lnvjpeg. Maybe you can manually add the two flags somewhere else.

[dbruning]

Thanks for the reply & the advice. I've managed to get it working (very messily) in my fork here. It runs with a good framerate pulling both RGB and depth frames (if I disable the registration code), but will crash after about 5 seconds, having used up all the memory on the Nano. Next time I get a chance to work on this I'll try to find the memory leak - perhaps something has changed in the Nvidia JPEG library since it was first used by libfreenect2. If I can get it all working properly I'll tidy it up & make a pull request.

[chunter1]

@dbruning I got the same probem. Could you fix the crashing?

[dbruning]

@chunter1 No I haven't dived back into it. Next steps would have been to try to understand how the current NVidia JPEG library "Tegra Multimedia API" mentioned by @xlz works & what changes would be needed to libfreenect2 to use it without memory leaks. Happy to collaborate with you on that, I'm not a C / embedded programmer so it's slow progress for me.

[chunter1]

I'm not an C++ expert either ;) However, since the Kinect RGB-video runs significantly faster with TegraRGB enabled, i think it's worth diving into it. Hope we can fix it somehow.

[chunter1]

Removing the "RTLD_LAZY" option in the "tegra_jpeg_rgb_packet_processor.cpp" file stops memory leakage, but performance is also reduced.

RTLD_LAZY Relocations shall be performed at an implementation-defined time, ranging from the time of the dlopen() call until the first reference to a given symbol occurs. Specifying RTLD_LAZY should improve performance on implementations supporting dynamic symbol binding as a process may not reference all of the functions in any given object. And, for systems supporting dynamic symbol resolution for normal process execution, this behavior mimics the normal handling of process execution.

[xlz]

I'll look into it.

[dbruning]

@xlz thankyou. Do you have some way I can make a small donation to say thanks for your time?

[xlz]

I fixed the build system in https://github.com/OpenKinect/libfreenect2/commit/da68a718cd7e5022f8e02852433f17a2e97dc8bc. It should build now.

The crash is due to memory leaking in libnvjpeg.so, see https://devtalk.nvidia.com/default/topic/1060896/jetson-tx2/jetpack-4-2-1-nvjpeg-leaking/.

You must download R32_2_libnvjpeg.zip in the thread and replace /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so (back up first of course). As stated in the thread, this fix in libnvjpeg.so probably will get shipped in L4T 32.3 (now it's 32.2.x).

I don't have a Jetson Nano so I don't know if this works on Nano.

No donation is needed. Necessary equipment happens to be around me, and it didn't take me much time.

[xlz]

Re RTLD_LAZY: Those dlopen hacks are required because libnvjpeg.so have conflicting symbols with libjpeg.so, which means if you dynamically link libnvjpeg.so to libfreenect2.so, any program linking libfreenect2.so will be unable to use libjpeg.so. Those dlopen hacks can hide the symbols in libnvjpeg.so.

[chunter1]

@xlz thank you very much for your help! I got it running now on my jetson nano without problems. The only thing i had to adapt was:

PATHS /usr/src/nvidia/tegra_multimedia_api/include

[eric-schleicher]

confirming here that on L4T 32.1 that the path is as @chunter1 . once i added the correct path cmake was able to find TegraJPEG.

I was able to compile and bring up the bin/Protonect. still get lots of skipped packets, but the framerate looks better.

Noticing a new issue with the RGB output. the colorspace doesn't seem right. So not sure if this is just a visualization problem within Protonect, or a more a fundamental issue with the TegraJPEG. I changed the findTegreJPEG.cmake but (so that that it's unfound) and recompiled without TegraJPEG and the colors are correct.

Note this is on 32.1

@chunter1 did you notice this?

[xlz]

The color is RGB rendered as BGR. Protonect doesn't bother to convert the color.