search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.64k stars 4.83k forks source link

Using NVIDIA JETSON with D435 camera #3617

Closed MaryBD closed 4 years ago

MaryBD commented 5 years ago

Hi all, I appreciate any opinions/advices about using NVIDIA JETSON with D435 camera. either: https://www.siliconhighwaydirect.co.uk/ProductDetails.asp?ProductCode=945-82972-0005-000 or: https://www.siliconhighwaydirect.co.uk/product-p/945-82771-0005-000.htm Right now I am using two D435 cameras win 10 python. Thak you.

Jrokisky commented 5 years ago

I am currently using 1 D435 with a Jetson tx2 and JetsonHacks.com was a big help.

I have yet to try multiple cameras yet.

DaleTron commented 5 years ago

Jrokisky - can you remember which install instructions you used? The Jetsonhacks info is awesome in general, but the install scripts in this case seem out of date as in very old librealsense version called. Updating the version in the script doesn't help as then the USB patch called no longer exists in the new librealsense...

MartyG-RealSense commented 5 years ago

Have you seen the SDK documentation page for installing Librealsense on the Jetson TX2, please?

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md

If you are using the newer Jetson AGX Xavier instead of TX2, links to installation guides on JetsonHacks can be found here:

https://github.com/IntelRealSense/librealsense/issues/3290

DaleTron commented 5 years ago

Thanks for the suggestion but it seems that page is also out of date. In step 2 - ./scripts/patch-realsense-ubuntu-xenial.sh - no longer exists in the latest SDK version. I could not see another script in the folder that would work as an alternative.

MartyG-RealSense commented 5 years ago

This link may be helpful:

https://github.com/IntelRealSense/librealsense/issues/2712#issuecomment-437996990

DaleTron commented 5 years ago

Thanks for your help, I shall give it a try tomorrow as have run out of time today now.

Jrokisky commented 5 years ago

There is also a PR currently open as well that has update jetson installation instructions: https://github.com/IntelRealSense/librealsense/pull/1438

I outlined what I ended up doing here: https://github.com/Jrokisky/ros2_system_setup/blob/master/instructions/2_realsense.md

DaleTron commented 5 years ago

Thank you both, I now have it working with ROS1. Sorry to Hijack your thread MaryBD but it seemed relevant and I'm sure will be helpful. So far that it is much better than the ZED due to not demanding much from the TX2's resources.

abylikhsanov commented 5 years ago

@ARRevolution Can you share the kernel script that made your realsense working? I have TX1 3.3 L4V r28.2.0, when running script from Jetsonhacks my RGB frames come not in full

DaleTron commented 5 years ago

I have not tested that yet and I am on a TX2. Also I have just noticed that the camera is recognised as USB2.1 instead of USB3 so I am unlikely to get good results atm. Script wise I followed this in the end the but skipped the first step as it should no longer be required since the kernel was upgraded in Jetpack a while ago. https://github.com/jetsonhacks/installRealSense2ROSTX

Also see Jrokisky's post above - it has some good detail on the individual steps.

abylikhsanov commented 5 years ago

@ARRevolution I see, I thought you had it fully working. With USB 2.1 you won't be able to get RGB image and it will be useless. I did modify the script and was able to get both RGB and Depth images but RGB sometimes loses the images and I get a slight lag.

I don't think that realsense is better than ZED in development wise. ZED is just plug and play and you do not have to spend several days modifying kernel scripts just to make it working. I would probably stick to ZED this time.

Jrokisky commented 5 years ago

@ARRevolution ensure that the usb cable you're using is a usb 3 cable and if you're using a hub, ensure that that is also usb 3.

DaleTron commented 5 years ago

@abylikhsanov - I thought I had it fully working too when I saw the depthcloud in rviz. It was only after my post that I tried the realsense viewer and saw it was listed as USB2... The ZED caused me massive issues in development. The IMU seemed unreliable plus it's data stream not published. Also it never really achieved a frame rate >15fps even at vga resolution. So I'm happy to try the Realsense even if it does take some time to get installed.

@Jrokisky - I'm using the cable that came with it which says USB3.1 on. Unfortunately I don't have any other USB3 type C cables to try so I guess I'll have to order one and keep reading around the issue.

Jrokisky commented 5 years ago

@ARRevolution I think I've run into a similar issue on my Thinkpad. Both ports are supposed to be 3.0, but I think the Realsense was only treating one of them as being actually 3.0. Have you tried other ports?

DaleTron commented 5 years ago

The TX2 only has the one USB3 port :( I thought I might have forgotten to run the setupTX.sh from: https://github.com/jetsonhacks/installRealSense2ROSTX/ Because it now seems to be recognised as USB3 in lsusb. But, when doing 'rs-enumerate-devices' it is no longer found.

I just tried to replicate your steps to install but when running the final modified script I got the following error: Unsupported distribution kernel version 4.4.38-tegra . The patches are maintained for Ubuntu LTS bionic/xenial with kernel versions 4.4, 4.8, 4.10 and 4.15 only

Any ideas? What Jetpack version did you use? I see 4.2 is now available but it has LTS 4.9 kernel. I am on Jetpack 3.3.

Jrokisky commented 5 years ago

@ARRevolution I also ran into that issue, but wasn't sure if it was unique to my system or not. I think when installing jetpack you end up renaming the kernel (which the script doesn't like). I believe I solved the issue by hardcoding: kernel_branch=$(choose_kernel_branch $LINUX_BRANCH) to kernel_branch="master" as the current jetpack version uses kernel 4.4.

I looked at the values in: https://github.com/IntelRealSense/librealsense/blob/master/scripts/patch-utils.sh and hardcoded it based on the idea that the current Jetpack version uses the 4.4 kernel.

abylikhsanov commented 5 years ago

@ARRevolution You don't need that script, you can just follow the instructions here: https://github.com/jetsonhacks/buildLibrealsense2TX

If you read the script you can see that essentially it does the same thing as the link above.

It is interesting to hear that you had issues with ZED. For me, it was plug and play, I have run their script and that's it.

zh-12345 commented 5 years ago

i am using win 10 python with D435，but i come across questions with "Module 'pyrealsense2' has no 'pointcloud' memberpylint(no-member)" and so on, i follow steps but no works,what should i do ? i need help! 7/5000

Jrokisky commented 5 years ago

@zh-12345 I am not sure how to proceed with your issue.

It may be best to create a new issue.

DaleTron commented 5 years ago

Thanks both for your help, I used the Jetsonhacks script and it appears to be successful. I managed to borrow a D435 with cable which works on USB3 as expected. However my D435i is still only recognised as USB2 and intermittently not at all. I upgraded the FW on both cameras to the latest but made no difference.

@abylikhsanov - I agree that the ZED worked out of the box which was great. The issues I had were in trying to make it work with ROS navigation. The ZED would initialise with seemingly random IMU data, sometimes it would think it was looking directly up when it was horizontal. It also seemed prone to drift over time so I would have to reinitialise it every 20 mins otherwise localisation would fail. Perhaps all of this was down to me being a ROS newb though, or the fact I bought it used off ebay....

@zh-12345 - sorry I have no experience of realsense on windows

DaleTron commented 5 years ago

Found the issue - The D435i isn't supported until Librealsense v2.16.2 but the Jetsonhacks script buildLibrealsense2TX installs v2.13. Guess I'll be recompiling the kernel yet again!

zh-12345 commented 5 years ago

@zh-12345 I am not sure how to proceed with your issue.

It may be best to create a new issue.

thanks!

zh-12345 commented 5 years ago

Found the issue - The D435i isn't supported until Librealsense v2.16.2 but the Jetsonhacks script buildLibrealsense2TX installs v2.13. Guess I'll be recompiling the kernel yet again!

but i used D435,

Found the issue - The D435i isn't supported until Librealsense v2.16.2 but the Jetsonhacks script buildLibrealsense2TX installs v2.13. Guess I'll be recompiling the kernel yet again!

thanks! it is so good to receive your reply! but i used D435,and i can open and take pictures . but the depth pictures have no color. errors lists the "Module 'pyrealsense2' has no 'pointcloud' memberpylint(no-member)" like this! i thought i've done something wrong! thanks again! out

MaryBD commented 5 years ago

@zh-12345 Make sure you have used colorizer and you have objects in different distances in front of camera.

MaryBD commented 5 years ago

I am working on a laptop so it seems my only option is NVIDIA JETSON AGX XAVIER and not TX2. Is there anyone that uses Jetson Xavier, and any more advices whether I need to use it at all or I can continue working on a laptop with two D435? Thanks.

MaryBD commented 5 years ago

@ARRevolution Dont worry! useful discussions!

RealSenseCustomerSupport commented 5 years ago

Hi @maryBD,

Still need any support for this topic? You can refer to the multiple camera whitepaper. Some USB3 bandwidth constraint there if two more more D435 cameras used. And, also the system loading concern of the overall SW stack processing.

MaryBD commented 5 years ago

Thanks

MaryBD commented 5 years ago

I tried to install realsense D435 on Jetson TX2 (Jetpack 3.3) following this: https://dev.intelrealsense.com/docs/nvidia-jetson-tx2-installation which didn't work, and using this: https://www.youtube.com/watch?v=9JgPZ2UDhiI

which at the end showed some errors: -- The CUDA compiler identification is unknown CMake Error at CMakeLists.txt:7 (project): No CMAKE_CUDA_COMPILER could be found.

Tell CMake where to find the compiler by setting either the environment variable "CUDACXX" or the CMake cache entry CMAKE_CUDA_COMPILER to the full path to the compiler, or to the compiler name if it is in the PATH.

-- Configuring incomplete, errors occurred! See also "/home/nvidia/librealsense/build/CMakeFiles/CMakeOutput.log". See also "/home/nvidia/librealsense/build/CMakeFiles/CMakeError.log". Building librealsense, headers, tools and demos make: *** No targets specified and no makefile found. Stop.

real 0m0.004s user 0m0.000s sys 0m0.000s librealsense did not build Retrying ... make: *** No targets specified and no makefile found. Stop.

real 0m0.004s user 0m0.000s sys 0m0.000s librealsense did not successfully build Please fix issues and retry build

DO you know how I can install it? Thank you.

DaleTron commented 5 years ago

The answer is in the thread above but it is a little convoluted.... Follow the jetson hacks instructions exactly and take my previous comment into account First- https://github.com/IntelRealSense/librealsense/issues/3617#issuecomment-478272810

https://github.com/jetsonhacks/buildLibrealsense2TX

This worked for me with D435i so should work with the D435

Jrokisky commented 5 years ago

@MaryBD I put together some instructions for installing the Jetson D435 on the TX2 here: https://github.com/Jrokisky/ros2_system_setup/tree/master/instructions

It may be out of date.

MaryBD commented 5 years ago

I tried to install D435 but it returns an error that CUDA does not exist. I flash Jeston TX2 using both JetPack 3.2 and 3.3, the installation can be completed successfully but there is no CUDA installed in Jetson. Is there anyone facing this issue and how do you overcome?

DaleTron commented 5 years ago

There are 2 parts to the Jetpack install. 1 - Flash OS 2 - Install CUDA etc. Sounds like you missed the second part. Follow the jetson hacks video: www.jetsonhacks.com/2017/03/21/jetpack-3-0-nvidia-jetson-tx2-development-kit/amp/

Jrokisky commented 5 years ago

Based on your errors it looks like you need to set CMAKE_CUDA_COMPILER.

Some info about CUDACXX: https://cmake.org/cmake/help/v3.13/envvar/CUDACXX.html

The following may be of interest: https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000457964-Not-Locating-CUDA-Compiler

On Thu, Aug 22, 2019, 4:48 PM ARRevolution notifications@github.com wrote:

There are 2 parts to the Jetpack install. 1 - Flash OS 2 - Install CUDA etc. Sounds like you missed the second part. Follow the jetson hacks video:

www.jetsonhacks.com/2017/03/21/jetpack-3-0-nvidia-jetson-tx2-development-kit/amp/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3617?email_source=notifications&email_token=AAPDV7HWLYICOKHQASS35QLQF33SXA5CNFSM4HCCIYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD46LDWQ#issuecomment-524071386, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPDV7D4Q2EHWWEKOY5ZLN3QF33SXANCNFSM4HCCIYXQ .

MaryBD commented 5 years ago

Flash OS

Thanks for your comments. I already followed this instruction, everything installed successfully no error but no CUDA installation.

MaryBD commented 5 years ago

I notice that when I flash the TX2 after installing everything in the host machine, on the terminal that runs flashing it shows: Obtaining the IP address of target but never pass this line and never shows the ip address. Do you know what should I do?

Jrokisky commented 5 years ago

Hi Mary, I believe that I skipped that with Ctrl-c but am not entirely sure. Did you see my comment above about the missing environment variable? I think your original problem was that the cuda compiler could not be found.

On Tue, Aug 27, 2019, 10:03 AM MaryBD notifications@github.com wrote:

I notice that when I flash the TX2 after installing everything in the host machine, on the terminal that runs flashing it shows: Obtaining the IP address of target but never pass this line and never shows the ip address. Do you know what should I do?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3617?email_source=notifications&email_token=AAPDV7BNACKIMPYBJVIIAU3QGUU3RA5CNFSM4HCCIYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5HYKBA#issuecomment-525305092, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPDV7EGPGYRQRJG2VV2MCTQGUU3RANCNFSM4HCCIYXQ .

MaryBD commented 5 years ago

Hi Mary, I believe that I skipped that with Ctrl-c but am not entirely sure. Did you see my comment above about the missing environment variable? I think your original problem was that the cuda compiler could not be found. … On Tue, Aug 27, 2019, 10:03 AM MaryBD @.***> wrote: I notice that when I flash the TX2 after installing everything in the host machine, on the terminal that runs flashing it shows: Obtaining the IP address of target but never pass this line and never shows the ip address. Do you know what should I do? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3617?email_source=notifications&email_token=AAPDV7BNACKIMPYBJVIIAU3QGUU3RA5CNFSM4HCCIYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5HYKBA#issuecomment-525305092>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPDV7EGPGYRQRJG2VV2MCTQGUU3RANCNFSM4HCCIYXQ .

Thanks for your both comments, I am going to check it can solve it.

MaryBD commented 5 years ago

I finally installed Realsense D435 on TX2 following this: https://github.com/IntelRealSense/librealsense/blob/master/doc/installation.md and then at "Prepare Linux Backend and the Dev. Environment" instead of what suggested for ubuntu 16 I followed ubuntu 14 and it works now. However the streaming is very slow and I see lots of errors while streaming.

MaryBD commented 5 years ago

Hi Mary, I believe that I skipped that with Ctrl-c but am not entirely sure. Did you see my comment above about the missing environment variable? I think your original problem was that the cuda compiler could not be found. … On Tue, Aug 27, 2019, 10:03 AM MaryBD @.***> wrote: I notice that when I flash the TX2 after installing everything in the host machine, on the terminal that runs flashing it shows: Obtaining the IP address of target but never pass this line and never shows the ip address. Do you know what should I do? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3617?email_source=notifications&email_token=AAPDV7BNACKIMPYBJVIIAU3QGUU3RA5CNFSM4HCCIYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5HYKBA#issuecomment-525305092>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPDV7EGPGYRQRJG2VV2MCTQGUU3RANCNFSM4HCCIYXQ .

I reinstall everything and skip the terminal with ctrl c but still no CUDA on jetson. I tried to follow instruction for installing CUDA but I got confused by different availabe instructions that fails. Is there any suggestion which one to follow? Thanks, I am also going to check the environment variable.

Jrokisky commented 5 years ago

Does the command: which nvcc

Return anything?

On Wed, Aug 28, 2019, 6:36 AM MaryBD notifications@github.com wrote:

Hi Mary, I believe that I skipped that with Ctrl-c but am not entirely sure. Did you see my comment above about the missing environment variable? I think your original problem was that the cuda compiler could not be found. … <#m-1294206892554894611> On Tue, Aug 27, 2019, 10:03 AM MaryBD @.***> wrote: I notice that when I flash the TX2 after installing everything in the host machine, on the terminal that runs flashing it shows: Obtaining the IP address of target but never pass this line and never shows the ip address. Do you know what should I do? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3617 https://github.com/IntelRealSense/librealsense/issues/3617?email_source=notifications&email_token=AAPDV7BNACKIMPYBJVIIAU3QGUU3RA5CNFSM4HCCIYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5HYKBA#issuecomment-525305092>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPDV7EGPGYRQRJG2VV2MCTQGUU3RANCNFSM4HCCIYXQ .

I reinstall everything and skip the terminal with ctrl c but still no CUDA on jetson. I tried to follow instruction for installing CUDA but I got confused by different availabe instructions that fails. Is there any suggestion which one to follow? Thanks, I am also going to check the environment variable.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3617?email_source=notifications&email_token=AAPDV7DMQ7NOIUL2VYAYDVDQGZIJDA5CNFSM4HCCIYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5KVJUY#issuecomment-525685971, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPDV7B5W4WZ2CIAXJ7SROTQGZIJDANCNFSM4HCCIYXQ .

MaryBD commented 5 years ago

No it doesn't return anything. I have CUDA 9 on host and nothing on jetson.

Jrokisky commented 5 years ago

Any chance something similar to: https://devtalk.nvidia.com/default/topic/1037725/jetpack-3-2-1-nvcc-is-not-in-path/ is happening?

On Wed, Aug 28, 2019, 7:20 AM MaryBD notifications@github.com wrote:

No it doesn't return anything. I have CUDA 9 on host and nothing on jetson.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/IntelRealSense/librealsense/issues/3617?email_source=notifications&email_token=AAPDV7H6D2LSTSYUEGHEHDDQGZNRDA5CNFSM4HCCIYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5KY3EA#issuecomment-525700496, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPDV7GZ26VTF33OVUIA6A3QGZNRDANCNFSM4HCCIYXQ .

MaryBD commented 5 years ago

I finally manage to install CUDA, and run D435 on Jetson TX2... Using Jetpack 3.3: I used Ethernet cable to connect both host and Jetson to the same router. After flashing the TX2 the installation stuck at "Obtaining the target ip address" I skip that but ctrl c. Then I run "ifconfig" at jetson and pick the address by the eth0, use that address at host "ping ", ping was succesfull so at host I run "ssh nvidia@". Then I run jetpack3.3 .run from another terminal and at the installation step by double clicking on the "install" icon infront of flash OS I could change it to none, and then push next. In the next steps I got a new window in which I could write the jetson address (the target address from ifconfig...), and then "nvidia" for both user and password, and afterward the installation of CUDA and other utilities performed succesfully. Hope it helps!

MaryBD commented 5 years ago

I tried to run two Intel realsense D435 cameras on Jetson TX2 using python. I can connect to both cameras but I cannot stream both of them at the same time and one of them looses streaming after a few seconds. Is there anyone else having the same issue?

MikeFlanigan commented 5 years ago

The answer is in the thread above but it is a little convoluted.... Follow the jetson hacks instructions exactly and take my previous comment into account First- #3617 (comment)

https://github.com/jetsonhacks/buildLibrealsense2TX

This worked for me with D435i so should work with the D435

I've been working on getting a D435i working on a JetsonTX2 and have read through this thread and tried multiple solution paths.

I'm a little unclear on this (quoted) post. Did it work with Jetpack 4.2? The jetson hacks article https://github.com/jetsonhacks/installRealSense2ROSTX is for Jetpack 3.3 no?

MaryBD commented 5 years ago

The answer is in the thread above but it is a little convoluted.... Follow the jetson hacks instructions exactly and take my previous comment into account First- #3617 (comment) https://github.com/jetsonhacks/buildLibrealsense2TX This worked for me with D435i so should work with the D435

I've been working on getting a D435i working on a JetsonTX2 and have read through this thread and tried multiple solution paths.

I'm a little unclear on this (quoted) post. Did it work with Jetpack 4.2? The jetson hacks article https://github.com/jetsonhacks/installRealSense2ROSTX is for Jetpack 3.3 no?

Jetpack 4.2 [L4T 32.1] worked best for me on JetsonTX2 to run two realsense D435. I followed this link to build patched kernel and install librealsense: https://github.com/Tengyun-Mo/buildLibrealsense2TX2

MikeFlanigan commented 5 years ago

The answer is in the thread above but it is a little convoluted.... Follow the jetson hacks instructions exactly and take my previous comment into account First- #3617 (comment) https://github.com/jetsonhacks/buildLibrealsense2TX This worked for me with D435i so should work with the D435

I've been working on getting a D435i working on a JetsonTX2 and have read through this thread and tried multiple solution paths. I'm a little unclear on this (quoted) post. Did it work with Jetpack 4.2? The jetson hacks article https://github.com/jetsonhacks/installRealSense2ROSTX is for Jetpack 3.3 no?

Jetpack 4.2 [L4T 32.1] worked best for me on JetsonTX2 to run two realsense D435. I followed this link to build patched kernel and install librealsense: https://github.com/Tengyun-Mo/buildLibrealsense2TX2

Ah, since we're working with a JetsonTX24gb which is supported by Jetpack 4.2.2 (& 4.2.1) but not by Jetpack 4.2 (L4T 32.1) we have L4T 32.2 and are trying to figure out how to modify the kernel patches in the links you referenced to get them to work on the new Jetson.

dorodnic commented 4 years ago

Should be significantly improved today

shaoxiang commented 4 years ago

Thanks @Jrokisky In Jetson Nano Jetpack4.4 run the cmd: export PATH="${PATH}:/usr/local/cuda-10.2/bin" solved my No CMAKE_CUDA_COMPILER could be found error run: which nvcc shows: /usr/local/cuda-10.2/bin/nvcc