IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.6k stars 4.83k forks source link

D415 and D435 stop connecting to PC after a few hard exits #7060

Closed azalac closed 4 years ago

azalac commented 4 years ago
Required Info
Camera Model D415 / D435
Firmware Version Unknown - recently updated
Operating System & Version Linux (Ubuntu 18)
Kernel Version (Linux Only) 5.0.0
Platform PC
SDK Version 2.35.0
Language python
Segment Desktop

Issue Description

I have a python script that I'm using to process the output of the camera. Since this script is still in development, it occasionally hard crashes (seg faults, when I sigkill it due to freezes, etc). After doing this a few times (3-4), my D415 stops connecting to my PC entirely, forcing me to reboot to fix it. I've previously used a D435, and I still had the issue. Unfortunately, I don't have access to the D435 anymore, so I won't be able to use it.

I've tried unplugging the camera, and plugging it back in, but that doesn't fix anything. I can't send a hardware reset to the camera, since it's not even detected in lsusb.

I don't believe this is a power issue. I can run the script for many hours without issue. The only time this occurs is after my script crashes, and the pipelines don't get closed properly.

Looking at dmesg, it seems like there's a driver issue. I've tried reloading the uvcvideo kernel module, but it didn't have any effect. I've also tried to disable and re-enable the entire hub, but it only caused the hub to stop working properly.

Here's the relevant parts from dmesg:

[144063.981695] usb 2-3: USB disconnect, device number 89                         <--- initial connection
[144064.334486] usb 2-3: new SuperSpeed Gen 1 USB device number 90 using xhci_hcd
[144064.363100] usb 2-3: New USB device found, idVendor=8086, idProduct=0ad3, bcdDevice=50.b1
[144064.363103] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[144064.363105] usb 2-3: Product: Intel(R) RealSense(TM) Depth Camera 415 
[144064.363106] usb 2-3: Manufacturer: Intel(R) RealSense(TM) Depth Camera 415 
[144064.363107] usb 2-3: SerialNumber: 009223023958
[144064.381614] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[144064.383098] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 415  (8086:0ad3)
[144064.403106] uvcvideo: Unable to create debugfs 2-90 directory.
[144064.403216] uvcvideo 2-3:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[144064.403217] uvcvideo 2-3:1.0: Entity type for entity Processing 2 was not initialized!
[144064.403218] uvcvideo 2-3:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[144064.403219] uvcvideo 2-3:1.0: Entity type for entity Camera 1 was not initialized!
[144064.403291] input: Intel(R) RealSense(TM) Depth Ca as /devices/pci0000:00/0000:00:01.3/0000:01:00.0/usb2/2-3/2-3:1.0/input/input113
[144064.406359] uvcvideo: Unknown video format 36315752-1a66-a242-9065-d01814a8ef8a
[144064.406362] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 415  (8086:0ad3)
[144064.426609] uvcvideo: Unable to create debugfs 2-90 directory.
[144064.426732] uvcvideo 2-3:1.3: Entity type for entity Processing 7 was not initialized!
[144064.426734] uvcvideo 2-3:1.3: Entity type for entity Extension 8 was not initialized!
[144064.426736] uvcvideo 2-3:1.3: Entity type for entity Camera 6 was not initialized!
[147447.005828] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147447.549770] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147448.065756] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147448.605789] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147449.117763] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147449.629766] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147463.709783] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147464.221762] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147464.733764] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147465.245764] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147465.757771] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147466.269771] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147952.573754] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147953.085804] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147953.597763] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147954.109749] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147954.621753] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147955.133763] uvcvideo: Failed to query (GET_LEN) UVC control 1 on unit 3: -110 (exp. 2).
[147966.553663] usb 2-3: USB disconnect, device number 90                                     <--- camera stops working
[147983.574426] usb 2-3: new SuperSpeed Gen 1 USB device number 91 using xhci_hcd
[147988.669789] usb 2-3: device descriptor read/8, error -110
[147988.777625] usb 2-3: new SuperSpeed Gen 1 USB device number 91 using xhci_hcd
[147994.045763] usb 2-3: device descriptor read/8, error -110
[147994.378419] usb 2-3: new SuperSpeed Gen 1 USB device number 92 using xhci_hcd
[147999.421789] usb 2-3: device descriptor read/8, error -110
[147999.529611] usb 2-3: new SuperSpeed Gen 1 USB device number 92 using xhci_hcd
[148004.801756] usb 2-3: device descriptor read/8, error -110
[148004.937652] usb usb2-port3: attempt power cycle
[148005.866399] usb 2-3: new SuperSpeed Gen 1 USB device number 93 using xhci_hcd
[148010.941791] usb 2-3: device descriptor read/8, error -110
[148011.049623] usb 2-3: new SuperSpeed Gen 1 USB device number 93 using xhci_hcd
[148016.317756] usb 2-3: device descriptor read/8, error -110
[148016.650399] usb 2-3: new SuperSpeed Gen 1 USB device number 94 using xhci_hcd
[148021.693793] usb 2-3: device descriptor read/8, error -110
[148021.801620] usb 2-3: new SuperSpeed Gen 1 USB device number 94 using xhci_hcd
[148027.069753] usb 2-3: device descriptor read/8, error -110
[148027.205657] usb usb2-port3: unable to enumerate USB device
[149795.241688] usbcore: deregistering interface driver uvcvideo                          <--- reloading uvcvideo
[149798.244833] media: Linux media interface: v0.10
[149798.262965] videodev: Linux video capture interface: v2.00
[149798.293096] usbcore: registered new interface driver uvcvideo
[149798.293098] USB Video Class driver (1.1.1)
[149870.838368] usb 2-3: new SuperSpeed Gen 1 USB device number 95 using xhci_hcd    <------ re-plugging camera
[149875.901759] usb 2-3: device descriptor read/8, error -110
[149876.009590] usb 2-3: new SuperSpeed Gen 1 USB device number 95 using xhci_hcd
[149881.277723] usb 2-3: device descriptor read/8, error -110
[149881.610365] usb 2-3: new SuperSpeed Gen 1 USB device number 96 using xhci_hcd
[149886.653765] usb 2-3: device descriptor read/8, error -110
[149886.761588] usb 2-3: new SuperSpeed Gen 1 USB device number 96 using xhci_hcd
[149892.029727] usb 2-3: device descriptor read/8, error -110
[149892.165625] usb usb2-port3: attempt power cycle
[149893.094369] usb 2-3: new SuperSpeed Gen 1 USB device number 97 using xhci_hcd
[149898.173763] usb 2-3: device descriptor read/8, error -110
[149898.281585] usb 2-3: new SuperSpeed Gen 1 USB device number 97 using xhci_hcd
[149903.549733] usb 2-3: device descriptor read/8, error -110
[149903.882363] usb 2-3: new SuperSpeed Gen 1 USB device number 98 using xhci_hcd
[149908.925771] usb 2-3: device descriptor read/8, error -110
[149909.033585] usb 2-3: new SuperSpeed Gen 1 USB device number 98 using xhci_hcd
[149914.301720] usb 2-3: device descriptor read/8, error -110
[149914.437621] usb usb2-port3: unable to enumerate USB device

I'm not sure if the 'Failed to query' errors are normal - I get many of them even when the camera's working properly.

During this specific issue, I didn't re-plug the camera before reloading the uvcvideo module, but I've done it in the past and it's had no effect. After re-plugging it, I just get lots of 'new device / device read error' messages in dmesg before I finally get an 'unable to enumerate' error.

I've had this issue countless times (at least 10 times by now), so I've looked at a lot of usb related realsense issues but I haven't found any that are similar to my issue. Any help would be greatly appreciated.

MartyG-RealSense commented 4 years ago

Hi @azalac In situations where the camera can no longer be detected and hardware_reset() does not solve the problem, some RealSense users have solved the problem by resetting the entire USB port with an Ubuntu script instead of just resetting the camera. I recommend googling for ubuntu usb reset script for research leads.

The link below has an example of a USB port reset project that a RealSense user developed.

https://github.com/IntelRealSense/librealsense/issues/2161#issuecomment-587457793

Another RealSense user with camera enumeration problems greatly improved reliability by using industrial grade USB cables.

https://github.com/IntelRealSense/librealsense/issues/2741#issuecomment-548914984

If a program uses set_real_time in the False state then it may also be more prone to issues when stopping the pipeline, whereas such errors are less likely to occur if set_real_time is set to True. Here is an example:

https://github.com/IntelRealSense/librealsense/issues/3126

azalac commented 4 years ago

Hi @MartyG-RealSense, I believe there may be additional variables in this issue, but I'm not able to determine them. I tried reproducing the issue manually, and I wasn't able to do it. After using a script to start my python script and kill it repeatedly I was able to reproduce it, but only after quite a few runs.

Ultimately, I got this error: RuntimeError: set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: Connection timed out

With this kernel message repeated many times: [57075.652123] uvcvideo: Failed to query (SET_CUR) UVC control 1 on unit 3: -110 (exp. 1024).

This is the output from the usb reset script (I didn't realize I had to run as root, but I reran it as root after):

001  002  003  004
 ::::: > found camera at bus: 002 port: 004
Resetting USB device /dev/bus/usb/002/004
Error in ioctl: No such device
 ::::: > reset 002 004 with cpp ioctl
 ::::: > unload driver
rmmod: ERROR: ../libkmod/libkmod-module.c:793 kmod_module_remove_module() could not remove 'uvcvideo': Operation not permitted
rmmod: ERROR: could not remove module uvcvideo: Operation not permitted
 ::::: > reload driver

Curiously, the camera was still on lsusb before running this script. After running the script, the camera was no longer on lsusb.

Here's the most interesting logs from dmesg after running the script:

[57366.189379] uvcvideo: Unknown video format 36315752-1a66-a242-9065-d01814a8ef8a
[57366.189384] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 415  (8086:0ad3)
[57366.189388] uvcvideo: Failed to query (GET_INFO) UVC control 2 on unit 6: -19 (exp. 1).
[57366.189391] uvcvideo: Failed to query (GET_INFO) UVC control 3 on unit 6: -19 (exp. 1).
[57366.189394] uvcvideo: Failed to query (GET_INFO) UVC control 4 on unit 6: -19 (exp. 1).
[57366.189397] uvcvideo: Failed to query (GET_INFO) UVC control 2 on unit 7: -19 (exp. 1).
[57366.189399] uvcvideo: Failed to query (GET_INFO) UVC control 3 on unit 7: -19 (exp. 1).
[57366.189402] uvcvideo: Failed to query (GET_INFO) UVC control 6 on unit 7: -19 (exp. 1).
[57366.189405] uvcvideo: Failed to query (GET_INFO) UVC control 7 on unit 7: -19 (exp. 1).
[57366.189408] uvcvideo: Failed to query (GET_INFO) UVC control 8 on unit 7: -19 (exp. 1).
[57366.189411] uvcvideo: Failed to query (GET_INFO) UVC control 9 on unit 7: -19 (exp. 1).
[57366.189414] uvcvideo: Failed to query (GET_INFO) UVC control 10 on unit 7: -19 (exp. 1).
[57366.189417] uvcvideo: Failed to query (GET_INFO) UVC control 1 on unit 7: -19 (exp. 1).
[57366.189419] uvcvideo: Failed to query (GET_INFO) UVC control 4 on unit 7: -19 (exp. 1).
[57366.189421] uvcvideo: Failed to query (GET_INFO) UVC control 5 on unit 7: -19 (exp. 1).
[57366.189423] uvcvideo: Failed to query (GET_INFO) UVC control 11 on unit 7: -19 (exp. 1).
[57366.194663] uvcvideo: UVC non compliance - GET_DEF(PROBE) not supported. Enabling workaround.
[57366.194665] uvcvideo: Failed to query (129) UVC probe control : -19 (exp. 48).
[57366.194667] uvcvideo: Failed to initialize the device (-5).
[57366.392762] usb 2-3: new SuperSpeed Gen 1 USB device number 5 using xhci_hcd
[57371.652151] usb 2-3: device descriptor read/8, error -110
[57371.759968] usb 2-3: new SuperSpeed Gen 1 USB device number 5 using xhci_hcd
[57377.028105] usb 2-3: device descriptor read/8, error -110
[57377.360750] usb 2-3: new SuperSpeed Gen 1 USB device number 6 using xhci_hcd
[57382.404155] usb 2-3: device descriptor read/8, error -110
[57382.511967] usb 2-3: new SuperSpeed Gen 1 USB device number 6 using xhci_hcd
[57387.780128] usb 2-3: device descriptor read/8, error -110
[57387.916021] usb usb2-port3: attempt power cycle
[57388.844770] usb 2-3: new SuperSpeed Gen 1 USB device number 7 using xhci_hcd
[57393.924155] usb 2-3: device descriptor read/8, error -110
[57394.031977] usb 2-3: new SuperSpeed Gen 1 USB device number 7 using xhci_hcd
[57399.300110] usb 2-3: device descriptor read/8, error -110
[57399.632754] usb 2-3: new SuperSpeed Gen 1 USB device number 8 using xhci_hcd
[57404.676151] usb 2-3: device descriptor read/8, error -110
[57404.783970] usb 2-3: new SuperSpeed Gen 1 USB device number 8 using xhci_hcd
[57410.052117] usb 2-3: device descriptor read/8, error -110
[57410.192025] usb usb2-port3: unable to enumerate USB device

I'm fairly sure the usb reset script misreported the usb port, because the camera was plugged into 2:3, not 2:4. Hopefully it reset the correct port.

I'm using the cable that comes with the camera, should I still find a better cable? The camera seems to be pretty reliable when running, but I can get a better cable if needed.

I'm not setting real_time to False, I don't think this is related to the last issue you mentioned.

Thank you for the help.

MartyG-RealSense commented 4 years ago

For most situations, using the official cable supplied with the camera is ideal because it has been validated by Intel for use with RealSense cameras and is high quality and very reliable.

For longer cables, the data sheet document for the 400 Series cameras officially recommends Newnex as a supplier of high-grade cable products that have been tested with RealSense.

https://www.newnex.com/realsense-3d-camera-connectivity.php

When you say that the script runs for "many hours" in your opening message, roughly how long is this please? In a long-run of a RealSense 400 Series camera, so long as the camera remains within recommended temperature tolerances then it should be able to run indefinitely. If a problem does occur in a long-run, it may typically appear after 3 to 4 days and be related to a problem in the USB system (the cable or the port), or a glitch with the computer hardware or the computer's operating system.

MartyG-RealSense commented 4 years ago

Hi @azalac Do you still reuire assistance with this case, please? Thanks!

MartyG-RealSense commented 4 years ago

Case closed due to no further comments received.