Closed jmachuca77 closed 5 years ago
I was not quite sure of the sequence of events from your description. From the title about a hard reboot, it sounds as though the camera works the first time that the computer is started up, but is no longer detected after a reboot of the computer. Is that what is happening, please?
If it is, it sounds similar to an effect reported in a couple of cases with 400 Series cameras, where the camera stops working after the computer's reboot but is corrected with an unplug-replug of the USB cable. It was believed that the problem in those cases may be related to an issue with the computer's BIOS configuration.
Below are links to the reported cases that I know of:
Sorry for the confusing name. But no this happens after any type of boot, be it initial boot (hard boot) or a soft reboot. Always the camera is detected as the Movidius processor without a way to change it to be the camera. Only after power cycling the camera (unplug usb and replug) then it can be accessed as a camera.
Thanks for the clarification. I have seen a couple of past cases where the T265 has been ID'd as Movidius. Here's one of them:
Yes, I have gone through those issues. But running rs-pose does not work as the camera is not detected. And I am running the latest release built from source on the jetson platform with the options
-DBUILD_PYTHON_BINDINGS=bool:true -DPYTHON_EXECUTABLE=/usr/bin/python -DBUILD_EXAMPLES=true -DBUILD_WITH_CUDA=true -DBUILD_WITH_TM2=ON
Which should include support for the t265 and they do as replugging the camera fixes the issue but coming from boot it does not work and unplugging the camera is not an option as it will be inside a housing.
It looks as though you have covered most of the possibilities that I can think of. There have been a couple of Jetson users who had different errors that were solved by trying a different USB cable.
https://github.com/IntelRealSense/librealsense/issues/4578
I apologize that I'm unable to provide more specific solutions, as my knowledge of the deeper workings of the T265 model is limited. One of the Intel support guys on this forum will likely be able to advise you well.
I have tried with different cables but the one that comes with the camera seems to be the best one to use (this is the one I am using) I am going through a powered USB3 hub. However I cannot connect the camera directly to the Xavier because when I do that I get disconnected from the camera randomly, I think the port on the Xavier cannot give enough power to keep the camera alive. I will try that again later today. And I'll report back on the findings here. Thank you.
USB power issues have also been known with the RealSense 400 Series camera models on some single-board computing devices such as Up Board and Intel Compute Stick. An Intel support representative once said that the camera needs 2W of power reliably supplied to it, which was why a powered USB 3 hub was ideal because it could supply around 12W and so could meet the camera's power needs if the power supply was unreliable when the camera was inserted directly into a USB port on the board.
Good luck!
Tried with different USB cables and even connecting the camera directly to the USB port on the Xavier and got exactly the same issue.
I am wondering if it is a physical issue with the USB port on the Xavier. With my own laptop computer, a Windows machine, I sometimes hear a repeating USB disconnect and reconnect sound effect from my mouse even if I am not touching it at the time (though my leg may brush against the cable slightly) My mouse will go dead and its lights go out, sometimes accompanied by an on-screen warning message, and comes back to life when I give its USB connector a gentle push into the port.
This happens even more with the ports on another PC I have too, a single-board device (an Intel Compute Stick).
This also happens on an UP Board running Ubuntu 16.04.6 LTS. with a NetMate USB hub. Is the only solution to unplug and re-plug? If so this makes the T265 not usable in my environment.
$ dpkg -l | fgrep realsense ii librealsense2:amd64 2.26.0-0~realsense0.1435 amd64 Intel(R) RealSense(tm) Cross Platform API - runtime ii librealsense2-dkms 1.3.5-0ubuntu1 all Modified kernel modules for librealsense2 ii librealsense2-gl:amd64 2.26.0-0~realsense0.1435 amd64 Intel(R) RealSense(tm) - GLSL-enabled extensions ii librealsense2-udev-rules:amd64 2.26.0-0~realsense0.1435 amd64 Intel(R) RealSense(tm) Camera Capture API - udev rules ii librealsense2-utils:amd64 2.26.0-0~realsense0.1435 amd64 Intel(R) RealSense(tm) Camera Capture API - utils and demos
$ modinfo uvcvideo | fgrep realsense version: 1.1.2.realsense-1.3.5
firmware version: 0.1.0.242
@rdinoff Is your NetMate hub one that plugs into the mains electricity please, or is it a "passive" hub that draws its power from the PC?
@rdinoff Is your NetMate hub one that plugs into the mains electricity please, or is it a "passive" hub that draws its power from the PC?
It is passive.
When a RealSense model has apparent USB related issues with an Up Board, a mains powered USB hub has a better chance of correcting the problem than a passive hub, which cannot provide the stable wattage that a mains hub can. I understand how it impacts mobility though.
Just checked with a J5 Create USB Hub that plugs into the mains electricity and it also does not work. Though plugging into a USB 3.0 port on the UP Board with no hub, T265 identified properly after running rs-enumerate-devices.
The original Up Board only has a 'USB 3.0 OTG' micro port (not a full size USB 3 port) and a USB 2.0 port. The newer UP Squared has a full USB 3.0 port. Is your Up model an Up Squared, please?
The original Up Board only has a 'USB 3.0 OTG' micro port (not a full size USB 3 port) and a USB 2.0 port. The newer UP Squared has a full USB 3.0 port. Is your Up model an Up Squared, please?
correct used the 3.0 OTG micro port. Have an UP board and UP core, but no UP Squared.
Ok, thank you very much.
Some further digging brought to light that the T265 has had issues earlier this year with rs-enumerate-devices not detecting the T265.
https://github.com/IntelRealSense/librealsense/issues/3465
Are you able to get the T265 to be detected in the RealSense Viewer even if rs-enumerate-devices said it is not detected when connecting via a hub?
If I use a USB hub, none of the rs commands or realsense-viewer works unless I unplug and replug the T265 after the system has booted.
I apologize that the diagnostic process is taking some time and I appreciate all the information that you and others with this problem are patiently providing.
Since the T265 can function after unplug-replug, it is presumably not a mechanical fault with the OTG port or the hubs that is preventing initial detection on boot.
If it can start from boot when plugged directly in, it is probably not due to a BIOS issue.
It is probably not an OTG dongle / adapter cable compatibility issue with the camera, since you need to use an adapter to connect the camera directly to the OTG port. Though that would not rule out an incompatibility between the adaptor and the hub, unless the powered hub works fine with the Up Board when non-RealSense devices are plugged into it.
I'll refer this case back to Intel member @ev-mp who had some initial contact with this case 6 days ago, though they may not be able to respond until at least tomorrow now.
Thanks @MartyG-RealSense , for keeping tabs on this. I don’t think it’s a problem with the port. I have a hub connected to each port on the Xavier. So two total. They are both powered. I’m currently running 3 D435, and 2 D435i plus the T265. All other cameras enumerate correctly after boot(with the ocasional One detected as usb2.1 but corrects after a SW, reset of that camera) the T265 always needs to be unplugged and replugged even if it’s the only camera connected.
@rdinoff, this is a known hardware issue. We are trying to work with Movidius to find a workaround. Currently, if the Movidius processor gets voltage before the hub itself, then it fails to detect the USB speed correctly.
@radfordi Thanks so much for the information!
@rdinoff. @jmachuca77 I am not an Intel employee, so are not on the inside loop on internal discussions. So I rely on the brains of kind fellows like @radfordi to fill in my knowledge gaps. :)
Thanks for the information, I also have the same problem
@rdinoff, this is a known hardware issue. We are trying to work with Movidius to find a workaround. Currently, if the Movidius processor get voltage before the hub itself, then it fails to detect the USB speed correctly.
@radfordi Thanks .. when you have updated firmware let me know and I will be more then happy to check it out.
@radfordi Thanks please keep us updated on the progress, I'd like to keep this open so we know when the fix is in place, and we can test and close this issue. Thanks again.
@rdinoff, this is a known hardware issue. We are trying to work with Movidius to find a workaround. Currently, if the Movidius processor get voltage before the hub itself, then it fails to detect the USB speed correctly.
Is there a rough estimate of when the problem will be fixed? Days, weeks or months? We would like to use the t265 on some production system and we need to make a decision soon.
Following... Same Issue on my mini powered hub
Sequence from cold start [ 5.185719] hub 1-1:1.0: 4 ports detected [ 5.519724] usb 1-1.1: new high-speed USB device number 3 using ehci-platform [ 5.689179] usb 1-1.1: config 1 interface 0 altsetting 0 bulk endpoint 0x81 has invalid maxpacket 64 [ 5.707505] usb 1-1.1: config 1 interface 0 altsetting 0 bulk endpoint 0x1 has invalid maxpacket 64 [ 5.740710] usb 1-1.1: New USB device found, idVendor=03e7, idProduct=2150, bcdDevice= 0.01 [ 5.757399] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 5.772002] usb 1-1.1: Product: Movidius MA2X5X [ 5.781053] usb 1-1.1: Manufacturer: Movidius Ltd. [ 5.790624] usb 1-1.1: SerialNumber: 03e72150
disconnect reconnects [ 496.769441] usb 1-1.1: USB disconnect, device number 3 [ 502.624214] usb 1-1.1: new high-speed USB device number 4 using ehci-platform [ 502.795307] usb 1-1.1: New USB device found, idVendor=03e7, idProduct=2150, bcdDevice= 0.01 [ 502.795324] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 502.795330] usb 1-1.1: Product: Movidius MA2X5X [ 502.795335] usb 1-1.1: Manufacturer: Movidius Ltd. [ 502.795340] usb 1-1.1: SerialNumber: 03e72150 [ 506.633665] usb 1-1.1: USB disconnect, device number 4 [ 507.133848] usb 1-1.1: new high-speed USB device number 5 using ehci-platform [ 507.285287] usb 1-1.1: New USB device found, idVendor=8087, idProduct=0b37, bcdDevice=ff.ff [ 507.285298] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 507.285304] usb 1-1.1: Product: Intel(R) RealSense(TM) Tracking Camera T265 [ 507.285310] usb 1-1.1: Manufacturer: Intel(R) Corporation [ 507.285315] usb 1-1.1: SerialNumber: 9053121115070000
Well... that could be a short term solution: uhubctl is utility to control USB power per-port on smart USB hubs. Smart hub is defined as one that implements per-port power switching. https://github.com/mvp/uhubctl
@radfordi Jim, here is my '' Plan B '' to get the T265 flying without having to disconnect-reconnect
Please note the Masking Tape Skirt to reduce direct sunlight exposure during outdoor tests
hi @jmachuca77 I have the same issue, how did you get around this problem ?
didn't, its still an issue.
hi @ev-mp @dorodnic when is Intel planning to patch this issue ?
At this point in time, the workaround is to make sure that the host platform or USB hub has power and ready for enumeration prior to power provided to the T265.
Datasheet Statement (https://dev.intelrealsense.com/docs/tracking-camera-t265-datasheet): If T261/T265 is being connected to host system via USB hub, keep in mind that T261/T265 will enter into USB enumeration protocol as soon as 5V has been provided on VBUS pin. USB protocol is handled by the Movidius MA215x device. Make sure USB hub is connected and enumerated to host system prior to power being supplied to T261/T265.
Why closing the issue? Does this mean this problem will not be solved? This make it hard to use in a production ready system
I have to agree with @doisyg ! Its a real pain to get the T265 going! Having to hack a cable, and having scripts to power on/of the camera is just not a solution.
Same for us, @RealSenseCustomerSupport you have now released the T261 for production environment but this issue make it really difficult for us to integrate it in our product. We have done complex HW modification to accommodate this lack of flexibility for Intel. We hope that a FW patch will solve this issue soon !
@RealSenseCustomerSupport Can we have this issue open while waiting for a solution? The problem can be experienced on multiple platforms as already reported here by many people and elsewhere on the internet. We are currently experiencing the issue on APU2 board. The suggested workaround is of no help since we cannot control when 5V is introduced to different parts of the board. T265 is alone with this problem.
@rdinoff, this is a known hardware issue. We are trying to work with Movidius to find a workaround. Currently, if the Movidius processor gets voltage before the hub itself, then it fails to detect the USB speed correctly.
@rdinoff Is any solution to this on the horizon?
@rdinoff Is any solution to this on the horizon?
Err sorry, I meant to call out @radfordi.
If you want I can share with you the hardware solution we have put in place to control the 5V alimentation using GPIO of the single board computer.
@RealSenseCustomerSupport - Please, don't be stupid. This is a USB sensor most commonly connected to some embedded/robotic device, there is every reasonable expectation that it will be plugged in at power/boot time and expected to work. This is a fault in your product, pure and simple. This is a recall level fault, if you can't fix it in firmware.
I can confirm that with an RPI4 at least, @patrickpoirier51's suggestion of using uhubctrl to power cycle the USB hub does work. I've commented on this issue as well: https://github.com/IntelRealSense/librealsense/issues/5527
@fnoop, the T265 comes with a 1yr warranty. Please contact customer support if you'd like to return your camera.
@zwn, the only solution we've found is to run uhubctrl
. We've considered doing the power cycle ourselves in librealsense
before trying to boot the camera to avoid needing a separate program, but even then, you would still need to have power control of the USB port for this to work.
Hi all, what's actually the real solution?
Same issue: T265 will not be detected if it was connected before booting the computer. I have to replug it to the computer after booting to let it work. Hope this issue could be solved soon.
TL;DR; soft reboots of the nano cause enumeration to fail. It can reliably be fixed using the command uhubctl -l 1-2 -p 1 -a cycle
if there is no hub plugged into the nano, and the t265 is plugged into ports 2-4, but not 1.
For the jetson nano developer kit, i was having inconsistent results with @radfordi's suggestion of using uhubctrl
.
It seems like uhubctrl
only effectively turns off the power for all dev kit USB ports on the nano if its issued for the first port on the 2-1 bus with ./uhubctl -l 1-2 -p 1 -a cycle
Also, I noticed that if I plugged the t265 into the 1st port, it would sometimes enumerate and sometimes not, indicating the race condition, with an error like:
$ rs-enumerate-devices
30/07 23:00:01,517 ERROR [547566735376] (tm-boot.h:39) Error booting T265
30/07 23:00:04,571 ERROR [547566735376] (tm-boot.h:39) Error booting T265
No device detected. Is it plugged in?
I also get the above error if I soft reboot the nano with sudo reboot
, regardless of the port i plug into.
However, plugging the t265 into any other port (2-4), and then cycling port 1 with uhubctl, i have repeatable behavior.
The error also occurs if the the t265 is plugged into another port, and that port is cycled with uhubctl. I assume this is because the port does not actually loose its power.
Listing hub and port state with uhubctl
$ ./uhubctl
Current status for hub 2-1 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports]
Port 1: 02a0 power 5gbps Rx.Detect
Port 2: 02a0 power 5gbps Rx.Detect
Port 3: 02a0 power 5gbps Rx.Detect
Port 4: 02a0 power 5gbps Rx.Detect
Current status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports]
Port 1: 0100 power
Port 2: 0100 power
Port 3: 0503 power highspeed enable connect [03e7:2150 Movidius Ltd. Movidius MA2X5X 03e72150]
Port 4: 0103 power enable connect [1d50:606f canable.io canable gs_usb 001D00385852430320363530]
Turn off port 1:
$ ./uhubctl -l 1-2 -p 1 -a off
Current status for hub 2-1 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports]
Port 1: 02a0 power 5gbps Rx.Detect
Sent power off request
New status for hub 2-1 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports]
Port 1: 00a0 off
Current status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports]
Port 1: 0100 power
Sent power off request
New status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports]
Port 1: 0000 off
List it again, notice the USB devices have disconnected.
$ ./uhubctl
Current status for hub 2-1 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports]
Port 1: 00a0 off
Port 2: 02a0 power 5gbps Rx.Detect
Port 3: 02a0 power 5gbps Rx.Detect
Port 4: 02a0 power 5gbps Rx.Detect
Current status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports]
Port 1: 0000 off
Port 2: 0100 power
Port 3: 0100 power
Port 4: 0100 power
Turn back on
$ # ./uhubctl -l 1-2 -p 1 -a on
Current status for hub 2-1 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports]
Port 1: 00a0 off
Sent power on request
New status for hub 2-1 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports]
Port 1: 02a0 power 5gbps Rx.Detect
Current status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports]
Port 1: 0000 off
Sent power on request
New status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports]
Port 1: 0100 power
List again, devices reconnect:
./uhubctl
Current status for hub 2-1 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports]
Port 1: 02a0 power 5gbps Rx.Detect
Port 2: 02a0 power 5gbps Rx.Detect
Port 3: 02a0 power 5gbps Rx.Detect
Port 4: 02a0 power 5gbps Rx.Detect
Current status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports]
Port 1: 0100 power
Port 2: 0100 power
Port 3: 0503 power highspeed enable connect [03e7:2150 Movidius Ltd. Movidius MA2X5X 03e72150]
Port 4: 0103 power enable connect [1d50:606f canable.io canable gs_usb 001D00385852430320363530]
Here is the output of dmesg -w
while the cycle command occurs, notice that the t265 disconnects and reconnects along with another usb device on port 4:
[16528.508935] usb 2-1: usb_suspend_both: status 0
[16528.509519] usb usb2: usb_suspend_both: status 0
[16528.684791] usb 2-1: usb_suspend_both: status 0
[16528.686002] usb usb2: usb_suspend_both: status 0
[16528.704161] usb usb2: usb_suspend_both: status 0
[16528.861067] usb 2-1: usb_suspend_both: status 0
[16529.168760] usb 2-1: usb_suspend_both: status 0
[16529.169469] usb usb2: usb_suspend_both: status 0
[16529.313631] usb 1-2.3: USB disconnect, device number 86
[16529.461699] usb 1-2.4: USB disconnect, device number 87
[16529.462256] gs_usb 1-2.4:1.0 can0: Couldn't shutdown device (err=-19)
[16529.648749] tegra-xusb-padctl 7009f000.xusb_padctl: power down UTMI pad 1
[16529.668747] usb 1-2: usb_suspend_both: status 0
[16531.316800] usb 2-1: usb_suspend_both: status 0
[16531.468778] usb 2-1: usb_suspend_both: status 0
[16531.469056] tegra-xusb-padctl 7009f000.xusb_padctl: power on UTMI pads 1
[16531.469169] usb usb2: usb_suspend_both: status 0
[16531.608749] tegra-xusb-padctl 7009f000.xusb_padctl: power down UTMI pad 1
[16531.628701] usb 1-2: usb_suspend_both: status 0
[16531.825162] tegra-xusb-padctl 7009f000.xusb_padctl: power on UTMI pads 1
[16532.032728] usb 1-2.3: new high-speed USB device number 88 using tegra-xusb
[16532.054736] usb 1-2.3: New USB device found, idVendor=03e7, idProduct=2150
[16532.054753] usb 1-2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[16532.054765] usb 1-2.3: Product: Movidius MA2X5X
[16532.054777] usb 1-2.3: Manufacturer: Movidius Ltd.
[16532.054787] usb 1-2.3: SerialNumber: 03e72150
[16532.352688] usb 1-2.4: new full-speed USB device number 89 using tegra-xusb
[16532.375323] usb 1-2.4: New USB device found, idVendor=1d50, idProduct=606f
[16532.375336] usb 1-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[16532.375346] usb 1-2.4: Product: canable gs_usb
[16532.375354] usb 1-2.4: Manufacturer: canable.io
[16532.375362] usb 1-2.4: SerialNumber: 001D00385852430320363530
[16532.377118] gs_usb 1-2.4:1.0: Configuring for 1 interfaces
[16533.116000] IPv6: ADDRCONF(NETDEV_CHANGE): can0: link becomes ready
Enumeration and disabling the power fails if there is a hub plugged into a port on the nano (haven't been able to test with one supported by uhubctl
), see this ticket https://github.com/mvp/uhubctl/issues/258
SOLVED
sudo sh -c "echo 141a0000.pcie > /sys/bus/platform/drivers/tegra194-pcie/unbind"
sudo sh -c "echo 141a0000.pcie > /sys/bus/platform/drivers/tegra194-pcie/bind"
Find the PCIe card to which the USB is connected. In my case 141a0000.pcie
. The above command will turn the power supply to the PCIe card off and then on thereby making my T265 tracker camera detectable.
Issue Description
Possibly related to #4284 When booting up a system the T265 is enumerated on the USB bus as 03e7:2150 which I believe is the Movidius Chip, so when trying to use the camera with any script like rs-enumerate-devices or rs-pose I get an error:
This is the sequence of events after boot: Running lsusb, rs-enumerate-devices and then lsusb again.
After unplugging and replugging the camera I get the correct behavior and everything works as expected
This the output from lsusb after replugging the camera, then rs-enumerate-devices, and then lsusb again:
This is the output from dmesg, Initial detection of the device is at
[ 12.274866]
camera is un-plugged and re-plugged in at[ 148.021150]
and I believe rs-enumerate-device triggers the second usb disconnect at[ 165.684438]
after which the camera is detected correctly: