ZED not working after boot but replugging it helps on J120 TX2

[ArkadiuszNiemiec]

I have a TX2+J120 carrier board with Logitech USB Camera connected to OTG Micro USB, LTE USB Modem connected to USB3 and ZED connected to the second USB3. Everything work but ZED camera requires me to remove and plug again after every reboot. Do you know what may be the cause of this?

Error before replugging: CAMERA DETECTION ISSUE

Full dmesg log: https://gist.github.com/ArkadiuszNiemiec/88a0116fc53319efc49b205a77b166e3 At around 120 seconds you can see me replugging the ZED camera.

Device found after reboot:

[    7.858892] usb 1-2: New USB device found, idVendor=2b03, idProduct=f681
[    7.866362] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    7.874230] usb 1-2: Product: ZED-M Hid Device
[    7.879310] usb 1-2: Manufacturer: STEREOLABS
[    7.884322] usb 1-2: SerialNumber: 10025962
[    7.889787] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 6
[    7.898718] hid-generic 0003:2B03:F681.0001: hidraw0: USB HID v1.11 Device [STEREOLABS ZED-M Hid Device] on usb-3530000.xhci-2/input0

Replugging:

[  117.945273] usb 1-2: USB disconnect, device number 2
[  120.021599] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 4
[  120.028642] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work ignore firmware MBOX_CMD_INC_SSPI_CLOCK request
[  120.130313] usb 1-2: new full-speed USB device number 6 using xhci-tegra
[  120.258795] usb 2-1: new SuperSpeed USB device number 2 using xhci-tegra
[  120.268201] usb 1-2: New USB device found, idVendor=2b03, idProduct=f681
[  120.274926] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  120.282143] usb 1-2: Product: ZED-M Hid Device
[  120.286674] usb 1-2: Manufacturer: STEREOLABS
[  120.291188] usb 1-2: SerialNumber: 10025962
[  120.296587] usb 2-1: New USB device found, idVendor=2b03, idProduct=f682
[  120.303389] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  120.310677] usb 2-1: Product: ZED-M
[  120.314418] usb 2-1: Manufacturer: Technologies, Inc.
[  120.320239] hid-generic 0003:2B03:F681.0002: hidraw0: USB HID v1.11 Device [STEREOLABS ZED-M Hid Device] on usb-3530000.xhci-2/input0
[  120.332746] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 6
[  120.343917] uvcvideo: Found UVC 1.10 device ZED-M (2b03:f682)
[  120.355831] input: ZED-M as /devices/3530000.xhci/usb2/2-1/2-1:1.0/input/input5

So after boot there's only ZED-M Hid Device. Restarting it did not help:

sudo usb_modeswitch -R -v 2b03 -p f681

or

echo -n "1-2:1.0" > /sys/bus/usb/drivers/usbhid/unbind
echo -n "1-2:1.0" > /sys/bus/usb/drivers/usbhid/bind

[plieningerweb]

Porblem was solved for me after message "CAMERA DETECTION ISSUE" by flipping the usb -C connector 180degree and plugging it in again.

BUT YOU DEFINITELY SHOULD FIX THAT IN THE FIMRWARE OR SOMEWERE OR AT LEAST GIVE CLEAR INSTRUCTIONS. FOund it only accidentially in the zed diagnotics tool where it says "flip or use usb3.0 port"

[molinadavid]

I have the exact same issue on a Jetson tx2, on boot up the camera does not work, but re connecting it solves the problem, however some time it takes more than one try.

[ArkadiuszNiemiec]

@plieningerweb it's true but it's also not the problem I described. @molinadavid what carrier board are you using? Dev board?

[molinadavid]

@ArkadiuszNiemiec I am using a Jetson TX2 Developer Kit.

[ArkadiuszNiemiec]

@molinadavid thanks, I was worried that it is the problem of my J120 carrier board.

[molinadavid]

@ArkadiuszNiemiec I can also confirm that is not a specific board, I have 7 different kits all present the same problem, also can confirm that this problem is only on the ZED-mini, I have also ZED normal type (the big one) and this problem does not happen.

[jonpol01]

This problem occurs too in a regular windows pc. Have tried both firmware 14xx & 15xx, but still, no camera detected and replugging the camera sometimes fixes it.

[ArkadiuszNiemiec]

Ok, so this is a general problem with ZED Mini, I will forward this issue to their tech support email.

[nesnes]

We were able to consistently reproduce the issue on Jetson and are looking for a software solution. From our initial tests, it seems that a powered USB 3.0 hub can fix this issue. @Myzhar got it working on reboot with the following one: https://www.amazon.it/CSL-alimentatore-SuperSpeed-compatibile-dispositivi/dp/B01NBJ6EDK/

[ArkadiuszNiemiec]

@nesnes Good news, thank you! Please keep us updated if you will find a fix. Do you know if it was fixed by the separate power rail or the USB Hub chipset?

[Myzhar]

@ArkadiuszNiemiec Removing the power cord from the hub the problem remains. That is a good USB3 HUB for ZED because it has a very short USB cable

[jonpol01]

@nesnes Actually using a self powered USB hub was our first workaround on it too.. But we found it very unpractical solution on our closed, compact mobile environment eg. Drone, Robot.

However, we appreciate your efforts on it, and are waiting for a real fix.

[WbHappy]

Hi @Myzhar,

Is there any update on this issue ? I also managed to reproduce it (jetson tx2 dev kit + ZED-M, classical ZED works fine at boot) and unfortunately the powered usb 3.0 is not a working solution for me :( !

[Myzhar]

@Wagdi-Benyaala the HW team is working on it. Stay tuned to further information

[WbHappy]

Thanks a lot for the fast answer !

[jeff-o]

We found we were able to adjust the "USB Power Delay" BIOS setting on our mini-ITX motherboard to 5 seconds to solve some of these issues.

[jonpol01]

@Myzhar Any news on this issue? we are on the verge of dropping the use of zed-mini and ask for refunds for 30units we bought in bulk from stereolabs.

[Myzhar]

@jonpol01 We are discussing with the Nvidia Embedded department to find a solution for this problem (maybe a kernel patch). We will update as soon as possible

[jonpol01]

@Myzhar any updates?

[obraun-sl]

The bug is in process in NVIDIA since this is related to the Jetpack boot configuration. Sorry for the inconvenience but we don't have the full control on the Jetpack. We are also looking at the jetpack kernel on our side to see if we can have a custom Jetpack to provide, but that would be best if it's official from NVIDIA.

[jonpol01]

@obraun-sl @Myzhar Im sorry but, if you gonna backtrack our reports of this bug, this too occurs in a ordinary PC environment. Its unfair to push the blame to Jetpack since its clear to us its Hardware Zed-mini problem related.

[ArkadiuszNiemiec]

@jonpol01 I can confirm what you wrote. Ubuntu 18.04 on Mi Notebook Pro 15" - same problem but solving it on TX2 will be enough for me.

Edit: missclicked 😞

[molinadavid]

@obraun-sl @Myzhar Any news regarding this? is getting really frustrating not having news or the fix to the problem, could you at least release a workaround for it? maybe a kernel patch or a custom Jetpack while the official comes out.

[obraun-sl]

Hi, We are currently working on a kernel patch. We have found that the problem does not happen on a TX1 , therefore we are looking at the difference to understand why it does not boot.

[molinadavid]

By the way I can confirm that the same issue still true on the newest Xavier as well.

[nominator]

any update on this? SDK 2.7.1 was released on 16th last week but there is no mention of this being fixed in the release notes. So I assume it is still up there.

However, I would like to know from people here who have experienced this issue that does it reliably comes back after replugging? I am asking this because I need to start work on a project and as long as it reliably get detected by replugging I can continue my work until a proper fix is provided.

Thanks

[ArkadiuszNiemiec]

However, I would like to know from people here who have experienced this issue that does it reliably comes back after replugging? I am asking this because I need to start work on a project and as long as it reliably get detected by replugging I can continue my work until a proper fix is provided.

Yes, it comes back after replugging, no problem after. For development it's OK, we just cannot apply it in production devices.

[nominator]

Thanks @ArkadiuszNiemiec. Big relief to know that :+1:

[AdarshK1]

Any update on this issue?

[molinadavid]

Since no official fix (or even updates on the issue) has been released I had success with a workaround that involves re building the TX2 kernel, decompiling/recompiling the device tree, removing the vbus regulator and triggering a script with a serviced to turn off and on the USB port on boot up.

For anyone interested on the rough steps, this is the thread on the Nvidia forum USB Power Control

[ahmad-aljabali]

I fond that launching the Zed Explorer (which doesn't require CUDA) will fix this problem without disconnecting and reconnecting.

I'm using Ubuntu 16.04, on an X86_64 pc. This simple bash script does the job for me so I can use it when the camera is inaccessible, but it doesn't solve the issue when multiple cameras are connected.

!/bin/bash

"/usr/local/zed/tools/ZED Explorer" & PID=jobs -p sleep 5 kill $PID

[adujardin]

The fix for the ZED-M detection has been integrated by NVIDIA into the new JetPack 4.2. The TX2 can now reboot with the ZED-M plugged in and it will be detected, like the ZED.

A compatible ZED SDK installer has been released for the TX2.

[jonpol01]

I wouldnt consider it a fix, since Jetpack 4.2 reflashes your tx2 system into ubuntu 18.04. Its a slap in our face thats says "go upgrade your 16.04 dev codes to 18.04"

However we did a workaround with the help from Nvidia guys on the developer forums to fix this ugly Zed mini hardware fault.

btw, stereolabs developers sucks on support and addressing this issue publicly.

We are considering lawsuits for this false advertising. "that it works on Ubuntu linux(jetpack and Zed sdk version from summer 2018 to this date) without doing workarounds on the dtb(which we solved for you guys)"

[ArkadiuszNiemiec]

@jonpol1 can you post a link to forums, where the problem was discussed? We have dropped the ZED Mini but I am still interested in the discussion.

[jonpol01]

@ArkadiuszNiemiec https://github.com/stereolabs/zed-ros-wrapper/issues/249#issuecomment-453387119

[obraun-sl]

@jonpol01 Thanks for your feedback.

I understand you're not happy that the fix requires you to update your Jetson to the new Jetpack 4.2 (which is based on Ubuntu 18). After discussion with the team and Nvidia, we can provide as well a patch to apply to the Jetpack 3.3 kernel (Ubuntu 16). It is a diff file (see attached) that you need to apply to the kernel sources. Once applied, you will need to rebuild the kernel and flash the Jetson with the new image. That way, you can keep using Jetpack 3.3 (and Ubuntu 16).

Nvidia will also backport the fix to JetPack 3.X, so that it does not require to rebuild the Jetpack. Again, sorry if we didn't not provide a fix quickly after the issue was discovered but we hope the above solution will help.

f8f9f49.zip

[ArkadiuszNiemiec]

Can someone test it so I can close this issue? My TX2 is currently without a carrier board.

[silent-code]

I have tested the current Jetpack 3.3 and can confirm that Jetpack 3.3 as of today, May 3rd 2019, does not fix this bug.

[tomcattiger1230]

You can try to add a rule in your system e.g. /etc/udev/rules.d/zedmini.rules // the name of rule is not important in this rule file, add a line to identify the id of the ZED device SUBSYSTEM=="usb", ATTR{idVendor}=="2b03", ATTR{idProduct}=="f681", MODE="0666" the two ids can be checked through lsusb. After adding this file, restart the TX2. In my case, I do not need to replug the usb cable after each new start.

[rhklite]

@tomcattigerkkk

You can try to add a rule in your system e.g. /etc/udev/rules.d/zedmini.rules // the name of rule is not important in this rule file, add a line to identify the id of the ZED device SUBSYSTEM=="usb", ATTR{idVendor}=="2b03", ATTR{idProduct}=="f681", MODE="0666" the two ids can be checked through lsusb. After adding this file, restart the TX2. In my case, I do not need to replug the usb cable after each new start.

this fix did not work for me. i also tried the following variations: SUBSYSTEM=="usb", ATTRS{idVendor}=="2b03", ATTRS{idProduct}=="f681", MODE="0666" SUBSYSTEM=="usb", ATTRS{idVendor}=="2b03", ATTRS{idProduct}=="f681", MODE:="0666" SUBSYSTEM=="usb", ATTR{idVendor}=="2b03", ATTR{idProduct}=="f681", MODE:="0666"

with reboot each time in between the changes.

is there anything that i might've missed?

ubuntu@tegra-ubuntu:~$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2b03:f681  
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

[obraun-sl]

@rhklite Please update to last Jetpack 4.2 to have the fix for the ZED Mini boot. It's not yet in previous jetpack. (The rule file is only for the HID device which is already detected at boot on your jetson (2b03:f681) )

[cosama]

Is there any news of this being backported to earlier Jetpack versions? Will it be necessary to reflash the TX2 or can the fix be applied through a software upgrade?

[Myzhar]

@cosama if the fix has been backported to older Jetpack versions a reflash is anyway required

[horverno]

there is a workaround for this issue: use an external supplied usb cable (hope this information is useful for somebody)

[ChealsieBains01]

I'm having this issue with the ZED Mini and Jetpack 4.6.

[amineKourta]

Same here ZED mini Jetpack 4.6 need to replug it everytime to get it working is there a workaround for this ?

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[maxpolzin]

Issue observed on Xavier NX on Auvidea JNX30 carrier board with Jetpack 5.0.1 and ZEDSDK 3.7 Turning the cable around fixed the issue.

[maxpolzin]

Also observed for the Seeedstudio A206 carrier with Jetpack 5.0.2 and ZEDSDK 3.8.