Open gbiggs opened 3 years ago
gbiggs
commented
3 years ago I forgot to mention that I have tried setting up something like the udev rules and script discussed in the installation instructions for Kernel VMs, but modified for LXD. It made no difference to the behaviour I'm getting.
Luxonis-Brandon
commented
3 years ago Sorry about the trouble @gbiggs . Unfortunately I don't personally know what LXD is (but I'll Google it shortly). What I'm thinking is happening though is that USB2 is being routed through to the container but USB3 is not. So this is why before boot you can see DepthAI, but not after.
This same thing will happen on Virtual Box for example if USB3 is not passed through. See here. And in Virtual Box case, it is necessary to actually run the pipeline to get the USB3 interface to show up (as it only shows up after DepthAI has booted over USB2) and then make sure to add it to pass through into Virtual Box.
So I'm wondering if a similar thing is necessary in LXD. But I'm guessing into the dark a little as I don't yet know what LXD is (will Google it shortly).
Thoughts?
Thanks, Brandon
Luxonis-Brandon
commented
3 years ago OK, learning a bit more about LXD. From my reading so far, I'm guessing our Docker container reference is probably what should be used to get LXD running, as it sounds like the libusb rework that was needed to work properly in Docker is likely also needed for LXD (since they seem to be very similar).
https://hub.docker.com/r/luxonis/depthai-library
And beyond that, I'll likely need the team to comment further.
Sorry again about the trouble.
Thanks, Brandon
gbiggs
commented
3 years ago I'm trying to mimic what the Dockerfile does in my LXD container, but I'm not sure where to install libusb to or what needs to build against it.
Luxonis-Brandon
commented
3 years ago Hi @gbiggs ,
Unfortunately I don't know it well enough - and I didn't do the work myself - but here is an overview of at least a bit of what was done to get Docker to work. I remember it being a confusing and fairly lengthy effort. And actually IIRC one of our customers is who actually figured it out and then kind of taught us the technique to do it.
That said, I think @themarpe may be able to provide some pointers. And also I think some recent work in libusb itself to make it so it's accessible in a non-priveleged manor (I think it requires root as of now) would help here. I think the main effort is to make it so libusb can be used in non-rooted ways in Android, but I think I remember a "hallway conversation" about how this would help make Docker (and thereby LXD) easier to make work.
https://github.com/libusb/libusb/pull/874
Thoughts?
Thanks, Brandon
gbiggs
commented
3 years ago I will keep playing around with libusb as I have time, but certainly some pointers would be helpful.
themarpe
commented
3 years ago @gbiggs I'd suggest you taking a look at this. https://community.intel.com/t5/Intel-Distribution-of-OpenVINO/NCS2-in-docker-container-on-RPI-results-in-RuntimeError-Can-not/m-p/1191905/highlight/true#M19803
It boils down to 2 steps:
And a third one which stems from first point - linking with newly built libusb.
This can be achieved in two ways:
Python wheels: Using LD_PRELOAD or specifying LD_LIBRARY_PATH to the folder where libusb without udev resides.
When working with prebuilt Python wheels, care must be taken as it packs the libusb library along side it, but it also mangles the library name. Example, depthai-2.9.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -> depthai.libs/libusb-1-150b88da.0.so.0.1.0. This means that freshly build libusb library has to be renamed accordingly as well.
Building from source (Python/C++): Other option is building the from source (check depthai-python/depthai-core for instructions) and pointing PATH to both include and lib directories of libusb (or just install the built library). This will pickup the new libusb library. Check depthai-python/ci/Dockerfile for reference.
Hopefully this maps to LXD neatly and helps you resolve the issue. If you'll have any additional information, feel free to tag and ask me:)
oto313
commented
3 years ago I had same problem.
this solution solves it. Thanks
gbiggs
commented
3 years ago @oto313 Did you have the same problem in an LXD container or in a Docker container?
@themarpe I've tried compiling libusb without udev support, and replaced the system-installed library with the new version (because I couldn't find proof that DepthAI was really finding the new one in the CMake output or cache). When I run an example from depthai-core I still get the same result: the OAK-D clicks, dmesg on the host shows it restarting with the new product ID of its new mode, and ls -R /dev/bus/usb in the container shows the initial USB device disappearing and the new USB device appearing, with correct access permissions. Nevertheless the example still gives the following output, and the OAK-D restarts back into its boot mode:
$ ./rgb_video
terminate called after throwing an instance of 'std::runtime_error'
what(): Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
zsh: abort (core dumped) ./rgb_videoThis is the device in /dev/:
/dev/bus/usb:
total 0
drwxr-xr-x 2 root root 40 Aug 13 07:02 001
drwxr-xr-x 2 root root 40 Aug 20 09:24 003
drwxr-xr-x 2 root root 60 Aug 20 09:24 004
/dev/bus/usb/001:
total 0
/dev/bus/usb/003:
total 0
/dev/bus/usb/004:
total 0
crw-rw-rw- 1 root root 189, 431 Aug 20 09:24 048
And here's dmesg output showing that device 48 is correct, and the restart back into boot mode (the disconnecting device number 88 is the boot mode device just before starting the example):
[3233363.346779] usb 3-2.3: USB disconnect, device number 88
[3233363.641018] usb 4-2.3: new SuperSpeed Gen 1 USB device number 48 using xhci_hcd
[3233363.665603] usb 4-2.3: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
[3233363.665606] usb 4-2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[3233363.665608] usb 4-2.3: Product: Luxonis Device
[3233363.665609] usb 4-2.3: Manufacturer: Intel Corporation
[3233363.665610] usb 4-2.3: SerialNumber: 14442C1061F95ED700
[3233371.460568] usb 4-2.3: USB disconnect, device number 48
[3233371.717366] usb 3-2.3: new high-speed USB device number 89 using xhci_hcd
[3233371.834706] usb 3-2.3: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[3233371.834710] usb 3-2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[3233371.834712] usb 3-2.3: Product: Movidius MyriadX
[3233371.834713] usb 3-2.3: Manufacturer: Movidius Ltd.
[3233371.834713] usb 3-2.3: SerialNumber: 03e72485@gbiggs Hi, I had problem in Docker container - arm32v7 arch
Client: Context: omni Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.5.1-docker) scan: Docker Scan (Docker Inc., v0.6.0)
Server: Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 17 Server Version: 19.03.15 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 3b3e9d5f62a114153829f9fbe2781d27b0a2ddac.m runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f-dirty init version: fec3683-dirty (expected: fec3683b971d9) Kernel Version: 5.5.19-a26c30ea-iota-devel Operating System: ------ OSType: linux Architecture: armv7l CPUs: 2 Total Memory: 3.708GiB Name: orion-305e-devel ID: APUF:7PEI:6SU3:ZE4D:Z3X3:36LO:34SN:WXF4:BC7Q:S22S:DORT:7YPA Docker Root Dir: /media/data/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
themarpe
commented
3 years ago @gbiggs All of the steps you've carried out above, these were run inside the LXD container?
Regarding libusb, if you linked to dynamic you can check using:
ldd depthai-core/examples/rgb_preview | grep usb`
# outputs
libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f391c9b7000)@themarpe Apart from dmesg, which can only be run in the host OS, all the other steps were carried out in the LXD container.
gbiggs
commented
3 years ago ldd depthai-core/examples/rgb_preview | grep usb shows that the example is linking to the correct version of libusb.
themarpe
commented
3 years ago @gbiggs not sure if you've tested this already, but can you try passing these settings. Seems similar as in Docker container use case, so I presume it could resolve your issue https://unix.stackexchange.com/a/137936
gbiggs
commented
3 years ago Thanks for the pointer to how to set a cgroup permission. I added it to my container instance using this command (LXD is slightly different from pure LXC):
$ lxc config set oakd_test raw.lxc="lxc.cgroup.devices.allow = c 189:* rwm"I confirmed that the configuration has been saved:
$ lxc config show oakd_test
architecture: x86_64
config:
image.architecture: amd64
image.description: |
Ubuntu focal
image.name: ubuntu-x86_64
image.os: ubuntu
image.release: focal
image.serial: "20210601_0155"
image.variant: default
raw.lxc: lxc.cgroup.devices.allow = c 189:* rwm
volatile.base_image: 4b5ebb97d70ed3d121ae8cd1c8137f8b68db5980e14f5015223168a90b90a5a8
volatile.eth0.host_name: vethba8797e8
volatile.eth0.hwaddr: 00:16:3e:08:35:1d
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000},{"Isuid":true,"Isgid":false,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":101001,"Nsid":1001,"Maprange":64535},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000},{"Isuid":false,"Isgid":true,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":101001,"Nsid":1001,"Maprange":64535}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000},{"Isuid":true,"Isgid":false,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":101001,"Nsid":1001,"Maprange":64535},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000},{"Isuid":false,"Isgid":true,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":101001,"Nsid":1001,"Maprange":64535}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000},{"Isuid":true,"Isgid":false,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":101001,"Nsid":1001,"Maprange":64535},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000},{"Isuid":false,"Isgid":true,"Hostid":1000,"Nsid":1000,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":101001,"Nsid":1001,"Maprange":64535}]'
volatile.last_state.power: RUNNING
volatile.uuid: 7073e7ef-3c8c-4561-8020-8e8327257ee2
devices:
oakd:
type: usb
vendorid: "03e7"
realsense:
type: usb
vendorid: "8086"
ephemeral: false
profiles:
- default
- container_marker
- gpu
- home_directory
- pulseaudio
- sshagent
- x11
stateful: false
description: ""I also restarted the instance, to be sure it took effect.
Running the rgv_video example or any other example still gives the same output as above, and the nodes in the device tree are identical as well. Possibly the lxc config command to add a USB device to an instance is adding the cgroup permissions under the hood already.
themarpe
commented
3 years ago @gbiggs
One extra thing worth checking out:
Use environment variable LIBUSB_DEBUG=4 inside the container when running a depthai application to get more debug information of libusb.
Check if the following message is printed out:
libusb: debug [linux_netlink_read_message] ignoring netlink message with non-zero sender UID 65534If so, there is a file libusb/os/linux_netlink.c with the following contents:
cred = (struct ucred *)CMSG_DATA(cmsg);
if (cred->uid != 0)
{
usbi_dbg("ignoring netlink message with non-zero sender UID %u", (unsigned int)cred->uid);
return -1;
}Remove the if statement check, recompile the libusb and depthai and retest.
If the above is not apparent, feel free to skim through libusb debug logs and try spotting anything unordinary. You may also diff compare it with a disabled udev libusb on host (otherwise messages won't be the same most likely)
gbiggs
commented
3 years ago That line doesn't appear in the debug output. I'll read through the log and see if I can find anything useful, and also try a diff.
Here's the debug output, in case you know what to look for. The last few lines are repeated forever until the application exits.
gbiggs
commented
3 years ago I've found a significant difference between the two logs. When running on the host, with a no-udev-libusb, the following line appears about 1.8s into the log:
[ 1.835360] [0003cd7b] libusb: debug [linux_netlink_read_message] netlink hotplug found device busnum: 4, devaddr: 17, sys_name: 4-2, removed: noThat device number is the OAK-D after rebooting into running-mode on the USB 3 bus. Following that line are many attempts to open the device, which fail with no access until (I assume) udev runs the rule that gives it access, then the device being opened, transfers happening, etc. The above line never happens inside the container. I think this means that the container is not properly announcing the new USB device being attached.
I've uploaded the log for the host so you can what I mean.
gbiggs
commented
3 years ago According to this article:
LXD USB devices support hotplug by default. So unplugging the device and plugging it back on the host will have it removed and re-added to the container.
gbiggs
commented
3 years ago Knowing a big more what to look for now, I've found that in the container, the following events are detected:
The removal of the boot-mode device when it reboots:
[ 1.445999] [0000164e] libusb: debug [linux_netlink_read_message] netlink hotplug found device busnum: 3, devaddr: 30, sys_name: 3-2, removed: yesThe appearance of the boot-mode device after the running-mode device gives up and reboots:
[10.156069] [0000164e] libusb: debug [linux_netlink_read_message] netlink hotplug found device busnum: 3, devaddr: 31, sys_name: 3-2, removed: noThe removal of the running-mode device when it gives up and reboots (note that this happens after the above line):
[10.467585] [0000164e] libusb: debug [linux_netlink_read_message] netlink hotplug found device busnum: 4, devaddr: 19, sys_name: 4-2, removed: yesNotably absent from the debug output in the container is the event for the running-mode device appearing, even though the device node in /dev/bus/usb is created properly, has the correct permissions, etc., and even though it gets an event for the device's removal from the USB bus.
Running on the host, the debug output does contain an event for the running-mode device appearance.
themarpe
commented
3 years ago @gbiggs can you try forcing USB2 mode in the script you are using? Its odd that only that message is missing. Maybe LXD has any options for that case? Also I saw that you have options for passing USB devices with vendor ID set to 0x03e7. I presume this goes for all devices, but note that the ROM Bootloader has product ID 0x2485 and running FW 0xf63b, just in case this comes up somewhere
gbiggs
commented
3 years ago can you try forcing USB2 mode in the script you are using?
I edited rgb_video.cpp to force USB2 mode and ran it, but it gives the same behaviour, except that the running-mode device shows up on the USB2 bus now instead of the USB3 bus (confirmed in dmesg).
Its odd that only that message is missing. Maybe LXD has any options for that case?
I agree, it's very odd that it consistently misses just that message. I haven't found any further information for LXD hotplugging other than "it just works"; which it does seem to mostly. It's just that one event...
Also I saw that you have options for passing USB devices with vendor ID set to 0x03e7. I presume this goes for all devices, but note that the ROM Bootloader has product ID 0x2485 and running FW 0xf63b, just in case this comes up somewhere
Adding a device with a vendor ID only makes the product ID a wildcard match, apparently. Just to be sure, though, I added devices for both product IDs specifically. It made no difference.
Sanjubisanal
commented
2 years ago @gbiggs is your issue resolved with the above fix ? I am also facing same issue as yours with intel movidius stick on LXD.
ParasPidurkar
commented
2 years ago @gbiggs Please provide us details regarding the resolution of the issue .
themarpe
commented
2 years ago @Sanjubisanal @ParasPidurkar You have the steps to reproduce here: https://github.com/lxc/lxd/issues/9136 It also seems that the issue was addressed, so it might work now. Give that a try and let us know how it goes.
Sanjubisanal
commented
2 years ago @themarpe We still have the same issue. On host:
root@raspberrypi4:/home/root/face_match# ./run-facedetect
Found stale device, resetting
Device 0 Address: 1.1 - VID/PID 03e7:2150
Device 1 Address: 1 - VID/PID 03e7:f63b
device attached
1.1
Starting wait for connect with 2000ms timeout
Found Address: 1.1 - VID/PID 03e7:2150
Found EP 0x81 : max packet size is 512 bytes
Found EP 0x01 : max packet size is 512 bytes
Found and opened device
Performing bulk write of 865724 bytes...
Successfully sent 865724 bytes of data in 128.084278 ms (6.445902 MB/s)
Boot successful, device address 1.1
Device 0 Address: 1 - VID/PID 03e7:f63b
Found Address: 1 - VID/PID 03e7:f63b
done
Booted 1 -> VSCInside LXD:
Device 0 Address: 1.1 - VID/PID 03e7:2150
device attached
1.1
Starting wait for connect with 2000ms timeout
Found Address: 1.1 - VID/PID 03e7:2150
Found EP 0x81 : max packet size is 512 bytes
Found EP 0x01 : max packet size is 512 bytes
Found and opened device
Performing bulk write of 865724 bytes...
Successfully sent 865724 bytes of data in 132.178011 ms (6.246264 MB/s)
Boot successful, device address 1.1
Traceback (most recent call last):
File "/usr/bin/facenet-pipe.py", line 269, in <module>
sys.exit(main())
File "/usr/bin/facenet-pipe.py", line 204, in main
device.OpenDevice()
File "/usr/bin/mvnc/mvncsapi.py", line 149, in OpenDevice
raise Exception(Status(status))
Exception: mvncStatus.ERROR@Sanjubisanal
Highly likely yes - do try that and make sure new library is taken instead of system libusb (check using ldd) If you are using Python, make sure to recompile locally so new libusb is taken into account. See the Dockerfile for reference: https://github.com/luxonis/depthai-python/blob/develop/ci/Dockerfile
Description of the bug
Running the DepthAI demo with
python3 depthai_demo.pyusing a USB-C OAK-D in an LXD container fails to find the device after booting. Instead it gives the following errors:This is the
dmesgoutput from plugging in the USB cable until the demonstration application terminates with an error.Running it again after this fails to even find the device in bootloader mode:
There is no
dmesgoutput while running the application this second time.Reconnecting the USB cable and running the demonstration application again gives the first result above, where the device is at first found, then lost.
Launching the same application in the host OS works fine. The camera image and disparity map are displayed correctly until
Ctrl-Cis pressed.This is the
dmesgoutput for when the application runs correctly on the host OS, from connecting the USB cable through to pressingCtrl-C:To Reproduce
Steps to reproduce the behavior:
depthai_demo.pydemonstration applicationExpected behavior
The demonstration application launches in an LXD container, opens the OAK-D device and displays the camera image and the disparity map.
Attach system log
Output of log_system_information.py: