search

OpenKinect / libfreenect2

Open source drivers for the Kinect for Windows v2 device
2.08k stars 752 forks source link

can't open camera by index - when using Kinect2 has only camera for openpose model #1125

Open femifapo opened 4 years ago

femifapo commented 4 years ago

Overview Description: I am trying to use libfreenect2 with pytorch-openpose and Kinect2. I must say I actually have no issue using libfreenect2 on it's own. I am raising this issue here hoping that someone might be able to help me understand how to make all 3 parts work together. The pytorch-openpose model also works well independently.

Version, Platform, and Hardware Bug Found:

  1. git log -1 --oneline- fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

  2. uname -a - Linux femiNati2 5.3.0-61-generic #55~18.04.1-Ubuntu SMP Mon Jun 22 16:40:20 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

  3. lsusb -t -/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M | Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M | Port 2: Dev 3, If 1, Class=Vendor Specific Class, Driver=usbfs, 5000M | Port 2: Dev 3, If 2, Class=Audio, Driver=snd-usb-audio, 5000M | Port 2: Dev 3, If 3, Class=Audio, Driver=snd-usb-audio, 5000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M | Port 1: Dev 8, If 3, Class=Audio, Driver=snd-usb-audio, 480M | Port 1: Dev 8, If 1, Class=Video, Driver=uvcvideo, 480M | Port 1: Dev 8, If 2, Class=Audio, Driver=snd-usb-audio, 480M | Port 1: Dev 8, If 0, Class=Video, Driver=uvcvideo, 480M | Port 7: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M | Port 7: Dev 2, If 1, Class=Vendor Specific Class, Driver=, 12M | Port 8: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M | Port 14: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M |__ Port 14: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M

  4. lspci -nn -00:00.0 Host bridge [0600]: Intel Corporation Device [8086:3e30] (rev 0a) 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0a) 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model [8086:1911] 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10) 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10) 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10) 00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9560 [Jefferson Peak] [8086:a370] (rev 10) 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10) 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10) 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port 9 [8086:a330] (rev f0) 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a305] (rev 10) 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10) 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10) 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10) 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10) 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GV102 [10de:1e07] (rev a1) 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f7] (rev a1) 01:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad6] (rev a1) 01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad7] (rev a1) 02:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5002]

Steps to Reproduce:

  1. Plug in only the Kinect camera via USB 3 port
  2. Run the demo script from pytorch-openpose (https://github.com/Hzzone/pytorch-openpose)
  3. I get the error--> [ WARN:0] global /io/opencv/modules/videoio/src/cap_v4l.cpp (887) open VIDEOIO(V4L2:/dev/video2): can't open camera by index

Actual Results: Libfreenect2 works well independently. pytorch-openpose works well indenpendently as well, but only connects with the webcam and the frames are quite delayed.

Expected Results: require pytorch-openpose to connect to libfreenect2 and use the Kinect camera.

Reproducibility:

Additional Information:I am not sure if this step in the linux installation install would help, as I never used it.

You need to specify cmake -Dfreenect2_DIR=$HOME/freenect2/lib/cmake/freenect2 for CMake based third-party application to find libfreenect2.

floe commented 4 years ago

This isn't as straightforward as it might look at first... the Kinect doesn't appear as a standard webcam device, therefore OpenCV fails. Basically, you'd have to replace the OpenCV-based video source in pytorch-openpose with custom code based on libfreenect2. Alternatively, you could also give https://github.com/yoshimoto/gspca-kinect2 a try.

femifapo commented 4 years ago

@floe Thanks very much for pointing me towards the right direction. I am able to record a video with the Kinect2 now. but following further instructions here> https://github.com/yoshimoto/libk4w2. I am unable to get live streams. I get the error DRM_IOCTL_I915_GEM_APERTURE failed: Invalid argument and I see that you helped resolve a similar issue here> Beignet-opencl-icd no supported GPU found #606 , but my PC uses an Intel processor and Nvidia RTX 2080 Ti graphics card.

The full error given after running clinfo is DRM_IOCTL_I915_GEM_APERTURE failed: Invalid argument Assuming 131072kB available aperture size. May lead to reduced performance or incorrect rendering. get chip id failed: -1 [22] param: 4, val: 0 beignet-opencl-icd: no supported GPU found, this is probably the wrong opencl-icd package for this hardware

Please do you know how this can be resolved?

floe commented 4 years ago

The solution for #606 is 4 years old, I'm guessing that won't work anymore. beignet-opencl-icd is only for Intel GPUs, you'd have to install the corresponding OpenCL library for NVidia, I think that would be nvidia-opencl-icd-XXX (XXX == driver version).

femifapo commented 4 years ago

I used the command "dmesg --human --follow" based on solution number #492 and got the message below. I removed and Kinect2 usb and put it back in but still the same message.

Do you think the issue is with the motherboard bios? (After a recent software update, my Bluetooth stopped working and Intel support advised me to update the mother bios which I am reluctant to do because of this current project)

[ +0.000257] ACPI BIOS Error (bug): Could not resolve symbol [_SB.UBTC.RUCC], AE_NOT_FOUND (20190816/psargs-330) [ +0.000006] No Local Variables are initialized for Method [_PLD] [ +0.000001] No Arguments are initialized for method [_PLD] [ +0.000001] ACPI Error: Aborting method _SB.PCI0.XHC.RHUB.HS04._PLD due to previous error (AE_NOT_FOUND) (20190816/psparse-531) [ +0.001025] xhci_hcd 0000:00:14.0: xHCI Host Controller [ +0.000002] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2 [ +0.000002] xhci_hcd 0000:00:14.0: Host supports USB 3.1 Enhanced SuperSpeed [ +0.000023] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.04 [ +0.000001] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ +0.000000] usb usb2: Product: xHCI Host Controller [ +0.000001] usb usb2: Manufacturer: Linux 5.4.0-42-generic xhci-hcd [ +0.000001] usb usb2: SerialNumber: 0000:00:14.0 [ +0.000078] hub 2-0:1.0: USB hub found [ +0.000012] hub 2-0:1.0: 10 ports detected [ +0.000258] ACPI BIOS Error (bug): Could not resolve symbol [_SB.UBTC.RUCC], AE_NOT_FOUND (20190816/psargs-330) [ +0.000005] No Local Variables are initialized for Method [_PLD] [ +0.000000] No Arguments are initialized for method [_PLD] [ +0.000001] ACPI Error: Aborting method _SB.PCI0.XHC.RHUB.SS04._PLD due to previous error (AE_NOT_FOUND) (20190816/psparse-531) ... [ +0.007305] usb 1-14: device descriptor read/64, error -71 [ +0.236012] iwlwifi 0000:00:14.3: Collecting data: trigger 15 fired. [ +0.000079] usb 1-14: device descriptor read/64, error -71 [ +0.235822] usb 1-14: new full-speed USB device number 6 using xhci_hcd [ +0.012040] iwlwifi 0000:00:14.3: Not valid error log pointer 0x00000000 for Init uCode [ +0.000015] iwlwifi 0000:00:14.3: Fseq Registers: [ +0.000008] iwlwifi 0000:00:14.3: 0x6FB5CBF0 | FSEQ_ERROR_CODE [ +0.000008] iwlwifi 0000:00:14.3: 0x00000000 | FSEQ_TOP_INIT_VERSION [ +0.000009] iwlwifi 0000:00:14.3: 0xCB77A044 | FSEQ_CNVIO_INIT_VERSION [ +0.000008] iwlwifi 0000:00:14.3: 0x0000A384 | FSEQ_OTP_VERSION [ +0.000008] iwlwifi 0000:00:14.3: 0x720C1E4B | FSEQ_TOP_CONTENT_VERSION [ +0.000009] iwlwifi 0000:00:14.3: 0xDE1D1645 | FSEQ_ALIVE_TOKEN [ +0.000008] iwlwifi 0000:00:14.3: 0xEEE93970 | FSEQ_CNVI_ID [ +0.000008] iwlwifi 0000:00:14.3: 0x6B106B0E | FSEQ_CNVR_ID [ +0.000009] iwlwifi 0000:00:14.3: 0x01000100 | CNVI_AUX_MISC_CHIP [ +0.000041] iwlwifi 0000:00:14.3: 0xA5A5A5A2 | CNVR_AUX_MISC_CHIP [ +0.000073] iwlwifi 0000:00:14.3: 0xA5A5A5A2 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM [ +0.000074] iwlwifi 0000:00:14.3: 0xA5A5A5A2 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR [ +0.000177] iwlwifi 0000:00:14.3: SecBoot CPU1 Status: 0xa5a5a5a2, CPU2 Status: 0xa5a5a5a2 [ +0.000001] iwlwifi 0000:00:14.3: Failed to start INIT ucode: -110 [ +0.000036] iwlwifi 0000:00:14.3: Firmware not running - cannot dump error [ +0.012864] iwlwifi 0000:00:14.3: Failed to run INIT ucode: -110 [ +0.102642] usb 1-14: device descriptor read/64, error -71 [ +0.243971] usb 1-14: device descriptor read/64, error -71 [ +0.108208] usb usb1-port14: attempt power cycle [ +0.651797] usb 1-14: new full-speed USB device number 7 using xhci_hcd [ +0.000159] usb 1-14: Device not responding to setup address. [ +0.207986] usb 1-14: Device not responding to setup address. [ +0.207842] usb 1-14: device not accepting address 7, error -71 [ +0.128032] usb 1-14: new full-speed USB device number 8 using xhci_hcd [ +0.000125] usb 1-14: Device not responding to setup address. [ +0.208041] usb 1-14: Device not responding to setup address. [ +0.207871] usb 1-14: device not accepting address 8, error -71 [ +0.000462] usb usb1-port14: unable to enumerate USB device ... [ +0.517570] uvcvideo: Failed to query (GET_INFO) UVC control 2 on unit 1: -110 (exp. 1). [ +0.074851] uvcvideo: Failed to query (GET_INFO) UVC control 7 on unit 2: -32 (exp. 1). [ +0.119666] usb 1-2: USB disconnect, device number 9 [ +0.000005] uvcvideo: Failed to query (GET_INFO) UVC control 8 on unit 2: -19 (exp. 1). [ +0.000005] uvcvideo: Failed to query (GET_INFO) UVC control 9 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 10 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 1 on unit 2: -19 (exp. 1). [ +0.000002] uvcvideo: Failed to query (GET_INFO) UVC control 4 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 5 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 11 on unit 2: -19 (exp. 1). [ +0.000008] uvcvideo: UVC non compliance - GET_DEF(PROBE) not supported. Enabling workaround. [ +0.000002] uvcvideo: Failed to query (129) UVC probe control : -19 (exp. 26). [ +0.000001] uvcvideo: Failed to initialize the device (-5). [ +0.000123] usbcore: registered new interface driver uvcvideo [ +0.000001] USB Video Class driver (1.1.1) [ +0.321164] usb 1-2: new high-speed USB device number 10 using xhci_hcd [ +0.152940] usb 1-2: New USB device found, idVendor=05a3, idProduct=9331, bcdDevice= 1.00 [ +0.000002] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ +0.000002] usb 1-2: Product: HD Web Camera [ +0.000001] usb 1-2: Manufacturer: Ingenic Semiconductor CO., LTD. [ +0.000001] usb 1-2: SerialNumber: Ucamera001 [ +0.006744] uvcvideo: Found UVC 1.00 device HD Web Camera (05a3:9331) [ +0.025695] uvcvideo: Failed to query (GET_INFO) UVC control 6 on unit 2: -32 (exp. 1). [ +0.025909] uvcvideo: Failed to query (GET_INFO) UVC control 9 on unit 2: -32 (exp. 1). [ +0.025951] uvcvideo: Failed to query (GET_INFO) UVC control 4 on unit 2: -32 (exp. 1). [ +0.020693] uvcvideo: Failed to query (GET_INFO) UVC control 11 on unit 2: -32 (exp. 1). [ +0.026612] uvcvideo 1-2:1.0: Entity type for entity Processing 2 was not initialized! [ +0.000003] uvcvideo 1-2:1.0: Entity type for entity Camera 1 was not initialized! [ +0.000097] input: HD Web Camera: HD Web Camera as /devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/input/input20 [ +0.001808] usb 1-2: Warning! Unlikely big volume range (=3072), cval->res is probably wrong. [ +0.000003] usb 1-2: [6] FU [Mic Capture Volume] ch = 1, val = -1536/1536/1 [ +1.044295] uvcvideo: Failed to query (GET_DEF) UVC control 6 on unit 2: -32 (exp. 2). [Aug 1 15:16] usb 2-1: USB disconnect, device number 3 [ +3.631776] usb 2-1: new SuperSpeed Gen 1 USB device number 4 using xhci_hcd [ +0.024881] usb 2-1: New USB device found, idVendor=045e, idProduct=02c4, bcdDevice= 1.00 [ +0.000005] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=4 [ +0.000004] usb 2-1: Product: Xbox NUI Sensor [ +0.000003] usb 2-1: Manufacturer: Microsoft [ +0.000003] usb 2-1: SerialNumber: 245577533947 [ +5.509513] uvcvideo: Failed to query (GET_DEF) UVC control 6 on unit 2: -32 (exp. 2). [Aug 1 15:18] usb 2-1: reset SuperSpeed Gen 1 USB device number 4 using xhci_hcd [ +10.773949] uvcvideo: Failed to query (GET_DEF) UVC control 6 on unit 2: -32 (exp. 2).

...

[ +0.517570] uvcvideo: Failed to query (GET_INFO) UVC control 2 on unit 1: -110 (exp. 1). [ +0.074851] uvcvideo: Failed to query (GET_INFO) UVC control 7 on unit 2: -32 (exp. 1). [ +0.119666] usb 1-2: USB disconnect, device number 9 [ +0.000005] uvcvideo: Failed to query (GET_INFO) UVC control 8 on unit 2: -19 (exp. 1). [ +0.000005] uvcvideo: Failed to query (GET_INFO) UVC control 9 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 10 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 1 on unit 2: -19 (exp. 1). [ +0.000002] uvcvideo: Failed to query (GET_INFO) UVC control 4 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 5 on unit 2: -19 (exp. 1). [ +0.000003] uvcvideo: Failed to query (GET_INFO) UVC control 11 on unit 2: -19 (exp. 1). [ +0.000008] uvcvideo: UVC non compliance - GET_DEF(PROBE) not supported. Enabling workaround. [ +0.000002] uvcvideo: Failed to query (129) UVC probe control : -19 (exp. 26). [ +0.000001] uvcvideo: Failed to initialize the device (-5). [ +0.000123] usbcore: registered new interface driver uvcvideo [ +0.000001] USB Video Class driver (1.1.1) [ +0.321164] usb 1-2: new high-speed USB device number 10 using xhci_hcd [ +0.152940] usb 1-2: New USB device found, idVendor=05a3, idProduct=9331, bcdDevice= 1.00 [ +0.000002] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ +0.000002] usb 1-2: Product: HD Web Camera [ +0.000001] usb 1-2: Manufacturer: Ingenic Semiconductor CO., LTD. [ +0.000001] usb 1-2: SerialNumber: Ucamera001 [ +0.006744] uvcvideo: Found UVC 1.00 device HD Web Camera (05a3:9331) [ +0.025695] uvcvideo: Failed to query (GET_INFO) UVC control 6 on unit 2: -32 (exp. 1). [ +0.025909] uvcvideo: Failed to query (GET_INFO) UVC control 9 on unit 2: -32 (exp. 1). [ +0.025951] uvcvideo: Failed to query (GET_INFO) UVC control 4 on unit 2: -32 (exp. 1). [ +0.020693] uvcvideo: Failed to query (GET_INFO) UVC control 11 on unit 2: -32 (exp. 1). [ +0.026612] uvcvideo 1-2:1.0: Entity type for entity Processing 2 was not initialized! [ +0.000003] uvcvideo 1-2:1.0: Entity type for entity Camera 1 was not initialized! [ +0.000097] input: HD Web Camera: HD Web Camera as /devices/pci0000:00/0000:00:14.0/usb1/1-2/1-2:1.0/input/input20 [ +0.001808] usb 1-2: Warning! Unlikely big volume range (=3072), cval->res is probably wrong. [ +0.000003] usb 1-2: [6] FU [Mic Capture Volume] ch = 1, val = -1536/1536/1 [ +1.044295] uvcvideo: Failed to query (GET_DEF) UVC control 6 on unit 2: -32 (exp. 2). [Aug 1 15:16] usb 2-1: USB disconnect, device number 3 [ +3.631776] usb 2-1: new SuperSpeed Gen 1 USB device number 4 using xhci_hcd [ +0.024881] usb 2-1: New USB device found, idVendor=045e, idProduct=02c4, bcdDevice= 1.00 [ +0.000005] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=4 [ +0.000004] usb 2-1: Product: Xbox NUI Sensor [ +0.000003] usb 2-1: Manufacturer: Microsoft [ +0.000003] usb 2-1: SerialNumber: 245577533947 [ +5.509513] uvcvideo: Failed to query (GET_DEF) UVC control 6 on unit 2: -32 (exp. 2). [Aug 1 15:18] usb 2-1: reset SuperSpeed Gen 1 USB device number 4 using xhci_hcd [ +10.773949] uvcvideo: Failed to query (GET_DEF) UVC control 6 on unit 2: -32 (exp. 2).

...

floe commented 4 years ago

[ +0.102642] usb 1-14: device descriptor read/64, error -71 and related errors sound a lot like a hardware issue, this means that not even USB enumeration works. Either try a different cable, or maybe you can indeed get lucky with updating your BIOS...