Cannot reenumarate devices in same python process

[Teslafly]

It seems that once hidapi has been enumerated, it cannot detect / connect to new or replugged devices.

I have 2 examples. First, using the hid.enumerate() example code, I have an mcp2221 device disconnected and run

>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

Then I connect the mcp2221 and it shows up in lsusb:

$ lsusb -d 04d8:00dd 
Bus 001 Device 107: ID 04d8:00dd Microchip Technology, Inc. 

rerunning hid.enumerate() and the mcp2221 is not found:

>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

and I can't connect to the device either:

>>> #connect mcp2221 here
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "hid.pyx", line 113, in hid.device.open
    raise IOError('open failed')
OSError: open failed

but if I restart python I can see the device:

root@0b5c447f6028:/power-test/temp/cython-hidapi# python3
Python 3.7.10 (default, Feb 16 2021, 19:28:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hid
  keys = list(device_dict.keys())
    keys.sort()
    for key in keys:
        print("%s : %s" % (ke>>> 
y, device_dict[key]))
    print()>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 2
manufacturer_string : Power Engineering
path : b'0001:006b:02'
product_id : 221
product_string : Test Board
release_number : 256
serial_number : 01234567
usage : 0
usage_page : 0
vendor_id : 1240

interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

and I can connect to it:

>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering
>>> print("Product: %s" % h.get_product_string())
Product: Test Board
>>> print("Serial No: %s" % h.get_serial_number_string())
Serial No: 01234567

But if I unplug it and replug it, and close and reopen, I cannot reconnect to the same device:

...
>>> print("Serial No: %s" % h.get_serial_number_string())
Serial No: 01234567
>>> #replug here:
>>> h.close()
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "hid.pyx", line 113, in hid.device.open
    raise IOError('open failed')
OSError: open failed

It requires restarting python (or spawning a new python process) to get the device connected again. Though if you start python, import hid but do not use any functions, then connect your device and run an hid function it will connect/find your device just fine. Just after the first hid function has been run your device list is locked in.

Is this a limitation of libusb hidapi itself? or something this library can fix by properly reinitialising the hidapi library? It seems like the list of usb devices may be being cached and not updated.

This is running in a docker container (but works the same on a native ubuntu system) and I have built the hidapi library from source because it seemed like https://github.com/trezor/cython-hidapi/issues/91 might fix it. (Though I have not verified this source build library is the actual one being used vs pypi)

[todbot]

Another thing to check on Linux is the output of dmesg to see if the HID device in question is being grabbed by a device-specific driver and removing it from the HID subsystem so hidapi can't see it.

[Teslafly]

Disconnecting and reconnecting the mcp2221 to the system produces the below dmesg:

[944446.637428] usb 1-5.3.1: USB disconnect, device number 113
[944448.881642] usb 1-5.3.1: new full-speed USB device number 114 using xhci_hcd
[944448.977729] usb 1-5.3.1: New USB device found, idVendor=04d8, idProduct=00dd
[944448.977751] usb 1-5.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[944448.977762] usb 1-5.3.1: Product: Test Board
[944448.977774] usb 1-5.3.1: Manufacturer: Power Engineering
[944448.977785] usb 1-5.3.1: SerialNumber: 01234567
[944448.979421] cdc_acm 1-5.3.1:1.0: ttyACM0: USB ACM device
[944448.982831] hid-generic 0003:04D8:00DD.0057: hiddev0,hidraw2: USB HID v1.11 Device [Power Engineering Test Board] on usb-0000:00:14.0-5.3.1/input2

It does not seem to be getting grabbed by another driver. If I only restart python

$Ctrl - D
$python3
>>> #hid stuff

I can connect to it fine.

[todbot]

Yup, agreed. I think you're right that the patch to add hid_ext() could help.

[Teslafly]

except I'm pretty sure I compiled the version with that patch, and it still wasn't working.

root@0b5c447f6028:/power-test/temp/cython-hidapi# pip3 show hidapi
Name: hidapi
Version: 0.10.1
Summary: A Cython interface to the hidapi from https://github.com/libusb/hidapi
Home-page: https://github.com/trezor/cython-hidapi
Author: Pavol Rusnak
Author-email: pavol@rusnak.io
License: UNKNOWN
Location: /usr/local/lib/python3.7/site-packages/hidapi-0.10.1-py3.7-linux-x86_64.egg
Requires: setuptools
Required-by: power-mcp2221a

Has the version with the hid_ext() been pushed to pypi yet?

[prusnak]

except I'm pretty sure I compiled the version with that patch, and it still wasn't working.

You are not exiting the process, so hid_exit is not being called.

But anyways, enumerate should work just fine. You should not need to run hid_exit to be able to enumerate.

Has the version with the hid_ext() been pushed to pypi yet?

Not yet

[Teslafly]

So the thing I want to do is re enumerate a reconnected device without spawning a new python process. Ideally without disconnecting other devices as well (but can work around that).

I bumped the source version number and ensured that it had the hid_ext patch and that it was installed correctly. I reran the tests, it still doesn't automatically detect new devices, but calling hid_exit does seem to reinitialize things.

>>> import hid
>>> 
>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

>>> #plug in mcp2221
>>> hid.hidapi_exit()
>>> import hid
>>> 
>>> for device_dict in hid.enumerate():
...     keys = list(device_dict.keys())
...     keys.sort()
...     for key in keys:
...         print("%s : %s" % (key, device_dict[key]))
...     print()
... 
interface_number : 2
manufacturer_string : Power Engineering
path : b'0001:0076:02'
product_id : 221
product_string : Test Board
release_number : 256
serial_number : 01234567
usage : 0
usage_page : 0
vendor_id : 1240

interface_number : 0
manufacturer_string : ThingM
path : b'0001:000f:00'
product_id : 493
product_string : blink(1) mk2
release_number : 2
serial_number : 2000BE4C
usage : 0
usage_page : 0
vendor_id : 10168

>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering

So it is possible to reconnect to devices now, but you have to nuke everything else connected via hidapi to do so.

Also, calling hidapi_exit() with a device still connected produces a segmentation fault, or at least a core dump

$ python3
Python 3.7.10 (default, Feb 16 2021, 19:28:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hid
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering
>>> 
>>> h.close()
>>> hid.hidapi_exit()

$ python3
Python 3.7.10 (default, Feb 16 2021, 19:28:34) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hid
>>> h = hid.device()
>>> h.open(0x04D8, 0x00DD)  # TREZOR VendorID/ProductID
>>> print("Manufacturer: %s" % h.get_manufacturer_string())
Manufacturer: Power Engineering
>>>
>>> # h.close()
>>> hid.hidapi_exit()
python3: ../../libusb/io.c:2116: handle_events: Assertion `ctx->pollfds_cnt >= internal_nfds' failed.
Aborted (core dumped)

# I have also gotten "Segmentation fault (core dumped)"

[jonasmalacofilho]

I cannot reproduce this with the hidraw backend, it could be a bug in hidapi-libusb.

[Teslafly]

yeah, I am using libusb.

[jonasmalacofilho]

Actually, I still couldn't reproduce the enumeration issue with the libusb backend (using cython-hidapi 0.10.1, hidapi 0.10.1 and libusb 1.0.24; linux 5.11.4).

Plugging the device in and out should while running the program bellow should have showed me the problem, right?

import hid
import time

def enumerate_hids():
    print("HID devices found:")

    for x in hid.enumerate():
        print(
            "{:04x}:{:04x}  {} {}  at: {}".format(
                x["vendor_id"],
                x["product_id"],
                x["manufacturer_string"] or "(None)",
                x["product_string"] or "(None)",
                x["path"],
            )
        )

if __name__ == "__main__":
    while True:
        enumerate_hids()
        print("---")
        time.sleep(1)

---

Version directly using hidapi from C:

```c /* repro.c */ #include #include #include #include "hidapi/hidapi.h" int enumerate() { struct hid_device_info *head = hid_enumerate(0, 0); struct hid_device_info *cur = head; if (!head) return -EIO; printf("HID devices found:\n"); do { printf("- %04x:%04x %ls %ls at: %s\n", cur->vendor_id, cur->product_id, cur->manufacturer_string, cur->product_string, cur->path); cur = cur->next; } while (cur != NULL); hid_free_enumeration(head); return 0; } int main() { int ret; while (1) { ret = enumerate(); if (-ret) return -ret; printf("---\n"); sleep(1); } return 0; } ``` ``` $ gcc -o repro-libusb repro.c -lhidapi-libusb -g $ ./repro-libusb ```

[Teslafly]

yeah, that program looks like it should work. ... and it does, on the base os, but not in the docker container with the same setup. (removed devices never reappear until you restart the script/python, but not the container) so looks like this might be related to using a docker container.

The a dockerfile that should work to reproduce the error:

FROM python:3.8-buster

RUN apt-get update \
    && apt-get -y install --no-install-recommends \
    libudev-dev \
    libusb-1.0-0-dev \
    udev

RUN pip3 install hidapi
CMD /bin/bash

# docker build -t test_usbhidapi:local -f Dockerfile .
# docker run --rm -it --privileged -v /dev:/dev test_usbhidapi:local

Given that this now appears to potentially be related to passing devices through docker, and there is a workaround. It's possible you want to close this issue unless you feel like debugging that mess.

If I find out more useful information I will likely post it here.

It is interesting that restarting Python or hidapi_exit() works to refresh devices though.

[bearsh]

@Teslafly regarding Docker, you shouldn't need a privileged container or full access to /dev to have it working, all you need is something like this:

echo 'c 189:* rwm' > /sys/fs/cgroup/devices/docker/CONTAINER_HASH/devices.allow
docker run --rm -it -v /dev/bus:/dev/bus:ro test_usbhidapi:local

this works for me (I access multiple hid devices, which get plugged/unplugged by the user multiple times over the lifetime of the application, from a container using libusb). for more info see Marc Merlin's post http://marc.merlins.org/perso/linux/post_2018-12-20_Accessing-USB-Devices-In-Docker-_ttyUSB0_-dev-bus-usb-_-for-fastboot_-adb_-without-using-privileged.html

[amendelzon]

yeah, that program looks like it should work. ... and it does, on the base os, but not in the docker container with the same setup. (removed devices never reappear until you restart the script/python, but not the container) so looks like this might be related to using a docker container.

The a dockerfile that should work to reproduce the error:

FROM python:3.8-buster

RUN apt-get update \
    && apt-get -y install --no-install-recommends \
    libudev-dev \
    libusb-1.0-0-dev \
    udev

RUN pip3 install hidapi
CMD /bin/bash

# docker build -t test_usbhidapi:local -f Dockerfile .
# docker run --rm -it --privileged -v /dev:/dev test_usbhidapi:local

Given that this now appears to potentially be related to passing devices through docker, and there is a workaround. It's possible you want to close this issue unless you feel like debugging that mess.

If I find out more useful information I will likely post it here.

It is interesting that restarting Python or hidapi_exit() works to refresh devices though.

Hi @Teslafly , I'm having the same device reconnection issues only within a docker container. Did you manage to find a workaround? I haven't tried the hidapi_exit patch yet, but if possible I'd like to avoid it. Thanks, any help is greatly appreciated!

[mcuee]

I guess this is no longer an issue with cython-hidapi but rather docker container. So this can be closed, right?