search

xairy / raw-gadget

USB Raw Gadget — a low-level interface for the Linux USB Gadget subsystem
333 stars 35 forks source link

Error redirection a usb device to a qemu virtual machine #50

Closed kovalev0 closed 8 months ago

kovalev0 commented 8 months ago

The error is reproduced for both the master and dev branches, and for the module from the dev branch it is not possible to run the printer and keyboard example application, but a proxy device is successfully created using the usb_proxy application.

master: $ sudo ./keyboard ... ioctl(USB_RAW_IOCTL_EP_WRITE): Cannot send after transport endpoint shutdown

$ sudo ./printer ... ioctl(USB_RAW_IOCTL_EP_READ): Cannot send after transport endpoint shutdown ioctl(USB_RAW_IOCTL_EP_WRITE): Cannot send after transport endpoint shutdown

usbredir (example printer): $ sudo usbredirect --device 0525:a4a8 --as 127.0.0.1:4000 --verbose 5 (usbredirect:12580): usbredirect-ERROR **: 20:22:46.937: usbredirhost: error resetting device: LIBUSB_ERROR_NOT_FOUND Ловушка трассировки/останова

=====================================================================

dev: $ sudo ./keyboard

event: connect, length: 0 ep #0: name: ep1in-bulk addr: 1 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #1: name: ep2out-bulk addr: 2 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #2: name: ep5in-int addr: 5 type: ___ ___ int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #3: name: ep6in-bulk addr: 6 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #4: name: ep7out-bulk addr: 7 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #5: name: ep10in-int addr: 10 type: ___ ___ int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #6: name: ep11in-bulk addr: 11 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #7: name: ep12out-bulk addr: 12 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #8: name: ep15in-int addr: 15 type: ___ ___ int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #9: name: ep1out-bulk addr: 1 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #10: name: ep2in-bulk addr: 2 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #11: name: ep-aout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #12: name: ep-bin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #13: name: ep-cout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #14: name: ep-dout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #15: name: ep-ein addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #16: name: ep-fout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #17: name: ep-gin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #18: name: ep-hout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #19: name: ep-iout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #20: name: ep-jin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #21: name: ep-kout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #22: name: ep-lin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #23: name: ep-mout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 int_in: addr = 5

event: 6 (unknown), length: 0

fail: unknown event

$ sudo ./printer

event: connect, length: 0 ep #0: name: ep1in-bulk addr: 1 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #1: name: ep2out-bulk addr: 2 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #2: name: ep5in-int addr: 5 type: ___ ___ int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #3: name: ep6in-bulk addr: 6 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #4: name: ep7out-bulk addr: 7 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #5: name: ep10in-int addr: 10 type: ___ ___ int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #6: name: ep11in-bulk addr: 11 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #7: name: ep12out-bulk addr: 12 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #8: name: ep15in-int addr: 15 type: ___ ___ int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #9: name: ep1out-bulk addr: 1 type: ___ blk ___ dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #10: name: ep2in-bulk addr: 2 type: ___ blk ___ dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #11: name: ep-aout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #12: name: ep-bin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #13: name: ep-cout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #14: name: ep-dout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #15: name: ep-ein addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #16: name: ep-fout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #17: name: ep-gin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #18: name: ep-hout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #19: name: ep-iout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #20: name: ep-jin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #21: name: ep-kout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 ep #22: name: ep-lin addr: 255 type: ___ blk int dir : in ___ maxpacket_limit: 65535 max_streams: 16 ep #23: name: ep-mout addr: 255 type: ___ blk int dir : ___ out maxpacket_limit: 65535 max_streams: 16 bulk_out: addr = 2 bulk_in: addr = 1

event: 6 (unknown), length: 0

fail: unknown event

usb-proxy:

$ lsusb ... Bus 002 Device 004: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller ... $ sudo ./usb-proxy --vendor_id=0bda --product_id=0129 ... ep0: transferred 0 bytes (out) EP1(bulk_out): read 16 bytes from host EP82(bulk_in): wrote 4 bytes to host EP1(bulk_out): read 16 bytes from host EP82(bulk_in): wrote 4 bytes to host EP1(bulk_out): read 16 bytes from host EP82(bulk_in): wrote 4 bytes to host EP1(bulk_out): read 24 bytes from host EP1(bulk_out): read 32 bytes from host EP1(bulk_out): read 12 bytes from host EP1(bulk_out): read 24 bytes from host

new proxy-usb device: $ lsusb Bus 001 Device 003: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller ...

redir with virt-manager: usb_proxy redir_to_qemu err

(continue) $ sudo ./usb-proxy --vendor_id=0bda --product_id=0129 ... ioctl(USB_RAW_IOCTL_EP_READ): Cannot send after transport endpoint shutdown ioctl(USB_RAW_IOCTL_EP_WRITE): Cannot send after transport endpoint shutdown

xairy commented 8 months ago

Thanks for the report!

Re fail: unknown event: you need to refetch the dev branch, I fixed this issue recently.

Looks like the issue with running the examples is that usbredirect resets the device before connecting it to QEMU's guest, and the reset event (6) is not handled by the examples. See https://github.com/AristoChen/usb-proxy/issues/9#issuecomment-1726680983 for a related discussion.

I'm not sure whether it makes to add reset handling to existing examples or add another one to demonstrate proper reset handling.

usb-proxy also doesn't know how to handle the reset event yet. I was planning to try addressing this once we get https://github.com/AristoChen/usb-proxy/issues/9 resolved.

For now, could you share the command you use to run QEMU to enable USB redirection?

kovalev0 commented 8 months ago

For now, could you share the command you use to run QEMU to enable USB redirection?

#!/bin/bash

qemu-system-x86_64 \
-hda altlinux.qcow2 \
-m 4G \
-enable-kvm \
-cpu host,nx \
-monitor stdio \
-nic user,hostfwd=tcp::8888-:22 \
-device ich9-usb-ehci1,id=ehci,addr=1d.7,multifunction=on \
-device ich9-usb-uhci1,id=uhci-1,addr=1d.0,multifunction=on,masterbus=ehci.0,firstport=0 \
-device ich9-usb-uhci2,id=uhci-2,addr=1d.1,multifunction=on,masterbus=ehci.0,firstport=2 \
-device ich9-usb-uhci3,id=uhci-3,addr=1d.2,multifunction=on,masterbus=ehci.0,firstport=4 \
-chardev socket,id=usbredirchardev1,port=4000,host=127.0.0.1 \
-device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=4

The process of working in steps is described in this article:

https://www.altlinux.org/%D0%98%D1%81%D1%81%D0%BB%D0%B5%D0%B4%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5_%D0%B2%D0%BE%D0%B7%D0%BC%D0%BE%D0%B6%D0%BD%D0%BE%D1%81%D1%82%D0%B8_%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D0%BD%D0%B5%D0%BD%D0%B8%D1%8F_raw-gadget_%D0%BA%D0%B0%D0%BA_%D0%B8%D1%81%D1%82%D0%BE%D1%87%D0%BD%D0%B8%D0%BA%D0%B0_USB-%D1%82%D1%80%D0%B0%D1%84%D0%B8%D0%BA%D0%B0_%D0%B4%D0%BB%D1%8F_%D1%82%D0%B5%D1%81%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D1%8F_usbredir

xairy commented 8 months ago

I pushed a few changes to the dev branch that make keyboard.c handle the reset event. I haven't tested them with QEMU usbredir though, only with the usbreset tool. Please check if it works for you now.

Re the link you shared: interesting! Could you explain what is the purpose of redirecting a virtual USB device emulated with Raw Gadget on host into QEMU guest? You could emulate a device within the QEMU guest directly. Or do you want to use a Raw Gadget-based proxy to MitM a real device while redirecting its traffic to the QEMU guest?

kovalev0 commented 8 months ago

I pushed a few changes to the dev branch that make keyboard.c handle the reset event. I haven't tested them with QEMU usbredir though, only with the usbreset tool. Please check if it works for you now.

The module and the keyboard example (sleep(1) -> sleep(10) ) are compiled from the dev branch. The redirected device falls off when the virtual machine desktop is loaded or before the user is invited to the terminal in multi-user mode, however, if you go to the grub menu, the 'x' symbol is printed. Moreover, such a run does not always work the same way and falls with different errors. I attach one of the falls to the picture and log files:

keyboard_to_qemu_usbredirect_0

device: keyboard_0.txt

$ sudo ./keyboard
...
ep_int_in: key up: 8
ep_int_in: key down: 8
ep_int_in: key up: 8
event: reset
ep0: stopping ep_int_in thread
ep_int_in: thread interrupted, exiting
ep0: stopped ep_int_in thread
event: 5 (unknown), length: 0
ep0: ignoring unknown event
event: control, length: 8
  bRequestType: 0x80 (IN), bRequest: 0x6, wValue: 0x100, wIndex: 0x0, wLength: 64
  type = USB_TYPE_STANDARD
  req = USB_REQ_GET_DESCRIPTOR
  desc = USB_DT_DEVICE
ioctl(USB_RAW_IOCTL_EP0_WRITE): Value too large for defined data type

or

...
ep_int_in: key down: 8
ep_int_in: key up: 8
event: reset
ep0: stopping ep_int_in thread
ep_int_in: thread interrupted, exiting
ep0: stopped ep_int_in thread
ioctl(USB_RAW_IOCTL_EVENT_FETCH): Invalid argument

or

...
event: reset
ep0: stopping ep_int_in thread
usb_raw_ep_write_may_fail(): Cannot send after transport endpoint shutdown

usbredir:

$ sudo /usr/bin/usbredirect --device 046d:c312 --as 127.0.0.1:4000

(usbredirect:4336): usbredirect-CRITICAL **: 17:06:55.685: connection_handle_io_cb: Failed to read guest

qemu: qemu_0.txt

...
qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330560

qemu: usb-redir: adding packet id 17330560 to cancelled queue

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330272

qemu: usb-redir: detaching device

qemu: usb-redir: adding packet id 17330272 to cancelled queue

qemu: usb-redir: removing 2 packet-ids from cancelled queue

qemu: usb-redir: removing 0 packet-ids from already-in-flight queue

qemu: usb-redir: chardev close

qemu: usb-redir: removing 0 packet-ids from cancelled queue

qemu: usb-redir: removing 0 packet-ids from already-in-flight queue

qemu: usb-redir: destroying usbredirparser
...

usbdump (usage usbmon): usbdump_0.txt

$ sudo ./usbdump -d 046d:c312
  0.000000 <--5 intr
  1.828025 <--0 ctrl [0100 0000] 18: 1201 0002 0000 0040 6d04 12c3 0000 0001  0201
  1.832031 <--0 ctrl [0200 0000] 34: 0902 2200 0101 03c0 3209 0400 0001 0301  0104 0921 1001 0001 2241 0007 0585 0308  0005
  1.836036 <--0 ctrl [0302 0409] 4: 0403 7800
  1.836061 -->0 ctrl [0001 0000]
 18.054838 <--0 ctrl [0100 0000] 8: 1201 0002 0000 0040
 18.059836 <--0 ctrl [0200 0000] 9: 0902 2200 0101 03c0 32
 18.064836 <--0 ctrl [0200 0000] 34: 0902 2200 0101 03c0 3209 0400 0001 0301  0104 0921 1001 0001 2241 0007 0585 0308  0005
 18.067802 -->0 ctrl [0000 0000]
 18.071983 -->0 ctrl [0800 0000]
 18.079837 <--5 intr 8: 0000 1b00 0000 0000
 18.079839 <--5 intr 8: 0000 0000 0000 0000
 28.081838 <--5 intr 8: 0000 1b00 0000 0000
 28.081842 <--5 intr 8: 0000 0000 0000 0000

Re the link you shared: interesting! Could you explain what is the purpose of redirecting a virtual USB device emulated with Raw Gadget on host into QEMU guest?

The initial goal is fuzzing the usbredir library.

You could emulate a device within the QEMU guest directly. Or do you want to use a Raw Gadget-based proxy to MitM a real device while redirecting its traffic to the QEMU guest?

In the process of exploring the capabilities of raw_gadget and usb_proxy, there was also such an idea that at the beginning of the session, use redirection of a regular device and use a standard handler, and then send a packet with mutated data and monitor the reaction of qemu entities.

kovalev0 commented 8 months ago

Compared the logs with redirecting a real keyboard Bus 002 Device 008: ID 046d:c34b Logitech, Inc. USB Keyboard, attach: usbdump_real_keyboard_0.txt и qemu_real_keyboard_0.txt

At the time of the user's invitation, usbredir tries to execute qemu: usb-redir: set address 2, to which the keyboard emulator does not respond.

real:

...
qemu: usb-redir: reset device

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17317952

qemu: usb-redir: ctrl-in status 0 len 18 id 17317952

qemu: usb-redir: reset device

qemu: usb-redir: set address 2

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 18 id 17318144

qemu: usb-redir: ctrl-in status 0 len 18 id 17318144

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x200 index 0 len 9 id 17318336

qemu: usb-redir: ctrl-in status 0 len 9 id 17318336

qemu: usb-redir: Removed remote wake 046D:C34B
...

emulator:

...
qemu: usb-redir: reset device

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330560

qemu: usb-redir: adding packet id 17330560 to cancelled queue

qemu: usb-redir: reset device

qemu: usb-redir: ctrl-out type 0x80 req 0x6 val 0x100 index 0 len 64 id 17330272

qemu: usb-redir: detaching device

qemu: usb-redir: adding packet id 17330272 to cancelled queue

qemu: usb-redir: removing 2 packet-ids from cancelled queue

qemu: usb-redir: removing 0 packet-ids from already-in-flight queue

qemu: usb-redir: chardev close
...
xairy commented 8 months ago

Pushed another change to dev, could you test?

kovalev0 commented 8 months ago

Pushed another change to dev, could you test?

Thanks! Now keyboard redirection works fine both with qemu and via virt-manager.

xairy commented 8 months ago

Awesome! So Raw Gadget works with usbredir, the only thing is that the emulation code needs to handle the reset event properly.

Let's keep this issue open as a reminder for me to:

  1. Upstream the Raw Gadget patches from the dev branch;
  2. Add reset handling to usb-proxy.
AristoChen commented 8 months ago

Just noticed the discussion here :)

Probably not related to the issue, but I noticed that you are discussing about the reset. From my previous experience(I can't recall everything now, it was like 3-4 years ago), some USB devices behave very weird if they receive the reset event for the second time.

I was using https://github.com/usb-tools/USBProxy-legacy to do proxy, and I called libusb_reset_device() before expose the USB device to Host, but then Host send another reset event(which is a normal behaviour if the USB device is plugged to Host directly) to the device, which cause the device hang and not functioning

In the end, I added a config/variable to decide whether I shoud call libusb_reset_device() before expose the device to Host or not

xairy commented 8 months ago

@AristoChen Yeah, I wound't be surprised if some USB devices break on multiple resets :)

Most of them should handle them though. So I would say resetting the device if the host asks for it should be the default behavior of a USB proxy.

But yeah, this is not related to handling the reset event in Raw Gadget.

xairy commented 8 months ago

Sent the patches upstream and implemented usb-proxy support for resent handling (didn't send a PR yet).

xairy commented 8 months ago

Raw Gadget patches are now in usb-next, so I includes the changes into the master branch and updated the keyboard example to handle reset.

Closing this issue as resolved.

Handing of reset in USB proxy is tracked in https://github.com/AristoChen/usb-proxy/issues/9.