Closed LekKit closed 2 months ago
So according to ic2-hid
implementation if input_available()
is called twice, read_report()
should be called twice by the I2C-HID controller by holding an interrupt until the report queue is drained - which should have allowed my hid_keyboard
fix to work, but it doesn't exactly work on Linux guest.
If we immediately call input_available()
twice, sometimes second read_report()
call is made with huge (3-4 seconds) delay, and in between those calls the guest believes that the button is still pressed and performs repeated key input, etc.
What's even more intriguing is that interrupt line of I2C-HID device is asserted all this time and not lowered until guest reads the second report. It's not like the guest is overrun with interrupts either - numbers in /proc/interrupts
are around 10-100 mark, not gazzilions. So it must have ignore the raised interrupt of I2C-HID which signals there are more reports to be read, and only comes back after a huge delay.
I tried to test same VM with Haiku guest, no problems at all. Guest reads both reports immediately and registers a key click.
I went further and put repeated plic_send_irq()
each 10ms into I2C OC controller to make sure that device doesn't have any interrupts that guest might wait for (There was a bug a while ago, because it seems Linux expects more interrupt cases from this device than there are documented, like an interrupt on transaction stop). It fixed the issue! But ehm, I am not sure I2C controller is at fault... Any operation it could have done already signals an interrupt, I tried to send IRQ on any CRSR write except when it actually clears interrupts and nothing changed.
So here are possible causes of this particular bug
I will go have a look at how PLIC handles level-triggered IRQs as I have some second thoughts here.
@X547 can you give me any available pieces of documentation about I2C HID you have used / have access to? Seems problematic for me to find. It might be a good idea to have doc links around somewhere.
It is indeed a PLIC bug, plic_scan_irqs()
should be called when setting interrupt priority non-zero (Tho we also need to figure out what CTX this IRQ should be forwarded to).
Will include a fix soon.
I also want to redesign HID Device API somewhat. Instead of using report id queue, and relying on "read_report()` being called as many times as needed, why not queue complete report packets instead?
The old input_available()
function will only raise an interrupt, and report would be read once - this will be useful for mouse move events as many move events may be merged into a single IRQ and a single read_report()
if guest is slow. The read_report()
function will also return a bool indicating there are more reports to read (for future designs).
There will be another api, queue_report()
which will be called from input device side and handled by HID controller. It will queue entire report packet inside I2C HID (using a ring buffer), and allow bursts of input events without any complications inside actual input device implementations.
My fix to PLIC and initial fix for hid_keyboard will be merged beforehand.
Queuing events may cause other problems like shock wave effect when events are enqueued faster than processed.
@X547 can you give me any available pieces of documentation about I2C HID you have used / have access to? Seems problematic for me to find. It might be a good idea to have doc links around somewhere.
Queuing events may cause other problems like shock wave effect when events are enqueued faster than processed.
Ring buffer has limited size so they will be dropped in such case, current implementation in the repo technically works like a single buffered event inside input device state. With a ring buffer we can queue N events, but no more. It is beneficial for keyboard-like devices where loss of events upon burst will affect usability.
For devices which spam a lot of events (mouse movement) the older read_report()
API will be kept & optimized, so that multiple input events may coalesce into a single report with slower guest.
I think that would be a good combination however I want to fix PLIC first and upload initial hid_keyboard fix which doesn't change the design, only then experiment with anything new
Thanks!
There is another bug where Linux guest will sometimes hang if input_available
is called very early in the boot process (For example spamming input on a booting guest). It seems that I2C-HID interrupt is raised and never lowered. Perhaps the if (!i2c_hid->is_reset)
check isn't enough to prevent interrupt being sent too early?
Maybe Linux performs some i2c read/write operations before properly initializing interrupt handler? Is it possible to add logging of i2c_hid_stop
, i2c_hid_reset
calls?
Maybe Linux performs some i2c read/write operations before properly initializing interrupt handler? Is it possible to add logging of
i2c_hid_stop
,i2c_hid_reset
calls?
I have logged raise/lower events of IRQ 8 (assigned to I2C-HID) from PLIC when the state actually changes, and i2c_hid_start/stop events. This is combined with guest log so you can see where it's at.
The guest is not completely dead as the framebuffer tty cursor is blinking. However it is overflown with I2C-HID interrupts (It always claims them, but doesn't seem to perform any I2C HID activity after the last line in the log).
The actual interrupt storm reproduction occurs in window implementation and looks like this:
void fb_window_update(fb_window_t* win)
{
hid_keyboard_press(win->keyboard, HID_KEY_LEFTCTRL);
hid_keyboard_release(win->keyboard, HID_KEY_LEFTCTRL);
...
Point is, it sometimes manages to pass this and never hangs later on. But sometimes it hangs near i2c-hid init and it can be reproduced with user input. The timings have to be perfect, but it is still possible
I think I need to reproduce it myself. What Linux image/version is used?
I think I need to reproduce it myself. What Linux image/version is used?
Full setup (firmware/kernel/image). Run as rvvm fw_jump.bin -k Image -i rootfs.ext2
, also for the hang to be more likely do this:
void fb_window_update(fb_window_t* win)
{
for (size_t i=0; i<100000; ++i) {
hid_keyboard_press(win->keyboard, HID_KEY_LEFTCTRL);
hid_keyboard_release(win->keyboard, HID_KEY_LEFTCTRL);
}
...
It seems I managed to reproduce it and it seems caused by that Linux do not clear interrupt condition or disable interrupt during processing. It stuck in interrupt loop without any I2C IO.
In my Haiku implementation I schedule DPC and disable interrupt, so interrupt is reenabled after DPC processing is finished.
Haiku log:
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 0xa
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 0x51
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 00
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 00
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 0xa
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 00
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 00
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 00
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 0xa
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 0x51
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 00
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 00
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 0xa
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 00
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
rvvm: irq 8 disabled
rvvm: complete irq 8
i2c_hid_start(is_write: 1)
i2c_hid_write, io_offset: 0, val: 0x3
i2c_hid_write, io_offset: 1, val: 00
reg: 3
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 00
i2c_hid_read, io_offset: 1
val: 00
i2c_hid_read, io_offset: 2
val: 00
i2c_hid_read, io_offset: 3
val: 00
i2c_hid_read, io_offset: 4
val: 00
i2c_hid_read, io_offset: 5
val: 00
i2c_hid_read, io_offset: 6
val: 00
i2c_hid_read, io_offset: 7
val: 00
i2c_hid_read, io_offset: 8
val: 00
i2c_hid_read, io_offset: 9
val: 00
i2c_hid_stop
rvvm: irq 8 enabled
Interrupt condition should be cleared inside interrupt handler to avoid infinite interrupt loop, but I2C IO also use interrupts, so I can't find any other reliable solution except disabling I2C HID interrupt during processing it in DPC.
Note that in real hardware I2C HID device interrupt pin is usually connected to GPIO controller, not root interrupt controller. Linux may not properly handles cases where GPIO controller is not used. But I feel that using GPIO for I2C HID in emulator is wasteful.
Interrupt condition should be cleared inside interrupt handler to avoid infinite interrupt loop, but I2C IO also use interrupts, so I can't find any other reliable solution except disabling I2C HID interrupt during processing it in DPC.
So that's apparently a Linux bug? I will try with a newer kernel then.
Upd: Linux 6.7.9 still hangs the same way.
Problem may be caused by missing level triggered interrupt declaration in FDT. Second cell of interrupt controller data can be used to declare interrupt type, 4 is high active level interrupt.
See: https://www.kernel.org/doc/Documentation/devicetree/bindings/interrupt-controller/interrupts.txt
See: https://www.kernel.org/doc/Documentation/devicetree/bindings/interrupt-controller/interrupts.txt
Passing interrupts = <[IRQ] 4>;
or interrupts = <[IRQ] 5>;
(I.e. explaining to guest that this IRQ is level-triggered) doesn't fix the issue.
Using plic_send_irq()
instead of plic_raise_irq()
, plus sending IRQ again if we didn't empty the queue does fix the issue tho, Haiku guests are unaffected. Remind me again please what were the caveats of having edge-triggered I2C-HID interrupts?
Passing
interrupts = <[IRQ] 4>;
orinterrupts = <[IRQ] 5>;
(I.e. explaining to guest that this IRQ is level-triggered) doesn't fix the issue.
Have you changed #interrupt-cells
to 2?
Have you changed
#interrupt-cells
to 2?
Yes
I think we need to elaborate whether level-triggered interrupts are actually needed. Perhaps Linux driver actually expects edge-triggered interrupts.
Perhaps Linux driver actually expects edge-triggered interrupts.
No. i2c-hid-core.c
correctly handle level triggered interrupt by using IRQF_ONESHOT
flag and clearing interrupt condition in threaded interrupt handler. Something is wrong with setting PLIC level triggered interrupt mode.
IRQF_ONESHOT - Interrupt is not reenabled after the hardirq handler finished. Used by threaded interrupts which need to keep the irq line disabled until the threaded handler has been run.
See: https://www.kernel.org/doc/Documentation/devicetree/bindings/interrupt-controller/interrupts.txt
While the PLIC supports both edge-triggered and level-triggered interrupts,
interrupt handlers are oblivious to this distinction and therefore it is not
specified in the PLIC device-tree binding.
What happen if recompile kernel with following change?
- if (!irq_get_trigger_type(client->irq))
- irqflags = IRQF_TRIGGER_LOW;
+ irqflags = IRQF_TRIGGER_HIGH;
Also setting PLIC compatible
field to thead,c900-plic
and using 2 cells may work.
This patch seems fix problem (includes debug output):
diff --git a/src/devices/eth-oc.c b/src/devices/eth-oc.c
index d1a0974..123b087 100644
--- a/src/devices/eth-oc.c
+++ b/src/devices/eth-oc.c
@@ -543,7 +543,8 @@ PUBLIC rvvm_mmio_handle_t ethoc_init(rvvm_machine_t* machine, tap_dev_t* tap,
fdt_node_add_prop_reg(ethoc, "reg", base_addr, 0x800);
fdt_node_add_prop_str(ethoc, "compatible", "opencores,ethoc");
fdt_node_add_prop_u32(ethoc, "interrupt-parent", plic_get_phandle(plic));
- fdt_node_add_prop_u32(ethoc, "interrupts", irq);
+ uint32_t interrupt_cells[] = {irq, 4};
+ fdt_node_add_prop_cells(ethoc, "interrupts", interrupt_cells, 2);
fdt_node_add_child(rvvm_get_fdt_soc(machine), ethoc);
#endif
return handle;
diff --git a/src/devices/haiku_window.cpp b/src/devices/haiku_window.cpp
index 1d360d8..9dd215a 100644
--- a/src/devices/haiku_window.cpp
+++ b/src/devices/haiku_window.cpp
@@ -334,6 +334,12 @@ void fb_window_close(fb_window_t* win)
void fb_window_update(fb_window_t* win)
{
+#if 0
+ for (size_t i=0; i<100000; ++i) {
+ hid_keyboard_press(win->keyboard, HID_KEY_LEFTCTRL);
+ hid_keyboard_release(win->keyboard, HID_KEY_LEFTCTRL);
+ }
+#endif
View* view = win->data->wnd->GetView();
view->LockLooper();
view->Invalidate();
diff --git a/src/devices/i2c-hid.c b/src/devices/i2c-hid.c
index dd6a809..34ce0c2 100644
--- a/src/devices/i2c-hid.c
+++ b/src/devices/i2c-hid.c
@@ -316,7 +316,7 @@ static bool i2c_hid_start(void* dev, bool is_write)
{
i2c_hid_t* i2c_hid = (i2c_hid_t*)dev;
spin_lock(&i2c_hid->lock);
- //fprintf(stderr, "i2c_hid_start(is_write: %d)\n", is_write);
+ fprintf(stderr, "i2c_hid_start(is_write: %d)\n", is_write);
i2c_hid->is_write = is_write;
i2c_hid->io_offset = 0;
spin_unlock(&i2c_hid->lock);
@@ -327,15 +327,15 @@ static bool i2c_hid_write(void* dev, uint8_t byte)
{
i2c_hid_t* i2c_hid = (i2c_hid_t*)dev;
spin_lock(&i2c_hid->lock);
- //fprintf(stderr, "i2c_hid_write, io_offset: %u, val: %#02x\n", i2c_hid->io_offset, byte);
+ fprintf(stderr, "i2c_hid_write, io_offset: %u, val: %#02x\n", i2c_hid->io_offset, byte);
switch (i2c_hid->io_offset) {
case 0:
case 1:
i2c_hid->reg = bit_replace(i2c_hid->reg, i2c_hid->io_offset*8, 8, byte);
i2c_hid->io_offset++;
- //if (i2c_hid->io_offset == 2) {
- // fprintf(stderr, " reg: %u\n", i2c_hid->reg);
- //}
+ if (i2c_hid->io_offset == 2) {
+ fprintf(stderr, " reg: %u\n", i2c_hid->reg);
+ }
break;
default:
if (i2c_hid_write_reg(i2c_hid, i2c_hid->reg, i2c_hid->io_offset - 2, byte))
@@ -352,9 +352,9 @@ static bool i2c_hid_read(void* dev, uint8_t* byte)
{
i2c_hid_t* i2c_hid = (i2c_hid_t*)dev;
spin_lock(&i2c_hid->lock);
- //fprintf(stderr, "i2c_hid_read, io_offset: %u\n", i2c_hid->io_offset);
+ fprintf(stderr, "i2c_hid_read, io_offset: %u\n", i2c_hid->io_offset);
*byte = i2c_hid_read_reg(i2c_hid, i2c_hid->reg, i2c_hid->io_offset++);
- //fprintf(stderr, " val: %#02x\n", *byte);
+ fprintf(stderr, " val: %#02x\n", *byte);
spin_unlock(&i2c_hid->lock);
return true;
}
@@ -363,7 +363,7 @@ static void i2c_hid_stop(void* dev)
{
i2c_hid_t* i2c_hid = (i2c_hid_t*)dev;
spin_lock(&i2c_hid->lock);
- //fprintf(stderr, "i2c_hid_stop\n");
+ fprintf(stderr, "i2c_hid_stop\n");
i2c_hid->is_reset = false;
if (i2c_hid->command == I2C_HID_COMMAND_RESET) i2c_hid_reset(i2c_hid, false);
i2c_hid->reg = I2C_HID_INPUT_REG;
@@ -412,7 +412,8 @@ static void i2c_hid_init(rvvm_machine_t* machine, i2c_bus_t* bus, uint16_t addr,
fdt_node_add_prop_u32(i2c_fdt, "reg", addr);
fdt_node_add_prop_u32(i2c_fdt, "hid-descr-addr", I2C_HID_DESC_REG);
fdt_node_add_prop_u32(i2c_fdt, "interrupt-parent", plic_get_phandle(plic));
- fdt_node_add_prop_u32(i2c_fdt, "interrupts", irq);
+ uint32_t interrupt_cells[] = {irq, 4};
+ fdt_node_add_prop_cells(i2c_fdt, "interrupts", interrupt_cells, 2);
fdt_node_add_child(i2c_bus_fdt_node(bus), i2c_fdt);
#endif
}
diff --git a/src/devices/i2c-oc.c b/src/devices/i2c-oc.c
index c12ac31..3535476 100644
--- a/src/devices/i2c-oc.c
+++ b/src/devices/i2c-oc.c
@@ -225,7 +225,8 @@ PUBLIC i2c_bus_t* i2c_oc_init(rvvm_machine_t* machine, rvvm_addr_t base_addr, pl
fdt_node_add_prop_reg(i2c_fdt, "reg", base_addr, I2C_OC_REG_SIZE);
fdt_node_add_prop_str(i2c_fdt, "compatible", "opencores,i2c-ocores");
fdt_node_add_prop_u32(i2c_fdt, "interrupt-parent", plic_get_phandle(plic));
- fdt_node_add_prop_u32(i2c_fdt, "interrupts", irq);
+ uint32_t interrupt_cells[] = {irq, 4};
+ fdt_node_add_prop_cells(i2c_fdt, "interrupts", interrupt_cells, 2);
fdt_node_add_prop_u32(i2c_fdt, "clocks", fdt_node_get_phandle(i2c_clock));
fdt_node_add_prop_str(i2c_fdt, "clock-names", "clk");
fdt_node_add_prop_u32(i2c_fdt, "reg-shift", 2);
diff --git a/src/devices/ns16550a.c b/src/devices/ns16550a.c
index 478ef47..c078b50 100644
--- a/src/devices/ns16550a.c
+++ b/src/devices/ns16550a.c
@@ -230,7 +230,8 @@ PUBLIC rvvm_mmio_handle_t ns16550a_init(rvvm_machine_t* machine, chardev_t* char
fdt_node_add_prop_str(uart_fdt, "status", "okay");
if (plic) {
fdt_node_add_prop_u32(uart_fdt, "interrupt-parent", plic_get_phandle(plic));
- fdt_node_add_prop_u32(uart_fdt, "interrupts", irq);
+ uint32_t interrupt_cells[] = {irq, 4};
+ fdt_node_add_prop_cells(uart_fdt, "interrupts", interrupt_cells, 2);
}
fdt_node_add_child(rvvm_get_fdt_soc(machine), uart_fdt);
#endif
diff --git a/src/devices/pci-bus.c b/src/devices/pci-bus.c
index 3ead12f..cc0e1c0 100644
--- a/src/devices/pci-bus.c
+++ b/src/devices/pci-bus.c
@@ -295,25 +295,25 @@ PUBLIC pci_bus_t* pci_bus_init(rvvm_machine_t* machine, plic_ctx_t* plic, uint32
// Crossing-style IRQ routing for IRQ balancing
// INTA of dev 2 routes the same way as INTB of dev 1, etc
uint32_t plic_handle = plic_get_phandle(plic);
- uint32_t interrupt_map[96] = {
- 0x0000, 0, 0, 1, plic_handle, bus->irq[0],
- 0x0000, 0, 0, 2, plic_handle, bus->irq[1],
- 0x0000, 0, 0, 3, plic_handle, bus->irq[2],
- 0x0000, 0, 0, 4, plic_handle, bus->irq[3],
- 0x0800, 0, 0, 1, plic_handle, bus->irq[1],
- 0x0800, 0, 0, 2, plic_handle, bus->irq[2],
- 0x0800, 0, 0, 3, plic_handle, bus->irq[3],
- 0x0800, 0, 0, 4, plic_handle, bus->irq[0],
- 0x1000, 0, 0, 1, plic_handle, bus->irq[2],
- 0x1000, 0, 0, 2, plic_handle, bus->irq[3],
- 0x1000, 0, 0, 3, plic_handle, bus->irq[0],
- 0x1000, 0, 0, 4, plic_handle, bus->irq[1],
- 0x1800, 0, 0, 1, plic_handle, bus->irq[3],
- 0x1800, 0, 0, 2, plic_handle, bus->irq[0],
- 0x1800, 0, 0, 3, plic_handle, bus->irq[2],
- 0x1800, 0, 0, 4, plic_handle, bus->irq[1],
+ uint32_t interrupt_map[112] = {
+ 0x0000, 0, 0, 1, plic_handle, bus->irq[0], 4,
+ 0x0000, 0, 0, 2, plic_handle, bus->irq[1], 4,
+ 0x0000, 0, 0, 3, plic_handle, bus->irq[2], 4,
+ 0x0000, 0, 0, 4, plic_handle, bus->irq[3], 4,
+ 0x0800, 0, 0, 1, plic_handle, bus->irq[1], 4,
+ 0x0800, 0, 0, 2, plic_handle, bus->irq[2], 4,
+ 0x0800, 0, 0, 3, plic_handle, bus->irq[3], 4,
+ 0x0800, 0, 0, 4, plic_handle, bus->irq[0], 4,
+ 0x1000, 0, 0, 1, plic_handle, bus->irq[2], 4,
+ 0x1000, 0, 0, 2, plic_handle, bus->irq[3], 4,
+ 0x1000, 0, 0, 3, plic_handle, bus->irq[0], 4,
+ 0x1000, 0, 0, 4, plic_handle, bus->irq[1], 4,
+ 0x1800, 0, 0, 1, plic_handle, bus->irq[3], 4,
+ 0x1800, 0, 0, 2, plic_handle, bus->irq[0], 4,
+ 0x1800, 0, 0, 3, plic_handle, bus->irq[2], 4,
+ 0x1800, 0, 0, 4, plic_handle, bus->irq[1], 4,
};
- fdt_node_add_prop_cells(pci_node, "interrupt-map", interrupt_map, 96);
+ fdt_node_add_prop_cells(pci_node, "interrupt-map", interrupt_map, 112);
uint32_t interrupt_mask[4] = { 0x1800, 0, 0, 7 };
fdt_node_add_prop_cells(pci_node, "interrupt-map-mask", interrupt_mask, 4);
diff --git a/src/devices/plic.c b/src/devices/plic.c
index 1e36ee2..c13e674 100644
--- a/src/devices/plic.c
+++ b/src/devices/plic.c
@@ -23,6 +23,8 @@ along with this program. If not, see <https://www.gnu.org/licenses/>.
#include "mem_ops.h"
#include "atomics.h"
+#include <stdio.h>
+
#define PLIC_CTXFLAG_THRESHOLD 0x0
#define PLIC_CTXFLAG_CLAIMCOMPLETE 0x1
@@ -157,6 +159,7 @@ static uint32_t plic_claim_irq(plic_ctx_t* plic, uint32_t ctx)
static void plic_complete_irq(plic_ctx_t* plic, uint32_t ctx, uint32_t irq)
{
uint32_t raised = atomic_load_uint32(&plic->raised[irq >> 5]) & (1U << (irq & 0x1F));
+ fprintf(stderr, "rvvm: complete irq %" PRIu32 "\n", irq);
if (raised) {
// Rearm raised interrupt as pending after completion
atomic_or_uint32(&plic->pending[irq >> 5], raised);
@@ -217,6 +220,7 @@ static bool plic_mmio_write(rvvm_mmio_dev_t* dev, void* data, size_t offset, uin
uint32_t irq = offset >> 2;
if (irq > 0 && irq < PLIC_SOURCE_MAX) {
atomic_store_uint32(&plic->prio[irq], read_uint32_le(data));
+ fprintf(stderr, "rvvm: irq priority %" PRIu32 "\n", plic->prio[irq]);
plic_update_irq(plic, irq);
}
} else if (offset < 0x1080) {
@@ -228,7 +232,17 @@ static bool plic_mmio_write(rvvm_mmio_dev_t* dev, void* data, size_t offset, uin
uint32_t reg = ((offset - 0x2000) >> 2) & 0x1F;
uint32_t ctx = (offset - 0x2000) >> 7;
if (reg < PLIC_SRC_REG_COUNT && ctx < plic_ctx_count(plic)) {
+ uint32_t oldEnable = plic->enable[ctx][reg];
atomic_store_uint32(&plic->enable[ctx][reg], read_uint32_le(data));
+ uint32_t enable = plic->enable[ctx][reg];
+ for (uint32_t i = 0; i < 32; i++) {
+ uint32_t mask = 1 << i;
+ if ((oldEnable & mask) == 0 && (enable & mask) != 0)
+ fprintf(stderr, "rvvm: irq %" PRIu32 " enabled\n", 32*reg + i);
+
+ if ((oldEnable & mask) != 0 && (enable & mask) == 0)
+ fprintf(stderr, "rvvm: irq %" PRIu32 " disabled\n", 32*reg + i);
+ }
plic_update_ctx_irq_reg(plic, ctx, reg);
}
} else if (offset < 0x200000) {
@@ -242,6 +256,7 @@ static bool plic_mmio_write(rvvm_mmio_dev_t* dev, void* data, size_t offset, uin
plic_complete_irq(plic, ctx, read_uint32_le(data));
} else if (flag == PLIC_CTXFLAG_THRESHOLD) {
atomic_store_uint32(&plic->threshold[ctx], read_uint32_le(data));
+ fprintf(stderr, "rvvm: irq threshold %" PRIu32 "\n", plic->threshold[ctx]);
plic_update_ctx(plic, ctx);
}
}
@@ -323,9 +338,9 @@ PUBLIC plic_ctx_t* plic_init(rvvm_machine_t* machine, rvvm_addr_t base_addr)
}
struct fdt_node* plic_node = fdt_node_create_reg("plic", base_addr);
- fdt_node_add_prop_u32(plic_node, "#interrupt-cells", 1);
+ fdt_node_add_prop_u32(plic_node, "#interrupt-cells", 2);
fdt_node_add_prop_reg(plic_node, "reg", base_addr, 0x4000000);
- fdt_node_add_prop_str(plic_node, "compatible", "sifive,plic-1.0.0");
+ fdt_node_add_prop_str(plic_node, "compatible", "thead,c900-plic");
fdt_node_add_prop_u32(plic_node, "riscv,ndev", PLIC_SOURCE_MAX - 1);
fdt_node_add_prop(plic_node, "interrupt-controller", NULL, 0);
fdt_node_add_prop_cells(plic_node, "interrupts-extended", irq_ext, vector_size(machine->harts) * 4);
diff --git a/src/devices/ps2-altera.c b/src/devices/ps2-altera.c
index ad2dcb8..7d46b94 100644
--- a/src/devices/ps2-altera.c
+++ b/src/devices/ps2-altera.c
@@ -137,7 +137,8 @@ void altps2_init(rvvm_machine_t* machine, rvvm_addr_t base_addr, plic_ctx_t* pli
fdt_node_add_prop_reg(ps2, "reg", base_addr, ALTPS2_MMIO_SIZE);
fdt_node_add_prop_str(ps2, "compatible", "altr,ps2-1.0");
fdt_node_add_prop_u32(ps2, "interrupt-parent", plic_get_phandle(plic));
- fdt_node_add_prop_u32(ps2, "interrupts", irq);
+ uint32_t interrupt_cells[] = {irq, 4};
+ fdt_node_add_prop_cells(ps2, "interrupts", interrupt_cells, 2);
fdt_node_add_child(rvvm_get_fdt_soc(machine), ps2);
#endif
}
diff --git a/src/devices/rtc-goldfish.c b/src/devices/rtc-goldfish.c
index e71f27a..2a868fa 100644
--- a/src/devices/rtc-goldfish.c
+++ b/src/devices/rtc-goldfish.c
@@ -130,7 +130,8 @@ PUBLIC rvvm_mmio_handle_t rtc_goldfish_init(rvvm_machine_t* machine, rvvm_addr_t
fdt_node_add_prop_reg(rtc, "reg", base_addr, RTC_REG_SIZE);
fdt_node_add_prop_str(rtc, "compatible", "google,goldfish-rtc");
fdt_node_add_prop_u32(rtc, "interrupt-parent", plic_get_phandle(plic));
- fdt_node_add_prop_u32(rtc, "interrupts", irq);
+ uint32_t interrupt_cells[] = {irq, 4};
+ fdt_node_add_prop_cells(rtc, "interrupts", interrupt_cells, 2);
fdt_node_add_child(rvvm_get_fdt_soc(machine), rtc);
#endif
return handle;
Linux log:
rvvm: irq priority 0
rvvm: complete irq 9
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 0x8
rvvm: complete irq 5
i2c_hid_read, io_offset: 1
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 2
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 3
val: 0x33
rvvm: complete irq 5
i2c_hid_read, io_offset: 4
val: 0xd
rvvm: complete irq 5
i2c_hid_read, io_offset: 5
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 6
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 7
val: 00
rvvm: complete irq 5
i2c_hid_stop
rvvm: complete irq 5
rvvm: complete irq 5
rvvm: irq priority 1
rvvm: irq priority 0
rvvm: complete irq 9
i2c_hid_start(is_write: 0)
i2c_hid_read, io_offset: 0
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 1
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 2
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 3
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 4
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 5
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 6
val: 00
rvvm: complete irq 5
i2c_hid_read, io_offset: 7
val: 00
rvvm: complete irq 5
i2c_hid_stop
rvvm: complete irq 5
rvvm: complete irq 5
rvvm: irq priority 1
rvvm: complete irq 2
Also setting PLIC
compatible
field tothead,c900-plic
and using 2 cells may work.
In reality THead PLIC is subtly non spec conformant, and has a quirk PLIC_QUIRK_EDGE_INTERRUPT
in kernel driver.
Because of that quirk it has to encode actual interrupt behavior as two cells, and only for those chips kernel will decode it that way, as seen here: https://elixir.bootlin.com/linux/latest/source/drivers/irqchip/irq-sifive-plic.c#L327 (Which is why it didn't work when I tried it originally).
But this helps with debugging for now; I'll see what actually changes in kernel with this and perhaps either fix it or conclude it's a kernel bug with detailed info for a mailing list.
But this helps with debugging for now; I'll see what actually changes in kernel with this and perhaps either fix it or conclude it's a kernel bug with detailed info for a mailing list.
IRQF_ONESHOT
flag do not work without IRQF_TRIGGER_HIGH
/IRQF_TRIGGER_LOW
setting, which can't be set without PLIC_QUIRK_EDGE_INTERRUPT
in current Linux PLIC driver code.
But this helps with debugging for now; I'll see what actually changes in kernel with this and perhaps either fix it or conclude it's a kernel bug with detailed info for a mailing list.
IRQF_ONESHOT
flag do not work withoutIRQF_TRIGGER_HIGH
/IRQF_TRIGGER_LOW
setting, which can't be set withoutPLIC_QUIRK_EDGE_INTERRUPT
in current Linux PLIC driver code.
So basically an incompatibility of I2C-HID & SiFive PLIC?
So basically an incompatibility of I2C-HID & PLIC?
I think it is a Linux bug in PLIC driver so it do not properly support level triggered interrupts. Even if edge/level is hardcoded as hardware level, software still should know interrupt type for proper operation. I think that 2 interrupt cells should be always able to be specified, not only for T-Head PLIC.
As I understand real hardware SiFive PLIC do not expose interrupt pins outside of chip, so GPIO is the only option without redesigning hardware is to connect I2C HID device interrupt line.
Problem should be not present with APLIC because it supports software configurable interrupt types.
I will be heading to RISC-V Linux mailing list today to report this. Thanks for your cooperation.
Hopefully they will not reject this; If they do, it will either be wontfix
or we'll have to route I2C-HID IRQ line through SiFive GPIO (Which in turn will require changes on Haiku side?)
SiFive GPIO (Which in turn will require changes on Haiku side?)
Yes, but it is doable.
Yes, but it is doable.
If Haiku already has the infrastructure to route GPIO IRQs to I2C-HID (Which it should, as that's how most HW works), then it's a matter of adding SiFive GPIO driver which is not a complicated device at all. In future I can add JH7110 GPIO, etc on RVVM side.
It appears switching to THead PLIC fixes this for different reasons. I've checked all changes PLIC_QUIRK_EDGE_INTERRUPT
makes in the driver, and removing those 2 lines fixes the issues without patching RVVM.
https://elixir.bootlin.com/linux/latest/source/drivers/irqchip/irq-sifive-plic.c#L223
The issue is not fixed if I keep this return path enabled for THead PLIC and build RVVM with your patch... So perhaps a bug in RVVM PLIC? Altho I can't see how, really.
So perhaps a bug in RVVM PLIC? Altho I can't see how, really.
If there are no log output rvvm: irq xxx priority 0/1
or rvvm: irq xxx enabled/disabled
on each I2C HID IO transaction, it is definitely a software bug (wrong FDT definitions and/or Linux bug), not a problem of hardware emulation.
Also maybe SiFive PLIC is not supposed to support level-triggered interrupts at all, so it do not fully support features of PLIC spec. So setting T-Head PLIC may be not a bad choice.
Did you ask Linux kernel community about problem? If so, can you share a link?
Did you ask Linux kernel community about problem? If so, can you share a link?
Sorry I postponed this because I was still debugging related things. You can report this independently if you want, to riscv-linux
mailing list, I am really sloppy with this. Or should I instead hurry with this?
No hurry, just asked.
https://lists.infradead.org/pipermail/linux-riscv/2024-March/050429.html
I do hope that someone there will provide an explanation what the return path at irq-sivive-plic.c@223 does and possible fix in the kernel or perhaps a hint what (if anything) could be still wrong on RVVM side
I also figured that advertising THead PLIC in RVVM is not really a good idea moving forward, because it has additional functionality used by the M-mode firmware which we don't implement (kernel side changes are less invasive) , and it could break in future
Should I release a stable v0.6 in the meantime? It seems there are no immediate clues on the mailing list about what's wrong, but I also don't see what else we can do on the VM side.
Touching any PLIC/I2C-HID code at all right now seems not very wise because kernel dev (hopefuly) will be looking into this issue and they will not be pleased if we make random changes now. Having a stable v0.6 release also saves them from accidentally using v0.5 which is a year old by now. If anything comes up on the VM side, we can backport changes to i.e. v0.6.1
Should I release a stable v0.6 in the meantime?
It would be nice to fix keymappings in Haiku window backend before release. Some keys on Japanese keyboard are currently not mapped and ignored.
Reference code: https://github.com/X547/wayland-server/blob/1a8d982304077fb95f4de395b911466efbf343c3/HaikuSeat.cpp#L29
It would be nice to fix keymappings in Haiku window backend before release. Some keys on Japanese keyboard are currently not mapped and ignored.
Are there any other backends that are affected, or any other keys that were possibly left forgotten? I have a German keyboard without a numpad on my laptop, and a few usual UK/US-like keyboards so I don't even know what other keys exist honestly. So maybe I should just look at hid spec again
So maybe I should just look at hid spec again
HID to Haiku key code table: https://github.com/haiku/haiku/blob/a40cec84d61652d527ed800dc7653fc8706ad17b/src/add-ons/kernel/drivers/input/hid_shared/KeyboardProtocolHandler.cpp#L586
Added international keys & many more other keycodes into hid_api
in 48920ee. Window implementations will follow soon.
Please test a4baaee (I don't even have a keyboard with such keys). I hope it works.
It seems that Windows maps both Japanese keyboard ro (ろ) and German kerboard <>|
(Right next to the lshift, but honestly I don't use it a lot) to the same keycode VK_OEM_102
.
However, they are different keycodes in HID spec & RVVM API (HID_KEY_RO
and HID_KEY_102ND
respectively). I dunno what to do with this because WinAPI has no easy way to distinguish them from userspace.
However, it seems I got how to implement HID_KEY_KATAKANAHIRAGANA
, HID_KEY_HENKAN
, HID_KEY_MUHENKAN
. Perhaps I can drop the German <>|
variant on Windows and unconditionally send HID_KEY_RO
to the guest.
UPD: 4754f08
I dunno what to do with this because WinAPI has no easy way to distinguish them from userspace.
Windows also send PS/2 scancodes for keyboard messages, even if keyboard is USB.
Windows also send PS/2 scancodes for keyboard messages, even if keyboard is USB.
Oh well, thanks. I thought this was some kind of weird driver-specific mechanism but it seems this even works in Wine. I guess I will implement this.
Implemented full HID international keys support on X11 in 86ecb24, and on SDL2 in 82a8b39. SDL2 doesn't seem to have 102ND keycode tho (that German <>|
one), or perhaps I missed something.
SDL1 lacks any kind of international key support.
There are also multimedia keys (Vol+-, brightness, screen locking) that I don't really want being sent to the guest (When user presses Vol+ it would be handled by both host & guest systems which is weird). Idk what to do with this for now, but at least we have the definitions for them in hid_api
for now.
Nam Cao pinged me on the mailing list about the possible solution about the I2C-HID hang. I'll be testing the kernel patch right now.
Converted issue; See https://github.com/LekKit/RVVM/issues/126#issuecomment-1981278618
Older issue
### Steps to reproduce - Have a very slow guest (`rvvm_set_opt(machine, RVVM_OPT_MAX_CPU_CENT, 1)` or immediately call `hid_keyboard_release()`) - Some key presses will be lost when typing - Same possibly applies to hid_mouse ### Investigation - Press/release API calls just mark/unmark a bit in pressed keys bitmask, then mark that input is available on HID device - Guest reads input report, which enumerates pressed keys - if release() happens before this, no pressed keys are reported ### Suggested fix My initial idea is to have two key bitmasks: `keys_report` and `keys_pressed`. - On `hid_keyboard_press()`, a bit is set in both bitmasks; - On `hid_keyboard_release()`, a bit is cleared only in `keys_pressed` bitmask. - When guest is reading input report, a key should be reported as pressed if it's bit is set in **either** bitmasks. - Then, if the bit is actually cleared in `keys_pressed`, then that means we now should replay a key release event, so `input_avail()` should be called. Finally, the bit in `keys_report` is cleared. The problem is, `input_avail()` locking causes recursive deadlock inside `i2c-hid` when calling from `read_report` handler. This needs further investigation - I see there is some queueing mechanism available inside `i2c-hid`, so maybe we can actually queue input reports instead, but haven't read into that code much yet. @X547 perhaps you can point me in the right direction? I am wondering if HID reports should be just queued as packets and not call back into the device `read_report` handler, this would also allow copy-paste into `hid_keyboard` beyond just fixing this issue. In the mean time I'll be polishing network code for a new stable release, and will be back to this issue soon