Mouse spamming wheel events causing input queue to never catch up (was: Can I make imgui consume all the input events when ImGuiIO::ConfigInputTrickleEventQueue is on?)

[esert]

Version/Branch of Dear ImGui:

Version: 1.88 Branch: master

My Question:

Context: I have a mouse that can generate ~100 WM_MOUSEWHEEL events every second for 15secs when flicked hard. I use ImGuiIO::AddMousePosEvent(); ImGuiIO::AddMouseWheelEvent(); to pass all these events and imgui becomes very unresponsive until the events calm down. I tracked the problem down to ImGuiIO::ConfigInputTrickleEventQueue being true by default. If I turn it off there is no unresponsiveness. Similarly turning off vsync (so ImGui::NewFrame()/ImGui::Render() loop runs as fast as possible) also helps.

My question: Is there a way to make imgui consume all the input in the same frame (and update all the control datastructures multiple times until all the events are consumed) when ImGuiIO::ConfigInputTrickleEventQueue is on? I like the idea of ImGuiIO::ConfigInputTrickleEventQueue flag so I don't want to turn it off, but I also like to flick my mouse wheel.

Thanks

[ocornut]

Hello,

My question: Is there a way to make imgui consume all the input in the same frame (and update all the control datastructures multiple times until all the events are consumed) when ImGuiIO::ConfigInputTrickleEventQueue is on? I like the idea of ImGuiIO::ConfigInputTrickleEventQueue flag so I don't want to turn it off, but I also like to flick my mouse wheel.

Your question is worded like "can I disable Input Trickling without disabling Input Trickling", those are contradictory. If you want to consume all events in the same frame you simply need to set ConfigInputTrickleEventQueue to false.

However I'm particularly interested in this:

Context: I have a mouse that can generate ~100 WM_MOUSEWHEEL events every second for 15secs when flicked hard

Do you mean that a single fast action (say, over <0.5 second) will keep generating mouse wheel events over time 15 seconds due to smooth scrolling logic? Can you clarify what your mouse is and what drivers are involved? In that case you'll have interleaved MousePos and MouseWheel events during a long period and that will seriously mess up with our trickling logic. This is why it is important we understand how common and likely that situation is.

Similarly turning off vsync (so ImGui::NewFrame()/ImGui::Render() loop runs as fast as possible) also helps.

That statement appears to contract the statement "events every second for 15secs when flicked hard" so I'll need you to check on version and do some debugging. If faster framerate improves it you have a problem of accumulating events which takes ~15 seconds to trickle at regular framerate, which is very different from having events submitted continuously for 15 seconds.

Successive mouse wheel events SHOULD be merged, it was fixed fairly recently in 1.88 (May 2022, commit e346059). Can you confirm that you are using actual tagged 1.88 or an earlier 1.88 WIP? The fix happened precisely on version 18722. But mouse wheel events interleaved with mouse pos events for a long period of time is not a situation we expected to encounter.

If you have an earlier version you should update. If you have a later version we should be looking at the issue together.

Suggestion for debugging this: In the UpdateInputEvents() function near the end there is a commented block of three-lines with debug statement. Can you uncomment as well as the DebugPrintInputEvent() function it is calling (You will need to add a ImGuiContext& g = *GImGui; at the top of commented out DebugPrintInputEvent()) and shows how your events are queued?

Otherwise you may add IMGUI_DEBUG_LOG() calls in AddMousePosEvent() and AddMouseWheelEvent() and open Demo->Tools->Debug Log

[esert]

Your question is worded like "can I disable Input Trickling without disabling Input Trickling", those are contradictory. If you want to consume all events in the same frame you simply need to set ConfigInputTrickleEventQueue to false.

I don't think it is contradictory, but I might have not worded it sufficiently clear. I believe there is value in not collapsing multiple events (e.g. not collapsing [move cursor x1,y1, click left mouse, move cursor x2,y2] into [move cursor x2,y2, click left mouse]). So what I had in mind was something like the following:

imgui_io.ConfigInputTrickleEventQueue = true;
for (Event : events)
  imgui_io.AddXXXEvent();
ImGui::NewFrame();
while (imgui_context.InputEventsQueue.Size)
  // update imgui internal control structures and drain input events queue without needing to call ImGui::Render() somehow.

Events are not collapsed but they also don't live longer than the frame they are generated on.

Do you mean that a single fast action (say, over <0.5 second) will keep generating mouse wheel events over time 15 seconds due to smooth scrolling logic? Can you clarify what your mouse is and what drivers are involved?

I use Logitech M500 which has an "Hyper-fast scroll wheel". It is possible to make the wheel rotate physically for 15 seconds, so all the WM_MOUSEWHEEL messages originate from the device due to wheel rotation. See https://www.youtube.com/watch?v=aBW2LlciCng&t=9s for a mild demonstration of the wheel (you can imagine if you flick it hard the wheel rotates for longer). I use whatever driver default Windows 10 uses when you plug the mouse to the pc.

Successive mouse wheel events SHOULD be merged, it was fixed fairly recently in 1.88 (May 2022, commit https://github.com/ocornut/imgui/commit/e346059eef140c5a8611581f3e6c8b8816d6998e). Can you confirm that you are using actual tagged 1.88 or an earlier 1.88 WIP? The fix happened precisely on version 18722.

I use 18800. Please see https://vimeo.com/736919216 for a demonstration of the unresponsiveness. In the video, I start with all the sections in the demo window expanded, scroll bar at the very top position. I then flick mouse wheel down hard, immediately click on the window and move it. As you can see it follows the historical movement of my cursor rather than where the cursor actually is. I enabled the debug logging you suggested, I also added the new frame message for which I log with IMGUI_DEBUG_LOG() before each Imgui::NewFrame() call. This logging however made frame times go up to ~100ms. If the logging isn't activated I get a flat 16.67ms and ImGui is similarly unresponsive.

[esert]

fyi, version I use is 18800 (visible in the video), not 18000 as I mistakenly written in the first version of the post above.

[ocornut]

So what I had in mind was something like the following:

Your GUI code needs to run for events to be processed so you would need to run most of your frame. I suppose it would be equivalent to something like:

if (imgui_context.InputEventsQueue.Size > 0)
{
   // skip rendering, skip swapping, skip imgui backend update, get to next frame immediately
}

Which might work but honestly I can sense this may be causing obscure problem with backends.. Not mentioning it would be abnormally heavy, even more bizarrely slow considering the overall situation being "wheeling with a certain type of mouse reduce framerate by 30% and surfaces obscure app bugs". I think it is reasonable to rule out that solution completely.

--

You are opening quite a complicated problem here. We can't design a fix or workaround just for you, it needs to work for all apps by default since people shipping apps would expect things to work with that sorts of mouse (designed by devils I must say).

I guess for a start we need to consider: what do you generally expect to happen when doing such a "long" wheeling while simultaneously moving the mouse?

What happens in your typical game or application? Are things mostly relying on the fact that most (but maybe not all) modern software are "locking" on the wheeling target when starting to wheel?

I suspect we may need to involve two approaches:

• Input queue need to know how to react when its over-filling. It may be senseful to merge events and catch up in some cases. Maybe simply we stop trickling interleaved mouse pos/wheel events if that "fast rate of events" is a thing only specific to mouse wheel. Or we stop trickling only in some situation (based on event counts or frame/time of each event...).
• Input queue could be aware that something has been "locked" for mouse wheeling meaning mouse position won't interfere with use of wheel values. In that situation it becomes safe to merge interleaved pos/wheel events. When wheeling to scroll a window we set a latch (g.WheelingWindow in StartLockWheelingWindow()) we would need to generalize that to having a "wheel owner" and that wheel owner can state that it doesn't care anymore about mouse position and then merging can be done.
• Also means if no one is using the mouse wheel we can safely merge but right now we have no mechanism to tell that mouse wheel is being used or even polled. This is very tricky.

I think we need to first investigate the first solution.

Could you run the same code, copy the full log to Clipboard then into a file and attach to a file here?

Thank you...

[ocornut]

PS: In the very last commit under Tools>DebugLog I have added a [X] IO checkbox allowing to visualize input events and how they are trickled. This is equivalent to what I asked you to enable, but it is now always on.

[esert]

Hi, I'm very sorry for the late reply.

Which might work but honestly I can sense this may be causing obscure problem with backends.. Not mentioning it would be abnormally heavy, even more bizarrely slow considering the overall situation being "wheeling with a certain type of mouse reduce framerate by 30% and surfaces obscure app bugs". I think it is reasonable to rule out that solution completely.

You are opening quite a complicated problem here. We can't design a fix or workaround just for you, it needs to work for all apps by default since people shipping apps would expect things to work with that sorts of mouse (designed by devils I must say).

My intention for creating this issue was to look for something I can personally do myself, not change something for everyone, unless the change is trivial. I understand what I described could be heavy for general use.

I guess for a start we need to consider: what do you generally expect to happen when doing such a "long" wheeling while simultaneously moving the mouse?

I expect UI to stay interactive just like the file explorer or chrome browser do when "long" wheeling happens (e.g. when I click on a button during wheeling, button still works just like it normally does).

What happens in your typical game or application? Are things mostly relying on the fact that most (but maybe not all) modern software are "locking" on the wheeling target when starting to wheel?

I use DearImGui for simple debugging ui, so I don't want imgui to lose io events (by keeping io.ConfigInputTrickleEventQueue on) and I want it to perform reasonably when flooded with wheel events (admittedly, flooding due to wheel events is something I can prevent through my behavior).

I suspect we may need to involve two approaches:

• Input queue need to know how to react when its over-filling. It may be senseful to merge events and catch up in some cases. Maybe simply we stop trickling interleaved mouse pos/wheel events if that "fast rate of events" is a thing only specific to mouse wheel. Or we stop trickling only in some situation (based on event counts or frame/time of each event...).
• Input queue could be aware that something has been "locked" for mouse wheeling meaning mouse position won't interfere with use of wheel values. In that situation it becomes safe to merge interleaved pos/wheel events. When wheeling to scroll a window we set a latch (g.WheelingWindow in StartLockWheelingWindow()) we would need to generalize that to having a "wheel owner" and that wheel owner can state that it doesn't care anymore about mouse position and then merging can be done.
• Also means if no one is using the mouse wheel we can safely merge but right now we have no mechanism to tell that mouse wheel is being used or even polled. This is very tricky.

Approach 1 sounds reasonable to me. I imagine categorizing events into high (mouse/keyboard button press) and low priority (mouse pos/wheel) events and making sure high priority events are handled by merging/skipping over low priority events during event queue flooding could work in this case.

Could you run the same code, copy the full log to Clipboard then into a file and attach to a file here?

Please find the logs at https://gist.github.com/esert/5ec32d1dbf3be812795f15aa2216f58c

Thanks

[ocornut]

Thanks for your answer.

what do you generally expect to happen when doing such a "long" wheeling while simultaneously moving the mouse? What happens in your typical game or application? Are things mostly relying on the fact that most (but maybe not all) modern software are "locking" on the wheeling target when starting to wheel?

Currently traveling so will look in detais later, but you haven’t answered the core of my question quoted above. What happens with the wheeling inputs, what do they perform in term of acting with while simultaneously moving around ?

What would happens on an application where eg wheeling can be used to edit a spinner widget that is being hovered ?

Feels like this long wheeling may be a constraint on the design space.

categorizing events into high (mouse/keyboard button press) and low priority (mouse pos/wheel) events

This is making assumptions about what each application does with the inputs, I don’t know how far those assumptions can hold.

My intention for creating this issue was to look for something I can personally do myself, not change something for everyone, unless the change is trivial. I understand what I described could be heavy for general use.

If anybody can legally buy that mouse you have, my intuition is not to provide you with an opt-in workaround but make sure it is works for everyone.

[esert]

Currently traveling so will look in detais later, but you haven’t answered the core of my question quoted above. What happens with the wheeling inputs, what do they perform in term of acting with while simultaneously moving around ?

Moving around the mouse/window while wheeling was an example to illustrate the problem of event flooding. A more realistic example would be scrolling to the end of a long list of input fields from top of the window and hit submit button that is at the very bottom. That doesn't require moving the mouse that much while scrolling. What happens today (if I use the fast mouse wheel) is that button is hit a couple of seconds after I click the mouse button. Everything eventually works but experience isn't great when mouse wheel events are firing that often. Turning io.ConfigInputTrickleEventQueue off improves the experience greatly but as I mentioned in the very first post, I'd like to keep it on in order to not stomp on events I care about (e.g. repeated presses during low fps situations).

What would happens on an application where eg wheeling can be used to edit a spinner widget that is being hovered ?

What is a spinner widget? I searched for spin in imgui_demo.cpp but I don't have any hits.

This is making assumptions about what each application does with the inputs, I don’t know how far those assumptions can hold.

That's fair. If you think rest of the idea is valuable, priority can be a user input instead of a library default.

If anybody can legally buy that mouse you have, my intuition is not to provide you with an opt-in workaround but make sure it is works for everyone.

It isn't necessarily an obscure mouse https://www.amazon.com/Logitech-M500-Corded-Mouse-Hyper-Fast/dp/B002B3YCQM

[tmsrise]

If anybody can legally buy that mouse you have, my intuition is not to provide you with an opt-in workaround but make sure it is works for everyone.

It isn't necessarily an obscure mouse https://www.amazon.com/Logitech-M500-Corded-Mouse-Hyper-Fast/dp/B002B3YCQM

Yep, hyperscrolling mice are actually very common. The G502 is the world's best selling gaming mouse.

[tmsrise]

I'm ignorant of ImGui's implementation and design philosophy, so take my perspective for the little it's worth:

1) If the mouse is over a window/item that's non-scrollable, there's no point in processing scroll events.

2) If a scrollable item has reached its limit, there's no point in processing scroll events in that direction.

Perhaps this means the event queues need to be separated out?

[ocornut]

@esert

what do you generally expect to happen when doing such a "long" wheeling while simultaneously moving the mouse? I expect UI to stay interactive just like the file explorer or chrome browser do when "long" wheeling happens (e.g. when I click > on a button during wheeling, button still works just like it normally does).

That assume that the effect of scrolling is locked/targeted to the item where mouse was before starting to wheel, as when moving you may not be over the initial item anymore. See example below:

What would happens on an application where eg wheeling can be used to edit a spinner widget that is being hovered ? What is a spinner widget? I searched for spin in imgui_demo.cpp but I don't have any hits.

There's none in the current widget set, but I meant a widget that reacts to mouse wheel to edit a value when the widget is hovered. As mentioned in https://github.com/ocornut/imgui/issues/5545#issuecomment-1206834737 mouse wheeling to scroll windows sets a sort of lock already, because by definition scrolling changes what's being hovered. I'll need to find a way to generalize this locking to other systems such as polling mouse wheel in custom widgets - not obvious since "polling" suggest an anonymous reading source.

@tmsrise

If the mouse is over a window/item that's non-scrollable, there's no point in processing scroll events.

This is assuming that mouse-wheel is only used for scrolling, but it could be used by custom-code/widgets in many ways (e.g. zooming a node graph). And how things currently work they are simply polled.

If a scrollable item has reached its limit, there's no point in processing scroll events in that direction.

Perhaps yes. But neither would fix our issue well enough either way.

I'm purchasing a Logitech M500 now to be able get to the bottom of this and test/fix it....

[sergeiromanov]

I'm having similar issue on android. I'm sending mouse move events when finger moves on the screen. The UI keeps processing the events long after the finger is lifted. My workaround is to disable ConfigInputTrickleEventQueue and manually delay mouse buttons (and keys) events to the next frame.

[ocornut]

. I'm sending mouse move events when finger moves on the screen. The UI keeps processing the events long after the finger is lifted.

Could you post a few pages of contents from Debug Log->IO into a file here?

[sergeiromanov]

. I'm sending mouse move events when finger moves on the screen. The UI keeps processing the events long after the finger is lifted.

Could you post a few pages of contents from Debug Log->IO into a file here?

I realized that I also send wheel events to scroll with a finger. Looks like a known issue. It does feel like an awkward workaround but I've got the app running on desktop and mobiles (android and ios), and need to support multitouch. Ideally I should use proper gesture detection on each of mobile platforms but they are so different that it's easier to just handle touch up/down/move.

[ocornut]

Your message reada ambiguous to me. Pseudo-code of what you are doing (and the log) would be preferable.

[sergeiromanov]

Here is a simplified version. Android Java

class TouchListener implements View.OnTouchListener {
    @Override
    public boolean onTouch(View v, MotionEvent event) {
        if (event.getActionMasked() ==MotionEvent.ACTION_MOVE ) {
            JniInterface.onTouchMove(event.getX(), event.getY());
        }
    }
}

C++

void onTouchMove(float x, float y) {
    float prevY = mLastCursorPos.y;
    mLastCursorPos = Vec2(x, y);

    imguiContext->IO.AddMousePosEvent(x, y);
    imguiContext->IO.AddMouseWheelEvent(0, (y - prevY)*0.02f);
}

[ocornut]

Seems odd to try doing start, no interface would behave correctly if every Y move leads to move position and wheel changes. Wheel is generally mapped to two-fingers dragging and I am sure Android provides this data.

[sergeiromanov]

I don't expect the UI to handle this totally gracefully but it works reasonably well for me. I have a long list and want to just scroll through - pretty standard gesture on mobiles. The users don't expect to use two fingers just to scroll down. The proper solution is to use platform specific gestures and convert them to imgui events. The problem is windows, web browser, ios, android are painfully different. This is just a quick work-around for now.

[bbbradsmith]

I ran into this issue while looking at a missing-clicks problem in "furnace", and I think the suggestion I made there might be worthwhile for ImGui in general.

The history of the issue seems to be:

1. ImGui using the default setting ConfigInputTrickleEventQueue = true.
2. Have an issue with high-rate mice spamming the queue and making the program unresponsive while it catches up.
3. Set ConfigInputTrickleEventQueue = false to solve spam issue.
4. Some mice that report down/up too fast now lose clicks.
5. (My suggestion) Add an alternative trickle only-if a mouse click would be lost.

You can see my eventual proposed suggestion here, as 3 small changes to imgui.cpp: Furnace pull request 1203: imgui.cpp

The rationale was that it's fine to turn off trickle for most events, but the click is much more critical to this application, and MouseButton events should presumably always be "sparse", even with a high-polling device, so their trickling-interruption of the queue should be insignificant for the user.

Because different applications have different needs, I would suggest it being configurable rather than forced. The more general version might be to have a configuration option that can filter the trickle to just specific event types.

As a default, I would guess that most applications do have a need to trickle MouseButton, Key and Text events, but probably don't need it for MousePos or MouseWheel where "last position wins" (no trickle) is probably more desirable than "delay to process every position along the way" (trickle). This might remove the need for most applications to have to investigate changing the default, as MouseButton/Text trickling should presumably be sparse and not be a liability for high-poll spamming. (Note also that the default Text event trickle would probably not want to be triggered by MousePos/MouseWheel events as it currently is, instead only by the other sparse events.)