Memory Leak with GetMessage loop

[ryjacky]

I'm writing a mouse hook with the following code (at the end) and it causes a memory leak at the message loop:

    while (GetMessage(msg, NULL, 0, 0) != 0) {
      TranslateMessage(msg);
      DispatchMessage(msg);
    }

To reproduce, just copy the following code into a flutter project and call MouseHook.isolated(), this will create a new isolate that runs the message loop. Keep moving the mouse and the memory usage will rise as long as the mouse is moving.

The following is what I've tried:

• Running the code in the main isolate without creating a new one but the issue doesn't resolve.
• Removing the while GetMessage loop and the issue is resolved.
• Writing the same code in C++, compile it and run it with Process.run(), the issue doesn't exist.
• Looking into the Dev Tools > Memory, the only interesting thing I found is the rss memory keeps rising while others remain low.
• The memory leak does not happen in a dart-only project (non-flutter project)

Could anyone help to check if there is a problem with my code or is it a bug in the win32 package?

class MouseHook {
  final SendPort _sendPort;
  final ReceivePort _receivePort = ReceivePort();
  int _hookHandle = NULL;

  static Future<ReceivePort> isolated() async {
    final receivePort = ReceivePort();

    Isolate.spawn((sendPort) => MouseHook._(sendPort), receivePort.sendPort);

    return receivePort;
  }

  MouseHook._(this._sendPort) {
    _sendPort.send(_receivePort.sendPort);
    _receivePort.listen((message) {
    });

    final callback = NativeCallable<CallWndProc>.isolateLocal(mouseHookProc,
        exceptionalReturn: 0);

    int hInst = GetModuleHandle(nullptr);
    _hookHandle =
        SetWindowsHookEx(WH_MOUSE_LL, callback.nativeFunction, hInst, NULL);
    final msg = calloc<MSG>();

    while (GetMessage(msg, NULL, 0, 0) != 0) {
      TranslateMessage(msg);
      DispatchMessage(msg);
    }

    free(msg);
  }

  int mouseHookProc(int nCode, int wParam, int lParam) {
    final pMouseStruct = Pointer<MOUSEHOOKSTRUCT>.fromAddress(lParam);
    if (nCode >= 0 && wParam == WM_MOUSEMOVE) {
      _sendPort.send(MouseMoveEvent(pMouseStruct.ref.pt.x, pMouseStruct.ref.pt.y));
    }

    return CallNextHookEx(_hookHandle, nCode, wParam, lParam);
  }
}

The same C++ code

#include <iostream>
#include <Windows.h>

HHOOK g_hookHandle = nullptr;

LRESULT CALLBACK MouseHookCallback(int nCode, WPARAM wParam, LPARAM lParam)
{
    if (nCode == HC_ACTION)
    {
        MSLLHOOKSTRUCT* pMouseStruct = (MSLLHOOKSTRUCT*)lParam;

        if (wParam == WM_MOUSEMOVE)
        {
            std::cout << pMouseStruct->pt.x << "\t" << pMouseStruct->pt.y << std::endl;
        }
    }

    return CallNextHookEx(g_hookHandle, nCode, wParam, lParam);
}

int main()
{
    HINSTANCE hInstance = GetModuleHandle(nullptr);

    g_hookHandle = SetWindowsHookExW(WH_MOUSE_LL, MouseHookCallback, hInstance, 0);
    if (g_hookHandle == nullptr)
    {
        std::cerr << "Failed to set mouse hook." << std::endl;
        return 1;
    }

    MSG message;
    while (GetMessage(&message, nullptr, 0, 0))
    {
        TranslateMessage(&message);
        DispatchMessage(&message);
    }

    UnhookWindowsHookEx(g_hookHandle);

    return 0;
}

[halildurmus]

Thank you for providing detailed information and code snippets.

I tested the provided code in both a Dart-only project and a Flutter app, and I can confirm your observation about the memory usage going up when the mouse is moving. Interestingly, I also observed this behavior in a Dart-only project, both when running it with dart run --observe example.dart and after compiling to an executable with dart compile exe example.dart.

After poking around with DevTools, I saw that a significant number of MouseHookStruct and POINT struct instances have been created. This is not surprising, since the contents of these structs are accessed from the following line:

  _sendPort.send(MouseMoveEvent(pMouseStruct.ref.pt.x, pMouseStruct.ref.pt.y));

I also noticed that when the number of struct instances reached significant levels (thousands, even tens of thousands), the Dart GC stepped in like a hero, and memory usage dropped significantly.

Based on my analysis, there is no clear evidence of a memory leak here. If you have any doubts, I recommend you consider filing an issue to the Dart SDK repository for further investigation.