focus_visible_window breaks if interrupted by mouse click

Describe the bug kitty 0.34.1 on macOS Sonoma 14.4.1

When using focus_visible_window , after the numbers are shown over the windows, if I use the mouse to click on a window. kitty will "hang", I won't be able to type in any of the windows.

To Reproduce Steps to reproduce the behavior:

start kitty fresh /
kitten @ goto-layout grid
kitten @ new-window
kitten @ new-window # we need at least 3 windows
Run cat on one window just to have some program running
F7 (I have `map f7 focus_visible_window)
The window number overlay will be shown
Use the mouse to click on another window
now type, I will just hear a "beep" on every keypress, but the keypress does not seem to reach the shell
I can mouse click on other windows and the cursor blinks in the window selected but still keypresses are not sent
Close kitty, seems to close ok. (I even get the "Are you sure you want to close this OS window? It is running the xxx program", and I'm able to click on Yes, but not press Y)

Screenshots If applicable, add screenshots to help explain your problem.

Environment details

kitty 0.34.1 (7f61f1f9f3) created by Kovid Goyal
Darwin Ruben-Laguna 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:11:05 PDT 2024; root:xnu-10063.101.17~1/RELEASE_X86_64 x86_64
ProductName:        macOS ProductVersion:       14.4.1 BuildVersion:        23E224
Frozen: True
Paths:
  kitty: /Applications/kitty.app/Contents/MacOS/kitty
  base dir: /Applications/kitty.app/Contents/Resources/kitty
  extensions dir: /Applications/kitty.app/Contents/Resources/Python/lib/kitty-extensions
  system shell: /bin/zsh
Loaded config files:
  /Users/rubelagu/.config/kitty/kitty.conf

Config options different from defaults:
allow_remote_control               yes
bold_font                          JetBrains Mono ExtraBold
bold_italic_font                   JetBrains Mono ExtraBold Italic
copy_on_select                     clipboard
disable_ligatures                  2
enabled_layouts                    ['tall:bias=50;full_size=1;mirrored=false', 'horizontal', 'stack', 'grid']
font_family                        JetBrains Mono
font_features:
{'FiraCode-Bold': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Light': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Medium': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Regular': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-Retina': ('+cv02', '+ss03', '+ss07'),
 'FiraCode-SemiBold': ('+cv02', '+ss03', '+ss07'),
 'JetBrainsMono-BoldItalic': ('+zero',),
 'JetBrainsMono-ExtraBold': ('+zero',),
 'JetBrainsMono-ExtraBoldItalic': ('+zero',),
 'JetBrainsMono-Italic': ('+zero',),
 'JetBrainsMono-Regular': ('+zero',)}
font_size                          13.0
italic_font                        JetBrains Mono Italic
listen_on                          unix:/tmp/mykitty
macos_option_as_alt                2
macos_quit_when_last_window_closed True
shell                              /bin/zsh --login --interactive
symbol_map:
    U+23fb - U+23fe → Symbols Nerd Font Mono
    U+2665 - U+2665 → Symbols Nerd Font Mono
    U+26a1 - U+26a1 → Symbols Nerd Font Mono
    U+2b58 - U+2b58 → Symbols Nerd Font Mono
    U+e000 - U+e00a → Symbols Nerd Font Mono
    U+e0a0 - U+e0a3 → Symbols Nerd Font Mono
    U+e0b0 - U+e0d4 → Symbols Nerd Font Mono
    U+e200 - U+e2a9 → Symbols Nerd Font Mono
    U+e300 - U+e3e3 → Symbols Nerd Font Mono
    U+e5fa - U+e6aa → Symbols Nerd Font Mono
    U+e700 - U+e7c5 → Symbols Nerd Font Mono
    U+ea60 - U+ebeb → Symbols Nerd Font Mono
    U+f000 - U+f2e0 → Symbols Nerd Font Mono
    U+f300 - U+f32f → Symbols Nerd Font Mono
    U+f400 - U+f4a9 → Symbols Nerd Font Mono
    U+f500 - U+f8ff → Symbols Nerd Font Mono
    U+f0001 - U+f1af0 → Symbols Nerd Font Mono
visual_bell_duration               0.2
Added shortcuts:
    ctrl+. →  layout_action bias 50 62 70
    ctrl+/ →  layout_action mirror toggle
    f1 →  toggle_layout stack
    f7 →  focus_visible_window
Removed shortcuts:
    cmd+, →  edit_config_file
    cmd+enter →  new_window
    cmd+h →  hide_macos_app
    cmd+k →  clear_terminal to_cursor active
    cmd+m →  minimize_macos_window
    cmd+n →  new_os_window
    cmd+q →  quit
    cmd+t →  new_tab
    cmd+w →  close_tab
    ctrl+cmd+, →  load_config_file
    ctrl+cmd+f →  toggle_fullscreen
    opt+cmd+h →  hide_macos_other_apps
    opt+cmd+r →  clear_terminal reset active
    opt+cmd+s →  toggle_macos_secure_keyboard_entry
    shift+cmd+/ →  open_url https://sw.kovidgoyal.net/kitty/
    shift+cmd+[ →  previous_tab
    shift+cmd+] →  next_tab
    shift+cmd+d →  close_window
    shift+cmd+w →  close_os_window

Important environment variables seen by the kitty process:
    PATH                                /Applications/kitty.app/Contents/MacOS:/usr/bin:/bin:/usr/sbin:/sbin
    LANG                                en_US.UTF-8
    SHELL                               /bin/sh
    DISPLAY                             /private/tmp/com.apple.launchd.hrQvMCN20U/org.xquartz:0
    USER                                ruben.laguna@xxxx.com

Additional context

I did reproduced using kitty --config NONE ( using the default control+shift+f7 to switch to the grid layout and performing the same actions described above

kovidgoyal / kitty

focus_visible_window breaks if interrupted by mouse click #7390