Move/resize gestures ignored by Microsoft Excel (and other non-standard window apps)

adamm commented 5 years ago

Love the app, and works exactly as expected everywhere, except on Microsoft Excel on Office 365

Mac OS: 10.14.4 Hummingbird: 2.0.0(3) bc3358d Microsoft Excel: v16.24

What's interesting is that it works fine on Microsoft Word, Microsoft Teams, Powerpoint, etc. Just Excel doesn't honour the move/resize gestures.

finestructure commented 4 years ago

Sorry for the very late reply - for some reason my notifications were turned off :/

I'm afraid there's probably not much I can do about this. In rare cases windows seem to be elusive. I don't have Excel to test this further but if you come across any other windows for apps that I could try this with, please let me know.

Also, have you tried changing the modifier keys? I'm wondering if perhaps Excel is capturing some of them in a weird way preventing Hummingbird from activating.

adamm commented 4 years ago

No worries :)

I can confirm other Move modifier keys have the same problem. However, I see all the Resize modifiers do work with Excel. Odd.

If I find another program with the same behavior, i'll update this issue log.

finestructure commented 4 years ago

Mmm, that's bizarre. And I'm sure you've swapped the modifiers between move/resize, i.e. the set that's working for Resize isn't working for Move.

In that case the only thing I can think of is that there's something particular about the window that makes it unresizeable. Although move and resize are using the same APIs - I would expect them both to behave the same...

I have come across some other app in the past that wouldn't respond to moves or resizes but I can't recall right now which one it was. Certainly not one I use on a regular basis.

I'll see if I can get my hands on Excel to inspect what the window handling code does.

finestructure commented 4 years ago

Ok, I've had a look now and it's quite bizarre. It does work, sometimes. The key seems to be to initiate the first Hummingbird move/resize near the toolbar menu. After that, the window can be move/resized as normal:

https://www.icloud.com/iclouddrive/0qHrtLWeH5SRyHnGt4FxDhqmw#Screen_Recording_2019-10-25_at_13.57

adamm commented 4 years ago

Interesting. I tried to replicate your video (I even set my Move hotkeys to Ctrl+Fn) and I can confirm the toolbar works as expected, but when i go back to the main spreadsheet it still doesn't move. One can assume that Microsoft is probably doing something funky with the rendering layer that's causing interference.

I found another program that has the same issue. System Preferences On all preferences windows, the hotkey on the toolbar moves the window just fine, but on some of them, using the hotkey on the body of the window does not.

General: Fails Desktop & Screensaver: Fails Dock: Works Mission Control: Works Language & Region: Works Security & Privacy: Fails Spotlight: Works Notifications: Fails ... Trackpad: Works Printers & Scanners: Fails

Clearly there is some sort of UI object used on these windows that is conflicting with Hummingbird.

Hope this helps!

finestructure commented 4 years ago

Yes, I've actually noticed that myself before and I suspect it's because of security context.

Hummingbird is using the Accessibility APIs to read and modify window geometry. There are other things the Accessibility APIs allow you to do and I believe any particularly critical controls completely opt out of the mechanism even if you grant the accessibility permissions.

Regarding Excel: my video test didn't work 100% of the time. There were definitely cases where the menu bar "trick" didn't allow subsequent move/resize of the main window. Really weird.

I'm hoping to take a closer look this week!

finestructure commented 4 years ago

I don't see any way to fix this, I'm afraid. Excel windows seem to be playing by their own rules and I can't figure out how to get a handle on them to trigger move/resize.

vaclavhanzl commented 4 years ago

Aquamacs is another app which does not move/resize. When I try to move/resize Aquamacs, some other window is move/resized instead, so I guess that it is somehow undiscovered in the list of windows HB considers. Setting different modifiers does not matter. Also, trying with pointer at titlebar also does not work. I'd be happy to help with this somehow, otherwise HB is a lifesaver for me. I had to switch from Linux to Mac and I found window management really painful. HB mostly saved me but Aquamacs is important in my work and not being able to move it spoils the joy. I am quite experienced programmer on Linux, new to Mac but I already did first hacks to fix things driving me crazy on Mac (End key going to end of document instead of end of line, no normal way to make truly global shortcut to run a terminal etc.) and I considered writing something like HB but fortunately found it exists already (thanks!!!). So I am willing to invest some effort into fixing HB on Aquamacs.

cotfas commented 4 years ago

(End key going to end of document instead of end of line, no normal way to make truly global shortcut to run a terminal etc.)

For keyboard remaping you can try Karabiner (you can also assing keys to execute scripts)

vaclavhanzl commented 4 years ago

Thanks @cotfas , I already solved all my key-related itches with 100 lines of C code, this way I have full control :) Fixing HB for Aquamacs is my only pain left uncured...

finestructure commented 4 years ago

Hey @vaclavhanzl , thanks for reporting the issue with Aquamacs! To be honest, I'm not that surprised there are issues. There seem to be a few apps that don't use standard windows and therefore bypass the detection method.

I'm currently at a loss as to how to fix it. I've just pushed a branch were I added more detailed error logging when detection failed but as I recall (it's been a while since I looked at it) there was no rhyme or reason for why it's failing.

If you'd like to take a stab at it and manage to solve that puzzle that'd be fabulous! :)

vaclavhanzl commented 4 years ago

Hi @finestructure, I found out what is the problem with Aquamacs. In the GUI hierarchy, well behaved apps have 'window' elements under 'application', see e.g. (run in terminal, maybe allowing some security exception if asked) :

osascript -e 'tell application "System Events" to get the entire contents of every UI element whose name is "Terminal"'|tr ',' '\012'|head

which prints

window ... of application process Terminal

and it is similar for other movable apps like "Google Chrome", however for "Aquamacs", I get:

text field ... of application process Aquamacs

so the 'window' type element is missing in the hierarchy. I can move Chrome window with:

osascript -e 'tell application "System Events" to set position of first window of application process "Google Chrome" to {100, 100}'

however I need this to move Aquamacs:

osascript -e 'tell application "System Events" to set position of first text field of application process "Aquamacs" to {100, 100}'

So the problem is that AXUIElement.window(at: event.location) defined here only finds 'window' elements but not 'text field' elements. I am unable to find out how to make it to work for 'application' child nodes of any type but you may know :)

vaclavhanzl commented 4 years ago

I can move Aquamacs, however is does not mean that it can be discovered under pointer. So to answer this, press Cmd+Shift+4 Space and start roaming around the screen. Aquamacs really is properly discovered. (Press Esc to get out of it.) However I have no idea what API is used by the Apple's print screen tool.

finestructure commented 4 years ago

Hi @vaclavhanzl , that's a very interesting find! window(at:) is looking specifically for NSAccessibility.Attribute.window but there might be a similar attribute for text fields.

The method could fall back to that if no window is found or special case it for Aquamacs, Excel (if the same applies), ...

Very interesting also that you can move Aquamacs by telling the text field to move!

I'm not surprised the window screenshot feature can find the window - I'm sure there's another API at play here than the Accessibility access route that HB is limited to.

Thanks for reporting back, I'll definitely take a look. I'm very busy with another project at the moment so I'm afraid it'll be a while though :(

vaclavhanzl commented 4 years ago

Thanks for your attention at busy time @finestructure. Looking at docs, I guess the fall back could be to test for Attribute.role being application and if yes, use the element itself as a window (sorry, just guessing, never seen swift before and what I see using applescript - which I also never used before - only loosely corresponds). I guess so because I found movable targets using

osascript -e 'tell application "System Events" to get the name of every UI element whose role is "AXApplication"'

and it would be nice if other people could use it to find names for any other non-movable apps (like Excel) and then try to analyze and move them using commands I wrote before. Aquamacs itself could certainly be fixed (it likely violates some standard and is opensource so I could file a bug and/or fix it) however I'd not hope for any fix in Excel :) so workaround in HB may be the only way.

finestructure commented 4 years ago

Super useful tips there for debugging! That should come in handy. Thanks again for sharing :)

tigger04 commented 3 years ago

Could it be that all these applications that don't work with hummingbird are XQuartz apps? (X11 mac implementation)

You can launch XQuartz alone and it has some interesting options athat I could see potentially interfering with Hummingbird ... i have fiddled around with these now so that i can't even remember what the defaults were I'm afraid, but still haven not persuaded it to do what Hummingbird tells it (or would tell it, if it could find it?)

finestructure commented 3 years ago

Hi @tigger04 , thanks for posting. I don't think that's the issue here - Aquamacs isn't an XQuartz app, I believe, and Excel certainly isn't. However, I'm almost certain XQuartz apps won't work with HB, either.

As to the particular settings: these look more like they might interfere with particular shortcuts you've set up in XQuartz. The issue at play here, however, is that the windows themselves aren't recognised by HB, which would be a different issue (with the same end result, though 😅 ).

tigger04 commented 3 years ago

A frustrating one! perhaps as Apple continues to lock down the platform further with each release, apps will be forced to conform like they do on iOS. It might be a good day for uniformity, but that also brings with it the fear over what that means for developers and open source software in general.

Anyway thanks for putting together a nice little tool. I normally use BetterTouchTool but it's been quite unstable recently. This one seem to do one thing and does it well.

tigger04 commented 3 years ago

Aquamacs is another app which does not move/resize. When I try to move/resize Aquamacs, some other window is move/resized instead, so I guess that it is somehow undiscovered in the list of windows HB considers. Setting different modifiers does not matter. Also, trying with pointer at titlebar also does not work. I'd be happy to help with this somehow, otherwise HB is a lifesaver for me. I had to switch from Linux to Mac and I found window management really painful. HB mostly saved me but Aquamacs is important in my work and not being able to move it spoils the joy. I am quite experienced programmer on Linux, new to Mac but I already did first hacks to fix things driving me crazy on Mac (End key going to end of document instead of end of line, no normal way to make truly global shortcut to run a terminal etc.) and I considered writing something like HB but fortunately found it exists already (thanks!!!). So I am willing to invest some effort into fixing HB on Aquamacs.

@vaclavhanzl for emacs take a look at https://github.com/railwaycat/homebrew-emacsmacport also the repo at https://bitbucket.org/mituharu/emacs-mac/src/master/README-mac - in sort

emacs main package used to have a carbon UI option but carbon is the old mac UX API/library or 'appkit' that has gone out of fashion/support, so it takes some refactoring to build for cocoa (the new appkit/api based on obj-c/swift)
anyway, this is the same emacs just ported to cocoa, so works just like any other mac app and works well with HB
i'm not so familiar with aquamacs but i use doom emacs which is another variant, and this works perfectly for it and has solved all my problems if not working with aquamacs i can recommend doom is worth taking a look, it's emacs with some sensible defaults, EVIL (vim based) keybindings and a bunch of lazy loaded packages that is pretty snappy to use

re universal keybindings on them ac as already mentioned in this thread karabiner is the gold standard- if you don't mind writing your own json it is extremely powerful, basically remap your keyboard and any combo of keys/apps/variables to do what you want. If you want something quick and dirty try bettertouchtool where you can map what you want through the UI - it's freeware not FOSS, there is a paid option i believe

reagle commented 2 years ago

Drats. I just discovered and purchased Hummingbird and note that it doesn't work with KeePassXC 2.6.6 . I just found this bug and confirm it doesn't work for Excel...

finestructure commented 2 years ago

Sorry to hear it's not working for you, @reagle. If you send me an email with email address you used to purchase I can issue a refund.

reagle commented 2 years ago

@finestructure: Count it as an investment for a possible fix -- though I know you said it's not likely. Perhaps update the README to say it doesn't work with some apps?

@tigger04: I also have BTT, but haven't any idea how to implement this same functionality there. Do you have a pointer?

finestructure commented 2 years ago

Perhaps update the README to say it doesn't work with some apps?

That's a good idea, I'll do that. Thanks!

finestructure commented 2 years ago

Added https://github.com/finestructure/Hummingbird/blob/develop/readme.md#known-limitations

reagle commented 2 years ago

As you can see, KeePassXC closed the issue as an QT/upstream issue. It'd be good to file a bug with QT, but I'm a simple user and know nothing about Hummingbird or QT development.

tigger04 commented 2 years ago

@tigger04: I also have BTT, but haven't any idea how to implement this same functionality there. Do you have a pointer?

I used to have BTT and I think it was - open BTT config window, in the menu there's an item to get to general preferences (cmd+ comma ',' will probably do it). Behind a labyrinth of tabs was buried away a fairly well configurable drag window / resize functionality quite similar to this one.

Note BTT is a paid app. I stopped using it due to its ghastly user interface and the fact that it was becoming more and more unstable as time went on and updates would frequently break the whole thing - I just relied on it too much. When Karabiner bowed out permanently from functioning on Big Sur I migrated the whole thing over to Hammersoon - and stumbled upon a script that does the same thing for moving and sizing

reagle commented 2 years ago

@tigger04, with your help, I found it, thanks! BTT also works with Excel and QBittorrent (but not KeepassXC). I still prefer Hummingbird's "resize from corner" feature.

gdelafond commented 1 year ago

FYI: for KeePassXC 2.7.4 on macOS Ventura, it works but only from the title window bar.

cyaconi commented 1 year ago

Same with Kitty terminal, it works only from the title window bar

finestructure / Hummingbird

Move/resize gestures ignored by Microsoft Excel (and other non-standard window apps) #5