Allow grids to appear over menus and popups

[chilimangoes]

Invoking one of the grid commands will cause any menus or focus dependent popups to close because Tk windows are automatically focused when they're created. The solution to this is to show the grid windows using the SW_SHOWNOACTIVATE option. Since Tk doesn't allow such low level control over the window creation process, my proposal was to implement our own window base class using ctypes to give us ultimate control.

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/27570390-allow-grids-to-appear-over-menus-and-popups?utm_campaign=plugin&utm_content=tracker%2F1825907&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1825907&utm_medium=issues&utm_source=github).

[chilimangoes]

@synkarius Coding for this is, I think, basically done. Still need to test it on Win Vista through 10, but I'm not anticipating any issues since the APIs it uses are some of the most stable core APIs in Windows. I created this issue to track discussion related to getting it ready to PR.

First question I have is regarding where to put the code. I've got a new module containing a new Window class to take the place of the Tk base class being used currently. There are also a fair number of related ctypes definitions and Structures. I'm kind of thinking of putting all under a win32 folder in lib, so that we end up with namespace/import paths that look something like the following

from caster.lib.win32.windows import Window, EventHandler
from caster.lib.win32.windows.types import *   # required ctypes definitions and structures for any direct win32 calls (not needed for basic Window usage)

I'm not a python expert, but I think I can accomplish this using the following file structure:

/lib/win32/windows/types.py
/lib/win32/windows.py

Thoughts?

[chilimangoes]

OK, that didn't work. Python doesn't seem to like file modules and directory modules with the same name. Time to read up on python modules and import behavior.

[chilimangoes]

After some more thinking, other alternatives would be:

/lib/win32/windows.py
/lib/win32/types/windows.py

Or:

/lib/win32/windows.py
/lib/win32/windows_types.py

Or for now we could just stick with:

/lib/win32/windows.py
/lib/win32/types.py

And reorganize the code in the future if we add enough stuff to win32 to warrant it. I'm kind of leaning towards this last option.

[synkarius]

I would also lean toward the third option. I would have suggested giving this stuff its own package if you hadn't, and the third way is the simplest.

[Versatilus]

I agree that the third option sounds the best.

On Wed, Oct 21, 2015 at 7:46 PM, Jacob Eggleston notifications@github.com wrote:

After some more thinking, other alternatives would be:

/lib/win32/windows.py /lib/win32/types/windows.py

Or:

/lib/win32/windows.py /lib/win32/windows_types.py

Or for now we could just stick with:

/lib/win32/windows.py /lib/win32/types.py

And reorganize the code in the future if we add enough stuff to win32 to warrant it. I'm kind of leaning towards this last option.

— Reply to this email directly or view it on GitHub https://github.com/synkarius/caster/issues/114#issuecomment-150090573.

[chilimangoes]

Disappointing news (I was afraid of this): The grid window doesn't appear over the taskbar or start menu on Windows 10. The good news is that the current grids don't either, so we're not losing anything with the new grids. So, assuming it works on Vista, I'll probably just push the current solution and file an issue specifically for the Win 10 taskbar.

[Versatilus]

I can't really say I'm surprised by that, either. WSR is in the same graphical layer, too. By now, this topic should be well enough discussed in the programming sites that there should be a solution or at least a good explanation as to how and why we are screwed.

[chilimangoes]

Le sigh...

So it turns out this doesn't work on Windows 7 anymore either. I tried the code on my work computer (Win 7) today and it failed miserably. MS has been waging a war against devs "abusing" things like HWND_TOPMOST and has now apparently made it so that topmost windows can only appear over other topmost windows (like the taskbar and menus) if they have the focus. It looks like this only worked on my home Win 7 VM because automatic Windows updates are turned off and it's embarrassingly behind on updates.

After some more research, it appears the official way to do what we need to do is to use the UI Automation API introduced in Vista. From what I've been reading, asserting uiAccess="true" in the application's manifest file is how apps like the Magnifier tool and Task Manager are able to stay above topmost windows and menus. This also has the benefit of allowing them to display over metro apps in Windows 8 and 10.

That's the good news. The bad news is that uiAccess="true" allows the application to bypass all UAC protections (as far as I can tell, it's an even higher privilege than running as administrator), which means there's a laundry list of security requirements to fulfill before Windows will let you do it. In particular, the application needs to have a manifest, the exe needs to be signed with a trusted cert, and the app needs to be installed into one of a handful of trusted locations (eg Program Files). That post says the cert also needs to be installed as a trusted root authority, but some others seem to indicate that the cert simply need to be issued by a trusted CA. Either way, these requirements would seem to rule out using python, so it looks like it's back to the drawing board.

I'll try to put together a POC in C# to do this. However, that still leaves the question of how we would integrate such a solution into caster.

I should known it wouldn't be this easy.

[synkarius]

Well. That sucks, but thanks for doing so much research.

As for integrating grids written in C# or other languages into Caster, I don't think it necessarily complicates anything unless said language requires its runtime to be installed in order for the executable to run. Is that true of C# or are you able to create standalone executables? If the latter, the best scenario is where we can get the C# app to interact with Python XMLRPC calls, as that's how the grids work now. They used to use keystrokes to receive commands, but that was somewhat unreliable, not to mention annoying. (Unreliable because it required the grids to have focus-- if we made the grids capture global keystrokes, that problem would go away; annoying because if you only have keystrokes, sometimes you have to parse complex series of keystrokes and then deal with state.)

[ghost]

This was mentioned in Gatter but it seems potentially relevant and I didn't want the resource to be lost. Open Source Code Signing allows for open source projects to apply for root certificates that's recognized by Microsoft products and mainstream browsers valid for 1 year. It requires copy of an identity document of the person placing the order, verification of a person representing the organization and verification of the open source project.

It's a shame chilimangoes, you went through through all that hard work and not having it turn out as desired. Fortunately we're not back at square one, we know a lot more thanks to your research then we did previously.

[chilimangoes]

@synkarius C# applications require that the .NET framework be installed, but that shouldn't be a problem because .NET 2.0 or better comes installed by default on Vista and up. Doing it in C would be another option, but messing with sockets and XML parsing in C does not sound like my idea of a fun time. :)

@Zone22 Thanks, I had forgotten about you mentioning that code signing offer. That may come in handy now. And you're right: I was extremely frustrated last night, but if this UI Automation API works as advertised, it should provide a much better long-term solution than the Win32-hacking-cat-and-mouse game that I was engaged in before.

[synkarius]

@chilimangoes Seems like that's the next direction to explore then. And yeah... definitely not C, heh.