search

RedBearAK / toshy

Keymapper config to make Linux work like a 'Tosh!
https://toshy.app
GNU General Public License v3.0
377 stars 18 forks source link

(Enh) NixOS formula #104

Open RedBearAK opened 1 year ago

RedBearAK commented 1 year ago

This issue is for tracking progress and discussing the development of a way to install Toshy on NixOS. Toshy is an "offshoot" of Kinto, not precisely a fork. There are substantial differences at this point.

But this project still does some things in an odd way that would probably be difficult to "package" with regular distros, and will need to be reworked to be installable on a system like NixOS with a simple "enable" declaration. I will need a lot of help to make that happen.

The basic way things are done at this point:

The way all of this works is probably not considered particularly "sane" compared to the way most packaged software is put together. I'm doing my best to slowly push it toward being more rational, but I don't have enough experience with making Linux software to do this efficiently.

There are also the extra issues with Wayland support. On GNOME, the user must choose to install any (or all) of 3 different shell extensions to provide the app-specific window info. On KDE Plasma, Toshy installs a KWin script and starts a D-Bus service to make the window info accessible to the keymapper in Wayland. There is supposedly a way to do a generic wlroots method to get the window info, that should work with all the window managers like sway that are based around wlroots. But currently I have different methods specifically for sway and Hyprland.

Atemu commented 1 year ago
  • Native packages suitable for the distro type are installed

Please just declare a list of packages you need. Nixpkgs (or other distros too for that matter) will handle the dependence on them.

  • Toshy's files are placed in ~/.config/toshy and

Nit and OT: If you aren't already, use an XDG dir library for this to handle XDG_CONFIG_DIR.

  • then a Python virtual environment is set up in the same location, where all the supporting Python modules are installed

This obviously won't fly on NixOS. We'll just package the python project natively. Please make sure you don't assume this is running in a venv at a certain path anywhere; keep it location- and packaging-agnostic.

Also make sure your build setup is sane and you don't require ancient versions of other python packages so that we can actually package it without losing our sanity.

A udev rules file changes the permissions on /dev/uinput at boot, to give access to the input group.

Wouldn't this give full input monitoring to any of the user's processes?

Is it possible to have that part run as a privileged service? At the NixOS level, the difference between privileged vs. unprivileged service declaration is just a few characters, so there isn't any complexity in this for us.

The apps all run from ~/.config/toshy via desktop files and scripts in ~/.local/bin.

Again, please ensure this isn't assumed anywhere.

  • Further runs of the installer will back up that location (but remove the venv), and create a new toshy folder, with a new Python virtual environment, and will try to merge custom changes to the config file from the backup into the new config file. The custom changes are only taken from specially-marked "slices" in the config file. So in theory if your config is currently working it will keep working after the upgrade of the config file.

Please make this optional. You wouldn't want that if the app was properly packaged.

Btw, for "unpackaged" running, you should probably store the app itself in some dir other than XDG_CONFIG_DIR. There isn't really a fitting XDG dir for this but it certainly isn't config. I think STATE_DIR would be closest or perhaps .local/bin.

RedBearAK commented 1 year ago

@Atemu

Thanks for the input. Understand that I've stretched my knowledge of how anything in Linux works, and knowledge of Python, beyond what I thought I could do several times already. So I can't promise to be able to make quick progress on making this more compatible with Nix. I didn't write Kinto's original config, or the original keymapper or its fork, and I still have limited understanding of how Linux software is normally packaged. I just messed with stuff until it worked the way I wanted, so far.

use an XDG dir library for this to handle XDG_CONFIG_DIR

I'm sure I can do at least that much, but I'll also have to make sure the collection of management scripts (Bash) are able to adapt to that as well. I do try to implement generic references to things when I can, but there will still be a lot that need to be fixed.

This obviously won't fly on NixOS. We'll just package the python project natively.

I don't really know what that means in relation to Toshy, since it's a mixture of the keymapper and the Python module it needs, and the config file (also a Python module itself) and all the modules it imports as well, and then the shell scripts that need to activate the virtual environment and start the keymapper from within the venv in order for everything to install and work on a "managed" Python environment distro. Several distros lately have activated this "managed" environment that prevents the direct use of pip.

Also make sure your build setup is sane and you don't require ancient versions of other python packages so that we can actually package it without losing our sanity.

I have had to pin a couple of the Python package versions due to some incompatibilities or bugs in newer versions. I wouldn't call anything "ancient" yet. But I don't know exactly what "sane" would mean in this context.

Wouldn't this give full input monitoring to any of the user's processes?

Probably. I'm not that clear on exactly what to do about this, since Wayland works differently. From what I understand, Kinto uses xhost to open up the X11 server to root so that the xkeysnail process running from a system service can interact with input/output. In Wayland, it seems even if there was something equivalent to xhost it would be a compositor-specific thing, meaning it would work one way in GNOME and another way in KDE, etc. But I could be wrong on that.

The keyszer dev recommended using a separate user, but until I modified keyszer with a new window context module, it was still only compatible with X11, like xkeysnail. I don't know if that approach would work with Wayland environments.

Please make this optional. You wouldn't want that if the app was properly packaged.

Yes, one of the unfortunate things about the config file is that it is literally Python, rather than a sort of static/declarative configuration format like YAML. I think in theory all the modmaps and keymaps in the config file could be recreated in a static file and just parsed into the config, but that is probably way beyond me at this point.

I've expanded the functions in the config file significantly with custom functions, so it would not be easy to switch to a different keymapper that does use a static config format. The functionality would not be the same, as far as I know.

Btw, for "unpackaged" running, you should probably store the app itself in some dir other than XDG_CONFIG_DIR. There isn't really a fitting XDG dir for this but it certainly isn't config. I think STATE_DIR would be closest or perhaps .local/bin.

I've struggled to make some sense of where to put things the whole time I've been working on this project. I had a great deal of difficulty trying to move parts of it into another location, since the main Python modules that "do" things share other components that started out living in ~/.config/kinto.

Please just declare a list of packages you need. Nixpkgs (or other distros too for that matter) will handle the dependence on them.

I have yet to understand whether there is really a generic way to do that. I think I came at this process kind of "backwards". Right now I have lists of native package names that are needed for each distro "type", with some exceptions for various quirky problems with specific distro versions. Whether what the installer does with native packages can be translated into a more declarative "ask" for dependencies, is something I would like to learn more about.

This is the set of package names that work on the various distros I've managed to test at this point:

https://github.com/RedBearAK/toshy/blob/132dae2c187c4b3f0c9285654157550501447571/setup_toshy.py#L396

I think getting this project to a state that could be packaged even for "normal" distros is going to be a multi-step affair. Not impossible but it will probably take a while.

Atemu commented 1 year ago

Thanks for the input. Understand that I've stretched my knowledge of how anything in Linux works, and knowledge of Python, beyond what I thought I could do several times already. So I can't promise to be able to make quick progress on making this more compatible with Nix. I didn't write Kinto's original config, or the original keymapper or its fork, and I still have limited understanding of how Linux software is normally packaged. I just messed with stuff until it worked the way I wanted, so far.

I wouldn't be in a different position if I were you :)

I don't really know what that means in relation to Toshy, since it's a mixture of the keymapper and the Python module it needs, and the config file (also a Python module itself) and all the modules it imports as well, and then the shell scripts that need to activate the virtual environment and start the keymapper from within the venv in order for everything to install and work on a "managed" Python environment distro.

Let's break this down:

Several distros lately have activated this "managed" environment that prevents the direct use of pip.

I'm not a python person, so I don't know what this is about but I can tell you that, on NixOS, you cannot use pip either. Not even in venvs in many cases.

I have had to pin a couple of the Python package versions due to some incompatibilities or bugs in newer versions.

Hm, that doesn't sound promising but perhaps it's not too bad.

I don't know exactly what "sane" would mean in this context.

To understand what I mean by that, there are some things you should know about distros:

There is generally only one version of a given package. We at Nixpkgs have the luxury that we can have an arbitrary amount of versions of each package but we usually don't because someone (a human) needs to manage these and human resources are very limited. Additionally, there are some further limitations in this regard placed upon us by python. We usually only have one version, perhaps a few major versions for very large projects that have had a major API change in the past few years.

This one true version of any given package is as close to the newest upstream version as possible. If you depend on features no longer present in newer version of a dependency, that usually leaves distros with the choice of either patching your package to make it compatible with non-deprecated versions, patching the dependency to stay compatible somehow (or remove bugs) or drop the package entirely.

one of the unfortunate things about the config file is that it is literally Python, rather than a sort of static/declarative configuration format like YAML.

This on its own isn't too bad. As long as it's just a single file without external dependencies, this is very much manageable. Problems start when you need abitrary external deps aswell or if the file needs to be modified at runtime.

Could you shed a little light on this config file and how it works?

I think in theory all the modmaps and keymaps in the config file could be recreated in a static file and just parsed into the config, but that is probably way beyond me at this point.

That honestly doesn't sound too hard. If you think you could transform a static config, how exactly is the config file interpreted? What is the "end result" of that config file (as in: data type)? A giant dict of booleans and/or strings? Something more complex?

Basically: How does configuration work currently?

I had a great deal of difficulty trying to move parts of it into another location, since the main Python modules that "do" things share other components that started out living in ~/.config/kinto.

This is a very clear sign that a different approach is needed ;)

I have yet to understand whether there is really a generic way to do that.

There isn't. Listing upstream project names (or even URLs!) is good enough. Actual package names vary by distro as you've elaborated.

Figuring out which package names correspond to the dependencies you need is best left to the individual distros.

Whether what the installer does with native packages can be translated into a more declarative "ask" for dependencies, is something I would like to learn more about.

Well, the way to do that is to create a "proper" package. In every serious distro a package declares its dependencies and the package manager's job is to fetch and install these dependencies.

RedBearAK commented 1 year ago

@Atemu

Could you shed a little light on this config file and how it works?

To keep it short, config file is a Python module, it imports a bunch of API things like modmap() and keymap() from a module of the keymapper (keyszer in this case, which was modified a bit from xkeysnail to fix some issues). But there are also other functions defined directly in the config file, especially in my version of the config. Like a rudimentary double-tap function that should actually be inside the keymapper, but I don't know how to implement it there yet. And a rather complex function to simplify matching on multiple context properties at once.

The modmap() and keymap() functions have a name argument, a mapping (dict) argument that will contain all the "combos" you want to remap to something else, and then a "when" conditional parameter that determines what circumstances make the keymap or modmap active. Kinto made heavy use of this conditional to create different layers of remaps from general to app groups (like terminals) and then specific apps. I actually ended up making a lot of complex conditionals to get the Toshy config to auto-adapt to different desktop environments and changing preference parameters without needing to edit the config or restart the keymapper.

The "combos" or keys to be remapped are themselves processed by a Python function, so if you look at the config inside the mappings dicts you'll see a lot of lines like: 

C("Shift-Enter"): C("Alt-Enter"),

But in the modmaps it's more like: 

Key.LEFT_META: Key.LEFT_ALT,

It is my understanding that the conditionals are re-evaluated on every key press, which is no doubt not the best way to go about things, but shifting to being able to re-evaluate only when the context changes is not something I know how to implement yet.

As far as I can tell the conditionals also need to be functions, or evaluate to a function. If there is only one function in the conditional, it just automatically gets the context object as an argument, but if there are more function I have to give each one the context object argument explicitly, like function(arg)(ctx) or it doesn't work. This has been a source of confusion frequently when I put multi-part conditionals in place.

Activation shell scripts We won't need these, we simply configure systemd to run the required processes in NixOS. (Btw: I'd recommend you to simply ship systemd user units for other distros too.)

No, the shell scripts are for more conveniently managing and monitoring the keymapper and its related services. There are "user" systemd unit files that will run the keymapper config on systems that have systemd available. But there are also scripts for running just the config manually, and in verbose mode to see debugging, which is often needed to figure out why something isn't working quite right. The scripts also get called from the tray icon menu or GUI app if you need to stop or restart the services with the mouse.

I'm not a python person, so I don't know what this is about but I can tell you that, on NixOS, you cannot use pip either. Not even in venvs in many cases.

I'm confused by that, and the fact that there is no Python interpreter available in the path on NixOS. It is obviously very different from a typical distro. Python virtual environments are a pretty standard part of developing Python projects. How is that kind of thing supposed to work on NixOS?

There isn't. Listing upstream project names (or even URLs!) is good enough. Actual package names vary by distro as you've elaborated.

That explains why I didn't really find a good generic way to lay out the dependencies. There is the OpenSUSE Build Service that supposedly provides a way to make packages for a bunch of different distros, but I have no idea how that is supposed to work.

One sticking point is that I can't just say "Toshy needs keyszer", because I actually need a specific development branch from my own fork of keyszer, where I implemented the environment API and the new window context module that adds all the different Wayland methods to the original X11 method of getting the window info. The main keyszer project is still only compatible with X11.

Atemu commented 1 year ago

I'll have to ponder over the config file a little more, thanks for the explainer.

As I said, the config file should ideally be pure data (exact format doesn't really matter much). You can encode things like conditionals or functions in data (see pretty much any half-way complex declarative deployment or CI tool) but that can be rather complicated on both ends.

An alternative approach that could be workable would be to keep the config file a python snippet but make it just that; a dumb snippet. It should assume a fixed set of dependencies is present and not provide its own. These dependencies would then be provided by the "calling" end (your service) using its interpreter rather than i.e. a "config" venv.

the shell scripts are for more conveniently managing and monitoring the keymapper and its related services.

I see. If they're necessary, it's possible to just package these scripts aswell and put them into libexec/ or something.

there are also scripts for running just the config manually, and in verbose mode to see debugging, which is often needed to figure out why something isn't working quite right.

I'd highly recommend to make this a configuration flag in a config file instead.

The scripts also get called from the tray icon menu or GUI app if you need to stop or restart the services with the mouse.

Please ensure you don't have strong assumptions on where those scripts are. They will not be in .config on NixOS, I can tell you that much. Some ways to achieve this is adding the possibility to pass a path to those scripts to the tray menu/gui app as a command-line arg or env var.

I'm confused by that

This is due to the fact that you cannot run arbitrary dynamically linked binaries on NixOS as they assume FHS directories that do not exist. There is no /usr/lib which could contain a dynamic linker for instance. Any wheel bundling binaries that make such FHS assumptions will not work.

Pip is generally not a very user-friendly thing. It's a developer thing and it's good enough for that I guess but not suited for user package management. Leave that to the distros.

and the fact that there is no Python interpreter available in the path on NixOS.

You could make one available if you wanted to but there usually isn't one by default in the global environment.

For any python program, we craft an interpreter which has access to all the python packages the python program in question. It's a very "local", limited and precise scope where the desired interpreter and packages are available.

We generally rarely "pollute" the global environment with anything and rather build our own "venv"s for every package, including non-python ones. (Generic ELF binaries get all their "dynamically"-linked lib deps put into their rpath for instance.)

This approach allows us to use conflicting versions of things without any conflicts because almost nothing is global and therefore cannot conflict.

How is that kind of thing supposed to work on NixOS?

There isn't really a fixed way of doing things on NixOS and the way to do things varies depending on what it is you're trying to do.

For a development environment, you'd use tools like poetry2nix to create a local dev shell with all the dependencies in it. That's kinda like a venv but you can use any package in Nixpkgs too, not just python packages.

For deployment, you'd package the project and its dependencies "properly" (one hand-written derivation for each package; fixed versions) and then deploy it like any other package (i.e. put the binaries into the global PATH or run the binary as a service).

As I mentioned though, I'm not a python person, so some of the details here are lost to me but that's the general idea.

One sticking point is that I can't just say "Toshy needs keyszer", because I actually need a specific development branch from my own fork of keyszer, where I implemented the environment API and the new window context module that adds all the different Wayland methods to the original X11 method of getting the window info. The main keyszer project is still only compatible with X11.

Just.. say that. In plain old English; that's good enough. Packagers are people too, we can comprehend words ;)

We at Nixpkgs have the luxury that we could simply package your fork aswell with minimal effort but, keeping other distros in mind, it'd probably be easier for everyone if you either upstreamed your fork's changes or created a "proper" fork with different name, library namespaces etc. such that it can't or conflict with the upstream project or be confused.

RedBearAK commented 1 year ago

@Atemu

I'll have to ponder over the config file a little more, thanks for the explainer.

I forgot to include that there are a number of places in the config that combos actually remap to functions, or macros that include a function. So that's another crimp in trying to translate those parts of the config into something that could be parsed from a static declarative format. Probably there are ways, but developing and debugging such a thing might be unpleasant. I'm very used to the way VSCode can link everything together in a way that makes everything easier to understand, like being able to Cmd+click on a function name to go right to where it is defined.

I see. If they're necessary, it's possible to just package these scripts aswell and put them into libexec/ or something.

That's a partial path so I can't tell if you mean some sort of system-level libexec folder or something that would still exist in the user's home. All the scripts/commands are hosted in home now and in fact are set up to refuse to run as the superuser, since the config belongs to the user and the intent was to not have any system-level remapping going on. Again, partly to be sure it would work in a Wayland environment, but also so another user could have no remapping or a completely different type of config file or keymapper in place.

I'd highly recommend to make this a configuration flag in a config file instead.

Eh, that seems kind of inconvenient. A lot of times the user just needs to run the verbose logging long enough to see that they missed a comma or something, then restart back to the normal service with no verbose flag once they fix the config file and save again. I'm sure it's do-able though.

Please ensure you don't have strong assumptions on where those scripts are. They will not be in .config on NixOS, I can tell you that much. Some ways to achieve this is adding the possibility to pass a path to those scripts to the tray menu/gui app as a command-line arg or env var.

Making the paths more adaptable is definitely something that needs a lot of work. I do pull in a few things already from the environment, like the user's "run" path for temp files. (Ran into at least one distro that for some reason didn't clear /tmp on rebooting, so I had to start trying to use XDG_RUNTIME_DIR when available. Wish the distros could have a better consensus on such things.)

This is due to the fact that you cannot run arbitrary dynamically linked binaries on NixOS as they assume FHS directories that do not exist. There is no /usr/lib which could contain a dynamic linker for instance. Any wheel bundling binaries that make such FHS assumptions will not work.

Ah, so things really need to start out pretty self-contained. Understandable, if you want consistency.

Pip is generally not a very user-friendly thing. It's a developer thing and it's good enough for that I guess but not suited for user package management. Leave that to the distros.

It definitely has issues, such as the problems I ran into that led to needing to pin a couple of the Python packages installed in the venv with pip. Unfortunately most distros other than Arch haven't yet adopted the approach recommended by the Python devs of making every Python package available as a native package, individually.

If I'm understanding the notes on how the Nix package system works (and I've heard people talking about using it on distros besides NixOS), the Nix package basically creates an isolated environment sort of like a Flatpak or container, where everything the app needs is part of the package.

I've had issues trying to get Toshy working in containers, mainly with accessing the uinput device, but I'm guessing that won't necessarily be an issue with a Nix package, since it's more like a Python virtual environment and not as isolated as a container. Just guessing.

I'm thinking the main thing I can start looking at for now is just making all paths adaptable to whatever the environment provides.

Just.. say that. In plain old English; that's good enough. Packagers are people too, we can comprehend words ;)

That sounds like you're expecting someone besides me to step up and actually do the packaging, once the project is in more of a "sane" state.

It's a bit frustrating even thinking about making a bunch of big changes to the project when it technically works right now on quite a few different distros, with an installer script that I've taken great pains to make easily adaptable to using new native package managers and appropriate package lists. But, bit by bit, maybe it will be possible to make it more sane without straight up breaking it along the way.

Some specific ideas about how to adapt the path references might be helpful.

ddulic commented 11 months ago

Being on a Steam Deck, I would love to see this happen, as the current installer doesn't work and would get overwritten on an update anyway.

Installer Error ```bash venv ❯ ./setup_toshy.py install --override-distro arch (DD) Home user local bin not part of PATH string. (!!) NOTICE: It is ESSENTIAL to have your system completely updated. Have you updated your system recently? [y/N]: y The "~/.local/bin" folder is not in PATH. OK to add it? [Y/n]: § Getting environment information... ================================================================================ The active init system is: 'systemd' (Systemd) (EV) Toshy installer sees this environment: DISTRO_NAME = 'arch' DISTRO_VER = '3.5.7' VARIANT_ID = 'steamdeck' SESSION_TYPE = 'x11' DESKTOP_ENV = 'kde' DE_MAJ_VER = '5' ----------------------------------- -- PASSWORD REQUIRED TO CONTINUE -- ----------------------------------- [sudo] password for deck: Using elevated privileges... § Installing native packages for this distro type... ================================================================================ error: failed to init transaction (unable to lock database) error: could not lock database: Read-only file system (EE) ERROR: Problem installing package list for distro type: Command '['sudo', 'pacman', '-S', '--noconfirm', 'gobject-introspection', 'libappindicator-gtk3', 'pkg-config', 'python-pip']' returned non-zero exit status 1. venv ❯ ./setup_toshy.py install --override-distro arch (DD) Home user local bin not part of PATH string. (!!) NOTICE: It is ESSENTIAL to have your system completely updated. Have you updated your system recently? [y/N]: y The "~/.local/bin" folder is not in PATH. OK to add it? [Y/n]: § Getting environment information... ================================================================================ The active init system is: 'systemd' (Systemd) (EV) Toshy installer sees this environment: DISTRO_NAME = 'arch' DISTRO_VER = '3.5.7' VARIANT_ID = 'steamdeck' SESSION_TYPE = 'x11' DESKTOP_ENV = 'kde' DE_MAJ_VER = '5' ----------------------------------- -- PASSWORD REQUIRED TO CONTINUE -- ----------------------------------- [sudo] password for deck: Using elevated privileges... § Installing native packages for this distro type... ================================================================================ error: failed to init transaction (unable to lock database) error: could not lock database: Read-only file system (EE) ERROR: Problem installing package list for distro type: Command '['sudo', 'pacman', '-S', '--noconfirm', 'gobject-introspection', 'libappindicator-gtk3', 'pkg-config', 'python-pip']' returned non-zero exit status 1. ```
RedBearAK commented 11 months ago

@ddulic

Being on a Steam Deck, I would love to see this happen, as the current installer doesn't work and would get overwritten on an update anyway.

Can you duplicate this post in a new issue thread? I think the situation with the Steam Deck's immutable setup and only being able to use Flatpaks or AppImages is a bit different.

ddulic commented 10 months ago

@ddulic

Being on a Steam Deck, I would love to see this happen, as the current installer doesn't work and would get overwritten on an update anyway.

Can you duplicate this post in a new issue thread? I think the situation with the Steam Deck's immutable setup and only being able to use Flatpaks or AppImages is a bit different.

As of SteamOS 3.5 it has a permanent /nix partition, which I use for all desktop related setup, but yes, it is technically a different issue.

I have created a new issue, it is referenced above. Thanks!

mlyxshi commented 10 months ago

It is very difficult to package toshy into NixOS system. I tried to package toshy in NixOS, but give up(I have the basic idea how this works, but it seems that packaging toshy need to rewrite many parts of the project).

For nix based users, xremap is an alternative, and https://github.com/xremap/nix-flake is deigned specifically for NixOS/home-manager.

RedBearAK commented 10 months ago

@mlyxshi

seems that packaging toshy need to rewrite many parts of the project

Yes, most likely. I'm not even sure where to begin.

For nix based users, xremap is an alternative, and https://github.com/xremap/nix-flake is deigned specifically for NixOS/home-manager.

I'm sure there are a number of different Linux keymappers that can be installed on NixOS or via the Nix package manager on other distros. But the point of this repo is to set up a keymapper with a built-in config that makes the keyboard act like a Mac. The config is over 3,000 lines at this point, though half of that is the Option-key special character stuff that many users don't actually need.

If it were possible to translate the Toshy or Kinto config into something xremap or something else could understand, then you would actually have an "alternative" to Toshy.

On the other hand, if all you want is the keymapper (keyszer) to be installable on NixOS, you might want to open an issue on the keyszer repo:

https://github.com/joshgoebel/keyszer/issues

I believe that keyszer does have some functionality that xremap is still lacking at the moment (like matching on window titles, not just app classes).

But the main/sole dev on that repo hasn't really had time to mess with it for quite a while now. I've been adding all the Wayland support to keyszer in a development branch of my own fork of keyszer.

BTW, the flake repo you pointed to only seems to support setting up xremap for Sway and X11, while xremap has support for several Wayland environments. It says KDE and GNOME are not implemented, and Hyprland has an issue.

celesrenata commented 7 months ago

I assuming this task is still at square one.

I'm still new to NixOS, but I've made an attempt. So far I've been able to stand up python-xlib 0.31 (I am aware you were looking into seeing if this older dependence can be updated), keyzer 0.6.0, hyprpy 0.1.5, i3ipc 2.2.1, tk 0.1.0, created a setup.py for toshy and a requirements.txt and build the app. I still have to learn how to do the rest!

I will hopefully have time to fiddle with it more this weekend and post the progress.

RedBearAK commented 7 months ago

@celesrenata

I assuming this task is still at square one.

Uh, more like square zero.

Sounds like you're having fun. And making some progress.

FYI: Toshy clones a significantly modified branch of keyszer from my fork of the original repo, which provides all the Wayland support, and the custom API function (environ_api) that allows injecting the detected environment info into the keymapper from the config file. That lets the keymapper automatically pick the correct window context method to use, and if the env.py module is working it means you can easily log out of one environment and into a completely different desktop and session type and have Toshy keep working without the user doing anything. As long as each environment is actually compatible with the current state of the keymapper.

That environment module is part of Toshy because the keyszer dev didn't want anything complicated like that to be integrated into the keymapper itself. Without that module, you'd need to manually set the environment info in the config file to make this branch of keyszer work for anything but X11 sessions.

When you say "keyszer 0.6.0" it suggests you're drawing from the original fork. That's unlikely to work other than with X11 sessions, and X11 is rapidly being replaced by Wayland environments in multiple popular distros. I arbitrarily changed the version in my branch to 0.7.1 a while back just so it's easier to tell the difference in the log at startup, otherwise it doesn't really mean anything specific.

The current branch that is being cloned when you install Toshy the usual way is: 

        self.keyszer_branch         = 'device_grab_fix'
        # self.keyszer_branch         = 'environ_api_hyprland'
        self.keyszer_url            = 'https://github.com/RedBearAK/keyszer.git'
        self.keyszer_clone_cmd      = f'git clone -b {self.keyszer_branch} {self.keyszer_url}'

This periodically changes when I need to introduce some new fixes or context methods. I keep the older branches as backups in case I really screw something up.

I have a requirements.txt file in the project now but it's just for informational purposes, so GitHub can show some dependencies. It combines the requirements for both keyszer and Toshy's companion apps (the GUI and tray app).

I've never actually looked into doing a standard setup.py for this project, just wrote the setup_toshy.py from scratch based on what I needed the installer to do to get the system ready (like installing the udev rules file and getting the user into the necessary group). I have no idea how much of that can be done in a typical setup.py. Or how a NixOS "formula" works and what the limitations might be.

I am certainly interested in seeing how you get on if you stick with it, and hopefully gain some understanding of all the things I'm doing wrong that make it so difficult to get this project to install in NixOS.

Atemu commented 7 months ago

I arbitrarily changed the version in my branch to 0.7.1 a while back just so it's easier to tell the difference in the log at startup

It'd be great if you'd just rename it. keyszer-wayland for instance.

celesrenata commented 7 months ago

If you can make a new repo with keyszer-wayland alone it would make ingesting it easier as a dependency. I can work on updating that override once I have that.

RedBearAK commented 7 months ago

Don't wanna. But maybe I gotta.

Will update if I go that direction.

celesrenata commented 7 months ago

Refactored and uploaded to: https://github.com/celesrenata/nix-flakes/blob/main/overlays/toshy.nix:

RedBearAK commented 7 months ago

Refactored and uploaded

Whoa. "I am only an egg."

Still need to rewrite shell scripts.

Very curious what that will involve.

Still need to investigate what buildPythonApplication vs buildPythonPackages actually produces (does Application have its own venv already I can target instead?).

Well, the installer script builds a venv during each install, which ends up located at ~/.config/toshy/.venv. That's where all the pip-installed support stuff goes, including keyszer, and where the keymapper gets run from. You can see the venv getting activated in the shell scripts that actually launch the keymapper.

Not sure if you need more info, or if that venv is actually useful in the Nix context.

celesrenata commented 7 months ago

this was my modified requirements.txt (still old, for previous reasons stated)

###############################################################################
# WARNING: This is just a sample 'requirements.txt' file for Toshy. 
# This is NOT actually meant to be used manually!
# 
# The main purpose of making this file is for GitHub to be able to show dependencies.
# Native package installs are used to provide additional Python modules not named here, 
# and prep build/compile dependencies for some of these pip packages. 
# The 'setup_toshy.py' script handles native package installs and all these pip 
# package installs automatically. 
# It also installs all pip packages in a 'venv' Python virtual environment, 
# to be more compatible with "managed" Python environments where 'pip' usage
# is restricted. 

# The Toshy installer upgrades these first, to avoid showing error messages in the log.
pip
wheel
setuptools
pillow

# Pinning pygobject to 3.44.1 (or earlier) to ensure compatibility with RHEL 8.x and its clones.
pygobject<=3.44.1

# Standard packages required for the application.
lockfile
dbus-python
systemd-python
tk
sv_ttk
watchdog
psutil
hyprpy
i3ipc
pywayland

# Installing 'pywlroots' requires native package 'libxkbcommon-devel' on Fedora.
# pywlroots

# All dependencies below here are to smooth out the installation of the custom
# development branch of `keyszer` needed to make Toshy work.
# Will leave these exposed here even if they are not technically 
# direct dependencies of Toshy, since Toshy functionality 
# depends entirely on `keyszer`.

inotify-simple
evdev
appdirs
ordered-set
six

# TODO: Check on 'python-xlib' project by early-mid 2024 for a bug fix related to:
# [AttributeError: 'BadRRModeError' object has no attribute 'sequence_number']
# If the bug is fixed, consider updating the version pinning.
python-xlib==0.31
keyszer==0.6.0

this was my nix-shell.nix I have to stand up the venv manually. (Still old for the same reasons)

{ pkgs ? import <nixpkgs> {
  overlays = let

      in [
        (import ./overlays/toshy.nix)
      ];
    }
  }: 
(pkgs.buildFHSUserEnv {
  name = "pipzone";
  targetPkgs = pkgs: (with pkgs; [
    pkg-config
    cairo.dev
    xorg.libxcb.dev
    xorg.libX11.dev
    xorg.xorgproto
    dbus.dev
    glib.dev
    systemd.dev
    linuxHeaders
    gobject-introspection
    gobject-introspection.dev
    libffi.dev
    python311
    python311Packages.pip
    python311Packages.virtualenv
    python311Packages.pycairo
    python311Packages.wheel
    python311Packages.setuptools
    python311Packages.pillow
    python311Packages.pygobject3
    python311Packages.lockfile
    python311Packages.dbus-python
    python311Packages.systemd
    python311Packages.tkinter
    python311Packages.sv-ttk
    python311Packages.watchdog
    python311Packages.psutil
    python311Packages.i3ipc
    python311Packages.pywayland
    python311Packages.pywlroots
    python-xlib
    python-keyszer
    python-hyprpy
    wrapGAppsHook
    ]);
  profile = ''
    set -e
    # Tells pip to put packages into $PIP_PREFIX instead of the usual locations.
    # See https://pip.pypa.io/en/stable/user_guide/#environment-variables.
    export PIP_PREFIX=$(pwd)/_build/pip_packages
    export PYTHONPATH="$PIP_PREFIX/${pkgs.python3.sitePackages}:$PYTHONPATH"
    export GI_TYPELIB_PATH="/nix/store/*-gtk+3-*/lib/girepository-1.0:$GI_TYPELIB_PATH"
    export PATH="$PIP_PREFIX/bin:$PATH"
    unset SOURCE_DATE_EPOCH
    set +e
   '';
  runScript = "bash";
}).env
  1. nix-shell pip-shell.nix
  2. virtualenv ~/.config/toshy/.venv
  3. source ~/.config/toshy/.venv/bin/activate
  4. pip install -r ~/sources/toshy/requirements.txt
  5. python ~/sources/toshy/toshy_tray.py
Error: The file /home/celes/sources/toshy/toshy_config.py was not found.
    [Errno 2] No such file or directory: '/home/celes/sources/toshy/toshy_config.py'
Traceback (most recent call last):
  File "/home/celes/sources/toshy/toshy_tray.py", line 179, in <module>
    gi.require_version('Gtk', '3.0')
  File "/home/celes/sources/nixos/_build/pip_packages/lib/python3.11/site-packages/gi/__init__.py", line 126, in require_version
    raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gtk not available
~/sources/nixos via 🐍 v3.11.8 (.venv) via ❄️  impure (pipzone-shell-env) ❯

I will continue to fiddle with these issues. Though, this method is obviously not the most Nix way. It may help me figure out some other problems along the way.

I will rework the requirements as previously stated.

RedBearAK commented 7 months ago

@celesrenata

I've never been all that clear exactly where the gi.repository stuff comes from, but you seem to have the gobject-introspection packages in there already, so I'm not sure what else to point at to overcome the namespace error.

I'm working on a renamed fork of keyszer based on the branch that Toshy is currently cloning. But I don't know when I'll even be able to test it. With the new name I'll have to make a branch of Toshy and change the scripts that reference keyszer before I can even do a test run. And it still won't be available via pip. But maybe it will be easier than trying to install from the moving branch that I use in the Toshy setup script.

On the other hand, for now I could designate a specific branch on the existing keyszer fork repo that I just always merge new changes into. Like a "toshy_main" branch or something. Then I'd just manually edit the setup script if I need to test a development branch before merging it. It's my understanding that a Nix formula can install from basically anywhere, as long as it's a reliable reference point that doesn't change.

RedBearAK commented 7 months ago

@celesrenata

That toshy_main branch is active, and I switched over the setup script and did a test install. Seems to be working fine.

https://github.com/RedBearAK/keyszer/tree/toshy_main

RedBearAK commented 7 months ago

@celesrenata @Atemu

Unless there is some kind of objection, the new fork of keyszer is called xwaykeyz, and I've started it off with semantic versioning at v1.0.0. It is NOT available on PyPI.org since I don't have an account there and have never tried to publish anything that way. But I modified Toshy in the dev_beta branch to clone this new repo and do all its imports and keymapper command launches as xwaykeyz instead of keyszer, and it has so far installed and worked on multiple VMs with different desktop environments. So it seems to be fully working under the new name.

https://github.com/RedBearAK/xwaykeyz

The name is a combination of X11, Wayland, keyszer, and even Sway if you choose to pronounce it a certain way.

RedBearAK commented 7 months ago

@celesrenata

I've updated the requirements.txt in the dev_beta branch of Toshy, even including the xwaykeyz Git repo with syntax described by GPT-4. Supposedly this is also something pip can handle. 

# Migrating to using a new fork of `keyszer` called `xwaykeyz`
# Not available via `pip` yet!
# https://github.com/RedBearAK/xwaykeyz/
git+https://github.com/RedBearAK/xwaykeyz.git@main#egg=xwaykeyz

I also did some updates to the pyproject.toml of the xwaykeyz repo to hopefully bring it up to date with all the Wayland-related nonsense I had to add to keyszer. If I didn't miss anything, it should be possible to install and use xwaykeyz independently of Toshy's installer script and it will install all of its own dependencies by itself. This was something I hadn't paid much attention to since I only cared about installing it alongside the Toshy config, so I was just letting the Toshy installer take care of the extra dependencies.

Took the version numbers from what I have freshly installed in a new venv here. Seems to match up with what you've put in your Nix formula for the same packages. 

dependencies = [
    "appdirs ~= 1.4",
    "dbus-python ~= 1.3.2",
    "evdev ~= 1.5",
    "hyprpy ~= 0.1.5",
    "i3ipc ~= 2.2.1",
    "inotify_simple ~= 1.3",
    "ordered_set ~= 4.1",
    "python-xlib == 0.31",
    "pywayland ~= 0.4.17",
]

There's other stuff in there including a proper email address I just set up for this purpose.

I hope all of this will be helpful in closing some gaps and getting Toshy working more easily.

Keep in mind that other than verifying that switching to cloning xwaykeyz gives me a working Toshy install, and installing xwaykeyz one time in a new venv from the local clone to satisfy VSCode while working on the Toshy local repo clone, I haven't done any extensive testing of installing xwaykeyz in a clean environment like a fresh VM or anything, all by itself. So I could potentially still be overlooking something.

But I think it is in much better shape now than it was yesterday.

The Xlib pinning is firm, and may need to stay that way for a long time. There's no sign of any active maintainer on the python-xlib repo that may ever merge in a fix for the troublesome exception classes that were introduced in v0.32.

celesrenata commented 7 months ago

I have added the new dependencies you have specified and they have built. I will dive in again tomorrow to see what results I have netted! Requirements are updated in: https://github.com/celesrenata/nix-flakes/blob/main/overlays/toshy.nix

RedBearAK commented 7 months ago

@celesrenata

I'm concerned about something important that the standard Toshy does to allow users to safely update or reinstall Toshy, while retaining customizations they make to preferences (stored in an sqlite3 database file), and to the config file itself, where a few different settings often need to be tweaked a bit from the default config file state. And some users add a lot more than those little tweaks.

The way the Toshy setup script deals with this is a multi-step process.

This way, when the services restart, the user is left with Toshy behaving the same way as it was prior to the reinstall/upgrade, but with possibly new remaps/keymaps/functions in the config file (outside those editable "slices") or even completely new imports (like those going along with the migration from keyszer to xwaykeyz). A user could basically download a new Toshy zip every day and reinstall, without damaging their custom Toshy behaviors. At least, that was the idea, and I haven't run into any real problems with the procedure in testing. I even reinstall over my own personal configuration these days, periodically.

Without at least doing a full backup of the user's config folder each time the Nix formula is run, which would allow manually copying back the sqlite3 database and doing a manual merge of customizations from the previous state of the user's config file, the experience of reinstalling on Nix may be quite inferior to what the standard Toshy installer script provides on standard distros.

There should definitely be more sanity to the separation of the user's customizations and the rest of the "code" that makes things run, but right now that separation isn't available. It's been all I could do to make the situation better than what the Kinto installer does, which still just overwrites the user's config file entirely, with no backup. I think I alluded to some of this very early in this thread.

Handling any of this in the Nix formula can probably wait until later. Getting Toshy to install cleanly and run in the first place will be a big deal. But this process is an intrinsic part of keeping Toshy working across upgrades without inconveniencing the user excessively. For now.

celesrenata commented 7 months ago

That will be a bridge to cross for sure. As I learn more about this distro I might come up with some ideas.

Atemu commented 7 months ago

The way the Toshy setup script deals with this is a multi-step process.

  • Read out the contents of marked "slices" in the user's current config file into memory.

  • Copy the sqlite3 preferences file to a temporary location.

  • Create a backup of the user's entire config folder, minus the large venv.

  • Delete the user's current config folder.

  • Put an entirely new Toshy config folder in place.

  • Build a new venv with all necessary prerequisites.

  • Merge the custom content from the "slices" back into the new default config file.

  • Copy the sqlite3 database from temp location into the new config folder.

This way, when the services restart, the user is left with Toshy behaving the same way as it was prior to the reinstall/upgrade, but with possibly new remaps/keymaps/functions in the config file (outside those editable "slices") or even completely new imports (like those going along with the migration from keyszer to xwaykeyz). A user could basically download a new Toshy zip every day and reinstall, without damaging their custom Toshy behaviors. At least, that was the idea, and I haven't run into any real problems with the procedure in testing. I even reinstall over my own personal configuration these days, periodically.

Preface: i still haven't found the time/priority to look into Toshy in detail yet but I do have experience in designing configuration APIs in NixOS.

That procedure strikes me as overly complex. Mixing different kinds of state which don't belong with another together like that is also not a good idea.

Default configuration should ideally never be stored anywhere but rather implied at runtime. That way the exact same config state (i.e. config file) could be used on many different versions of Toshy with different defaults but otherwise same configured behaviour without any state changes required.
In such a design, each option would ideally have an additional state: The default/unset. A boolean option for instance could be set to either True, False or default/unset. In the unset case, the software would assume whatever it wants depending on the code.

Also, what is the use-case for sqlite here? Databases as "configuration files" are usually a pain to deal with for users and rarely necessary.

Without at least doing a full backup of the user's config folder each time the Nix formula is run, which would allow manually copying back the sqlite3 database and doing a manual merge of customizations from the previous state of the user's config file, the experience of reinstalling on Nix may be quite inferior to what the standard Toshy installer script provides on standard distros.

This is not possible. Nix "formulas" (derivations) cannot have such side-effects by design.

One way to work around this is to wrap the program executable with a script which checks the config state and runs the state migration.

Ideally though, none of this should be necessary. Apart from a set of static user config files, Toshy should be a stateless application without any need for complex migrations.

RedBearAK commented 7 months ago

@Atemu

That procedure strikes me as overly complex. Mixing different kinds of state which don't belong with another together like that is also not a good idea.

Yes, it is overly complex, and annoying. But it solved the problem of installing a new version of the config without having to manually merge back in potentially complex changes, or just losing those changes unless the user has a backup. I have a user that added their own whole custom modmap. With an editable "slice" to put it in, it isn't a big deal.

When something goes wrong with the merging the main thing that will happen is the config just won't start and the user will have to fix a syntax issue.

Also, what is the use-case for sqlite here? Databases as "configuration files" are usually a pain to deal with for users and rarely necessary.

It seemed the safest and easiest way (that I could discover and implement) to allow the tray icon menu, the GUI app, and the config file to save and/or "sync" settings "live" without the user needing to restart anything. You change a preference in the tray icon menu, watchdog sees that the sqlite3 file has changed, the config reloads settings from the database and changes its state in memory, and if the GUI app is open it will almost immediately reflect the same setting change by changing switch states in the UI. Nothing gets weirdly out of sync with what the currently saved settings actually are, or needs to be closed and re-opened to reflect the true settings. The sqlite3 connection is literally designed to be usable by multiple "users" nearly simultaneously (in this case separate processes), so less risk of some weird race condition from editing the same INI file or whatever.

By all means lay out a better way to do such things. There are probably several more sensible ways. But that was also how I tacked on the ability to watch the Synergy log file to allow disabling all the remaps when the cursor/focus leaves the screen. I don't know of any other way to deal with that strange case. Synergy doesn't present a proper class/name so it's otherwise difficult to know when to turn off the remapping in the same way as happens for other kinds of "remote" and VM apps.

Ideally though, none of this should be necessary. Apart from a set of static user config files, Toshy should be a stateless application without any need for complex migrations.

Just like at the start of the thread, the idea is nice, the implementation is out of my reach without specific concrete ideas for what to change.

One way to work around this is to wrap the program executable with a script which checks the config state and runs the state migration.

This seems like a concrete idea, that I will have to think about. I guess that could/should happen from within the launcher scripts, just before the keymapper process is launched. Though I don't know how the launcher scripts will need to be rewritten to operate in the Nix environment.

There is always a copy of the true default Toshy config files in the default-toshy-config folder. That is what is used to create the new user config file with the potentially customized "slices" from the previous config file merged into it. It's always available in case the user really screws up their config and just wants to start over.

The merging operation could probably be done (if necessary) prior to actually launching the keymapper, instead of during install.

I could store a hash of the default config file and only do the splicing if the default file has changed. Hmm...

Atemu commented 7 months ago

it solved the problem of installing a new version of the config

Why is it necessary to "install a new version of the config"?

A program should never change the structure of a config file. At the worst, it should be able to change a single value according to explicit user input (i.e. settings menu) while preserving structure. git config does this very well for instance.

You should treat configuration files like an API and apply the same principles you'd apply to APIs: Try to do as few breaking changes as possible and if you have to do them, do them in larger batches and guard them behind API versions; retaining the old API for a while to give users time to migrate their end of the API to a newer version.
It's then up to the user to migrate their config file. Here you could provide your automatic migration mechanism as a helpful aid that the user could opt to use but such a migration should be something the user explicitly chooses to do.

Designing the software following that principle will make it a lot more robust and user-friendly.

It seemed the safest and easiest way (that I could discover and implement) to allow the tray icon menu, the GUI app, and the config file to save and/or "sync" settings "live" without the user needing to restart anything. You change a preference in the tray icon menu, watchdog sees that the sqlite3 file has changed, the config reloads settings from the database and changes its state in memory, and if the GUI app is open it will almost immediately reflect the same setting change by changing switch states in the UI.

Okay but why would you need an sqlite database for that? The format of the "config file" does not affect the procedure you described at all; you could achieve the exact same using a plain text file.

Please note that the problem you describe is better solved using the observer pattern wherein you have one canonical place through which state changes must go through (or get noticed by) which then notifies all components that registered to care about such state changes.

This pattern could be retrofitted into your design by making the watchdog send a simple IPC message to the other daemons upon a state change; triggering them to reload. This is commonly done using the HUP signal.

The sqlite3 connection is literally designed to be usable by multiple "users" nearly simultaneously (in this case separate processes), so less risk of some weird race condition from editing the same INI file or whatever.

Ah, so that's why. What you're looking for is the design pattern of locking/mutexes. Locks facilitate exclusive access to a certain resource which in this case the config file. While another entity has exclusive access, all other entities must wait for it to finish and then acquire an exclusive lock for themselves before accessing the resource.

I'm not very well versed in python but I'm sure there are dozens of standard methods of achieving this and I'm sure you'll be able to pick out a python library which implements all of this for you now that you know about this pattern's existence and name.

But that was also how I tacked on the ability to watch the Synergy log file to allow disabling all the remaps when the cursor/focus leaves the screen. I don't know of any other way to deal with that strange case. Synergy doesn't present a proper class/name so it's otherwise difficult to know when to turn off the remapping in the same way as happens for other kinds of "remote" and VM apps.

Check whether they did their job properly and offer a sort of IPC mechanism observe such events (i.e. dbus).

Otherwise, while quite hacky, the solution of watching the log file as a proxy isn't that bad. Again though, sqlite is not required for that to work.

I don't know how the launcher scripts will need to be rewritten to operate in the Nix environment.

Im not sure what you mean by this. You can modify any state in any way you want but Nix aims to reduce software state (the files defining the programs themselves) to 0. A program packaged by Nix is immutable and therefore stateless. You can't (or rather shouldn't) change the software but you can freely modify program-specific state such as configuration files or caches.

As with any runtime executable, we need to explicitly declare its dependencies on other components. If this is a shell script, simply declare which executables you expect to have in the $PATH beyond coreutils. It's trivial for us to make the script work at that point.

As for the migration script contents, make sure it only does that one thing: Migrate the config state from config state format version n to n+1.

There is always a copy of the true default Toshy config files in the default-toshy-config folder. That is what is used to create the new user config file with the potentially customized "slices" from the previous config file merged into it. It's always available in case the user really screws up their config and just wants to start over.

So the idea I attempted to convery above is that, rather than applying this concept of a default set of configuration at the config file level, apply it at the level of the abstract representation of settings in the runtime program. If the user provides an empty configuration file, the program should set all values to their defaults in its abstract configuration representation at runtime. Should the config file contain some settings, you'd first apply the default values and then overwrite the settings the user modified with their desired values.
This is effectively a robust mechanism of merging the user config with the defaults without having to touch the user config files at any point.

This way, there should be no reason to store the default configuration state with the user configuration state and therefore no reason to modify the configuration state outside of explicit config format migrations which should ideally be rare.

RedBearAK commented 7 months ago

@Atemu

First of all, I'd like to observe that I appreciate any attempt at explaining important concepts or offering ideas about how to fix things. I'll just always have a limited ability to follow through on any of it (quickly).

Why is it necessary to "install a new version of the config"?

I didn't write the keymapper, or any of its ancestors. The keymapper was written such that what it calls its "config" file is unfortunately just another Python file/module, where you put together things like keymap and modmap functions (defined in the API module) and supporting lists and whatnot, in order to get the whole program to do what you want. In keyszer I think it changed a bit from xkeysnail, and uses a Python exec command to load the config. I believe this is somewhat analogous to a source command in a bash shell, which is why the keyszer dev insists that the "config" file doesn't actually need to do any of its own imports to function anymore. (But I still do those imports in my version of the config file so that I don't go crazy trying to maintain it in VSCode, while the keyszer dev apparently just "turns off linting".)

So I didn't start out modifying something that specified any kind of static configuration state to be parsed. It's literally "part of the program". Every time I add a new keymap or fix even a single shortcut remap, that whole file needs to be replaced in order for users to have that new "feature" or fix of its overall keymapping behavior. Basically the user's customizations of that same config file get mixed with my own "base" customizations of what I started with years ago (the default Kinto config). Thus the slicing and dicing strategy. It works as long as the users stick to their own areas and don't make any syntax errors, and I do the same on my side. Big changes in how the config works would probably necessitate telling users to do a manual merge to some new major "version" of the config. But so far changes are normally pretty minor.

I've never disagreed that this is not a great way to do things. But the fact that the "config" is just Python has enabled a number of interesting capabilities, like a whole custom function sitting inside the config file (matchProps) that makes it more intuitive to put together exact matching criteria for when keymaps/modmaps become active. I had wanted to add that function and other things into the keymapper code proper (config_api module), but the keyszer dev was not interested in complicating the keymapper base code. Now that I'm maintaining a full renamed fork of the keymapper, I might change that, and possibly even integrate the "environment" module that I designed for Toshy to abstract away the need for the user to do specific config changes to make the keymapper setup work correctly for different distros and desktop environments (and X11 vs Wayland sessions). All of that is handled automatically with Toshy.

The matchProps function is also where part of the Synergy fix had to be implemented, with a function/closure that checks the screen focus state variable and just disables all modmaps and keymaps when necessary. Fortunately a long time ago I had converted all the when conditionals to standardize on using matchProps, so this worked out perfectly.

Designing the software following that principle will make it a lot more robust and user-friendly.

Yes, it will. Eventually.

This pattern could be retrofitted into your design by making the watchdog send a simple IPC message to the other daemons upon a state change; triggering them to reload. This is commonly done using the HUP signal.

I will look into that. I think I tried something like it (some sort of pipe-related thing), but it just seemed actually simpler to both save the settings to a file and watch it to trigger the updates. In a sense it is just a slightly delayed IPC mechanism.

And, I have to load the settings from the file to begin with, so it seemed relatively easy to just reload those settings whenever the file changes. All of the components of Toshy import the same Python module/file to deal with the preferences, so there isn't really any code redundancy to speak of.

Check whether they did their job properly and offer a sort of IPC mechanism observe such events (i.e. dbus). Otherwise, while quite hacky, the solution of watching the log file as a proxy isn't that bad.

About that. The "solution" of watching the Synergy log file was actually suggested by the Synergy devs, to the user who needed to find a way to make Kinto stop remapping when the screen loses focus. Because Kinto has a much less dynamic nature, that simply wasn't possible, but I was able to implement a workable solution in Toshy. I get the impression that the company maintaining Synergy is not interested in doing something more usable like implementing an IPC mechanism, to make a use case like this easier to deal with.

Again though, sqlite is not required for that to work.

The database file for now just contains "preferences" like which optional features (e.g., "Media Arrows Fix") you have enabled or disabled in the GUI app or the tray icon menus, and the state of the "Dark Mode" switch in the GUI app. Recently, I also added a setting that stores your preference for whether to auto-load the tray icon at startup, which can be toggled from the tray icon menu. And, I've been thinking of eventually storing custom keyboard type info, which is one of the main things users may have to tweak. Right now they have to edit a custom Python dictionary inside the config file to get some keyboards to be treated as the correct "type" and have their modifiers moved to the right spots. (Mostly non-Apple keyboards that are designed for macOS or have switchable Mac/Windows modes, where I can't automatically identify them as "Apple" type keyboards.)

The Synergy fix was easy to add because I was already using watchdog to observe the database file for changes. So I just figured out how to add a second observer for the Synergy log file. And the other part was having matchProps return False whenever the screen loses focus, so all modmaps and keymaps disable themselves. The database file itself has nothing to do with the Synergy solution besides the fact that I'm using watchdog to observe both files for changes.

As with any runtime executable, we need to explicitly declare its dependencies on other components. If this is a shell script, simply declare which executables you expect to have in the $PATH beyond coreutils. It's trivial for us to make the script work at that point.

I see. The person making the Nix formula mentioned needing to rewrite some scripts. I don't really know the details of which parts of any of the scripts may or may not work on Nix. They don't generally do much besides providing a simple way to do multiple systemctl commands at once, or sourcing the venv and launching the keymapper, or providing a quick way to do a long, filtered journalctl command. Things that should work by just typing them all in the terminal yourself. Maybe it's just the path to the actual files that they will need to update, or replacing $HOME with some $XDG_ reference. 🤷🏽

design pattern of locking/mutexes

I do deal with lock files to keep multiple copies of the "apps" (tray icon and GUI app) from being initiated at the same time. If you run the tray app again it will just terminate the existing tray icon process and load a new one, for instance. But I'm not aware of any benefits that I would reap from switching to a simple text INI file, even if I use a lock strategy to reduce the risk of two processes editing the file at once. Sqlite is designed to be extremely lightweight, fast, and usable for exactly the kind of small "database" that I'm using it for. The sqlite connection takes care of any necessary "locking" during edits, and I don't have to use the awkwardness of configparser (I tried to use it and really didn't like it).

The configparser module is the main thing for this purpose that seems to exist in the standard Python library. There's probably some better sort of Python module available for the task, but for one thing I wanted to stick to having a minimal number of dependencies outside the standard Python library, and as outlined earlier the watchdog observer solution ended up having a secondary use case (Synergy) that I can't imagine how else to solve.

Whatever I might replace any of that with has to be better, while not breaking any existing functionality.

So the idea I attempted to convey above is that, rather than applying this concept of a default set of configuration at the config file level, apply it at the level of the abstract representation of settings in the runtime program. If the user provides an empty configuration file, the program should set all values to their defaults in its abstract configuration representation at runtime. Should the config file contain some settings, you'd first apply the default values and then overwrite the settings the user modified with their desired values.

The complexity of the configuration file and the many different remaps that it applies to general and specific apps or app groups make this concept of having "default" settings for things kind of awkward. You should really take a look at the Toshy config file and see what it does.

The way users override things presently is by having a keymap with a matching conditional defined earlier in the config file, in one of the editable "slices". If the same shortcut is defined, it will be used first, negating an identical shortcut defined later in the file. This "layering" effect has been used very strategically in Kinto's config, and it works the same way in Toshy's config.

deftdawg commented 4 months ago

@celesrenata did you get this going?

I downloaded your toshy-shell.nix and overlays/toshy.nix and then did nix-shell toshy-shell.nix.

I see it picked up a python venv from _build, however when I run python toshy_gui.py it errors that it can't find psutil nor pip list show psutil despite the being present at _build/pip_packages/lib/python3.11/site-packages/psutil (which pip is ..._build/pip_packages/bin/pip)... not sure what I'm missing.

celesrenata commented 4 months ago

I took a step back to learn more nixos before trying again. You do have good timing as I am back to looking at figuring this out. It looks like my issue was it failing to wrap gapps properly as GI was still complaining last I looked. I will work to see if I can get this figured out. As Keyd walks all over my touchbar changing the Fn keycode while I know Toshy doesn't. :)

deftdawg commented 4 months ago

it failing to wrap gapps properly as GI was still complaining last I looked

I had some similar errors with GI when I tried to do an overlay for kinto (which I don't ultimately think is feasible because it wants to install and run inside ~/.config/kinto)... 

        buildInputs = with super; [
          python3
          gtk3
          gobject-introspection
          vte
        ];

        propagatedBuildInputs = with super.python3Packages; [
          evdev
          pynput
          pyudev
          xlib
          inotify-simple
          six

          pygobject3
          pillow
        ];

More generally speaking when I hit errors, I tend to feed them to claude.ai or chatgpt to see if they can assist -- I find it generally faster than googling for solutions.

deftdawg commented 1 month ago

In the end I ended up using keyd because it was the most straightforward to configure in NixOS w/KDE, my config is included here