Avoiding clobbering user customizations with Kinto updates

[RedBearAK]

This is just something to think about as you move forward with trying to get Kinto packaged and included into distros. I have noticed that when I upgrade to a new version that my kinto.py gets replaced by the new one. I can't think off-hand whether the installer script tries to at least back up the kinto.py file it's about to overwrite, but overwriting it can end up causing the user to lose all the custom additions or changes they made since they first installed some version of Kinto. For some users it could be years between versions, and they may have made a lot of changes.

Obviously this is not ideal.

I wonder if you have a plan already for a way to help the user save their custom lines and integrate them into a new and updated kinto.py file as the file continues to evolve with each version. Besides continuing to manually splice back in my "fixes" every time a new Kinto release comes out, like the xfce4 fix I always had to apply to Linux Mint Cinnamon, I'm not quite sure how to avoid this problem. Especially since a lot of things in kinto.py rely on each other and their relationship in the loading order inside the file.

I have some vague idea about having a few external files that the main kinto.py would "inject" or "import" into itself whenever Kinto is (re)started. They would be placed in strategic locations in different sections of the main kinto.py file. Like terminals, browsers, GUI apps to override something in the "General GUI" block, and so on. For instance the main file may just contain the "general" mapping blocks while the sub-files would contain mapping blocks dedicated to individual apps, and be automatically inserted into the main file at the right spot when it gets processed.

Ideally these injection points would be wrapped by a function that checked for syntax errors before loading the external code and would just bypass the whole sub-file without loading it if there was a problem, but continue loading the rest of the kinto.py file and successfully start up Kinto, just without that particular set of custom mappings until it gets fixed. And present an informative error message to help the user fix the problem.

Any of this making any sense? I just feel like as Kinto becomes something that more and more people might have installed through their package manager, and thus possibly updating itself with every system update, there needs to be an easier way for the user to have custom mappings but also take advantage of new optimizations coming from your version of kinto.py in each new official release.

If you just ask during install if the user wants to keep their version of the file (I've seen some software do this when they notice a customized config file) and then don't copy the new one into place, then they never get the benefits of your progress. But if you overwrite their file, they lose whatever they've fixed or changed manually. There must be a Goldilocks approach somewhere in the middle.

[rbreaves]

I thought about doing some regex type kung fu, but to be honest.. this problem would likely be better solved by rethinking the overall architecture of the design of the config file itself. I am liking the design of yaml in general and the python config file could be designed to dynamically pull in and build out the python configuration I am willing to bet.

Then there could be a very clear dual yaml file configuration going on. the default.yaml file and a user.yaml file. Both could then be consulted, and merged, with the user.yaml config file taking precedence if there are conflicting remaps btwn the two.

[rbreaves]

Proposed future config file layout.

# defaults Mac, Win, IBM, Chrome, Winmac
keyboard-type: Mac
app-groups:
  terminals:
    - alacritty
    - deepin-terminal
    - eterm
    - gnome-terminal
    - guake
    - hyper
    - io.elementary.terminal
    - kinto-gui.py
    - kitty
    - konsole
    - lxterminal
    - mate-terminal
    - qterminal
    - sakura
    - terminator
    - tilda
    - tilix
    - xfce4-terminal
    - xterm
  remotes:
    - org.remmina.Remmina
    - xfreerdp
    - VirtualBox Machine
    - VirtualBox
  browsers:
    - Chromium
    - Chromium-browser
    - Discord
    - Epiphany
    - Firefox
    - Google-chrome
    - microsoft-edge
    - microsoft-edge-dev
  chromes:
    - Chromium
    - Chromium-browser
    - Google-chrome
    - microsoft-edge
    - microsoft-edge-dev
multipurpose-modmap:
  Enter2Cmd: false
  Caps2Esc: false
conditional-modmap-gui:
  IBM:
    - Key.LEFT_ALT: Key.RIGHT_CTRL
    - Key.LEFT_CTRL: Key.LEFT_ALT
    - Key.CAPSLOCK: Key.LEFT_META
    - 'Key.RIGHT_ALT: Key.RIGHT_CTRL,  # IBM - Multi-language (Remove)'
    - 'Key.RIGHT_CTRL: Key.RIGHT_ALT,  # IBM - Multi-language (Remove)'
  Chromebook:
  Win:
  Mac:
    - Key.LEFT_META: Key.RIGHT_CTRL  
    - Key.LEFT_CTRL: Key.LEFT_META   
    - 'Key.RIGHT_META: Key.RIGHT_CTRL, # Mac - Multi-language (Remove)'
    - 'Key.RIGHT_CTRL: Key.RIGHT_META, # Mac - Multi-language (Remove)'
conditional-modmap-terminals:
  IBM:
  Chromebook:
  Win:
  Mac:
    - Key.LEFT_META: Key.RIGHT_CTRL
    - 'Key.RIGHT_META: Key.RIGHT_CTRL, # Mac - Multi-language (- move)'
    - 'Key.RIGHT_CTRL: Key.LEFT_CTRL,  # Mac - Multi-language (Remove)'
define-keymaps:
  intellij:
    filter: ^jetbrains-(?!.*toolbox).*$
    remaps:
      - C-Key_0:M-Key_0
      - Super-Grave:C-Grave
  general-gui:
    blacklist: remotes
    remaps:
      - RC-Space:Alt-F1
      - RC-F3:Super-d
    ignore:
      - M-Tab

User yaml config can largely be left blank with all possible areas listed for the user to fill in.

# defaults Mac, Win, IBM, Chrome, Winmac
keyboard-type: Mac
app-groups:
  terminals:
  remotes:
  browsers:
  chromes:
multipurpose-modmap:
  Enter2Cmd: true
  Caps2Esc: true
conditional-modmap-gui:
  IBM:
  Chromebook:
  Win:
  Mac:
conditional-modmap-terminals:
  IBM:
  Chromebook:
  Win:
  Mac:
define-keymaps:
  intellij:
    filter:
    remaps:
  general-gui:
    blacklist:
    remaps:
    ignore:

Although I am not sure about leaving it blank for the user.. because that means if they want to remove something it will be harder, but I also don't want it to clobber the main config after an update by having everything fully defined.. I guess I can have it where they need to provide a - in front of a word or quoted word to see it be excluded, otherwise the main config will use it.

[rbreaves]

Some things may always be easier to configure in the actual python config - but in general that would be advised against as yaml will be the extracted parts that are generally safe to modify and update without any unexpected issues.

Will also make future updates to the GUI easier from a parsing perspective as well. Edit the yaml, restart the service and no weird regex stuff happening. What will be interesting will be applying this type of config on the Windows side as well.

Could also break things up into 4 yaml files, app-groups, modifiers, general-remaps and the app-specific config file.

Would be nice if a single config file could be used on either the Windows or Linux side, but I don't think that will be possible.

[RedBearAK]

@rbreaves

[southern_drawl] Why, Mr. Reaves, I do declay-ah, you may be givin' me the vay-pahs... [/southern_drawl]

I knew you'd have some kind of idea about this simmering on the stove.

I am relatively comfortable with the current python config file at this point, obviously, but the excessive "programmerish" syntax of the language is probably far more of an impediment to easy user customization than the relative simplicity of YAML. It looks pretty interesting. I worry a little bit that it might feel a little bit more "disconnected" than the python-based file, but I'd have to work with it a bit to see if I run into any mental stumbling blocks while trying to implement something.

That "blank" user config file would need to be fully documented, hopefully, and preferably right there within the file. Other than the two lines with "true", and without comparing back and forth with the main config file, I wouldn't immediately know what that mostly blank file is supposed to be filled with. You would know, but that's because you just wrote it. Editing an already existing structure is always a little easier than trying to figure out what to do in a totally open-ended unstructured "blank template" example. So if you don't want the user to even open the main config and mess with it at all under most circumstances, it needs to be blindingly obvious how to populate the user config file correctly.

I can promise to still be a relatively unskilled person who will faithfully represent an "average" user during the testing phase.

Although, I will say that whenever I make a mistake (missing comma or unrecognized term) in the current python config the xkeysnail error message in the log generally tells me very clearly exactly where that mistake was made. Not just the line number but it usually displays the actual thing that is causing the problem. I hope the troubleshooting of errors in the YAML file(s) would be equally or more straightforward. You know we'll find every possible way to make mistakes. That's what users are for.

[rbreaves]

The other option is to just have the user yaml to be an exact duplicate of the default.yaml file and not have any blanks at all.. When it comes to upgrade time then diff the 2 files, create a patch and then apply the update and overwrite the user.yaml (keeping an old backup, plus the diff patch) and attempt to re-apply the apply back into the current user.yaml updated file.

That might be simpler tbh and if not a normal diff command than run a comparison instead programmatically and extract out the additions that user made to the user.yaml with the appropriate descriptions, whether they be entirely new apps or just new keymaps for existing apps.

[RedBearAK]

The other option is to just have the user yaml to be an exact duplicate of the default.yaml file and not have any blanks at all.. When it comes to upgrade time then diff the 2 files, create a patch and then apply the update and overwrite the user.yaml (keeping an old backup, plus the diff patch) and attempt to re-apply the apply back into the current user.yaml updated file.

That might be simpler tbh and if not a normal diff command than run a comparison instead programmatically and extract out the additions that user made to the user.yaml with the appropriate descriptions, whether they be entirely new apps or just new keymaps for existing apps.

Oh, my. I do like the sound of that. Like a teacher using a template sheet full of holes to make it easy to check the answers on a multiple choice standardized test. Overlaying the user's version on the original. Yeah. Neat. Reminds me of non-destructive image editing. And that would totally make the user config file self-documenting, basically. Lots of complete examples to build on and learn from.

[rbreaves]

Tbh I just need to focus on getting all the PR backlog cleared up so I can start in on the changes. Also need to review what my roadmap had in it & up date it to reflect what we’ve been discussing.

[rbreaves]

So I am thinking that there will likely be 3 levels of upgrades, minor, semi-major, and major essentially.

Minor - If a default.yaml file is updated then that is the most minor update as it simply adds, removes or changes some normal and fully expected type of remaps. That update process would entail checking the current default.yaml in master any time the app runs if the network is available and apply differences. The user.yaml would then be rebuilt with any customizations intact, if there is a conflict then a conflict file will be created which the user can then resolve in the same manner as a merge conflict in git probably. Conflicts will not prevent the program from running but if the user wants the latest updates then they will need to fix the user.yaml.conflict file and save over their original (which will have a backup as well with date in the name).

The system tray and main app could also list any conflict file that needs to be opened and resolved as well. If the app detects that the user.yaml has been updated and successfully compiles and kinto runs ok then maybe I can have auto-delete the conflict file or simply delay a prompt to ask the user if they'd like to delete the conflict file. I feel like I should also have a user.yaml.diff file that would only have their changes, so worse case scenario the user will know exactly what to bring in if they end up deleting their user.yaml file - which will trigger Kinto on a restart to reconfigure your keyboard layout and then.. well I guess the setup could attempt to merge your diff in as a patch instead and that should work, if not then the diff will have to be ignored I guess and users will have to do it manually.

So at max you are looking at these 4 files.

• user.yaml
• user.yaml.2021-04-14-1230.bak
• user.yaml.diff
• user.yaml.conflict

Semi-major - The kinto.py file has had a new addition or change in its logic or flow. This may also result in bigger changes in the yaml files, but probably not. A check should be added though to still see if the user has modified this file - a backup can be made btwn upgrades if so, but no attempt to re-merge it will be made.

Major - Xkeysnail has had an update and basically the entire app will be reinstalled. All other checks and backups will apply here as well. Right now xkeysnail pulls from my fork, but this may change eventually once I submit back some changes to be merged upstream again. I may also not change this, because in theory an untested update to Xkeysnail's master could break Kinto unless I set it on specific releases/commit.

---

In all 3 cases attempts will be made to save the user.yaml file before upgrades and re-applying any custom changes. No attempt will be made to save customizations a user might have made to kinto.py. I just can't easily or reliably account for entirely new python written blocks of code compared to a yaml method. Reason for that is because I can actually write code to build out the same python code from a yaml file and likely in a couple of ways, and that would be reproducible - but if users start writing code all over the place in the python file itself I am concerned that a merge conflict would eventually occur and create more headaches for users than to just use a yaml file. JSON would work too, but I think that'd be too verbose.

Also I think it might make more sense for users to look at the yaml config file because I could very consistently reference the physical keys that we will be remapping instead of remapping an already remapped modifier key. I can actually programmatically account for that so that users no longer have to carry all of that mental baggage themselves. I can even call the Super/Win key Cmd for Apple/Mac keyboard users in yaml.

If a user feels compelled to add additional functionality to the kinto.py file then I hope they will decide to fork my project and do PRs if it is applicable to other users. If that file changes outside of the original default I can do backups of it as well during an upgrade - but not attempt will be made to put things back as they were unless it is written in yaml.

---

This should probably happen in a number of steps, so I might just focus on switching over to yaml first and getting things building that way and push that release out before having all of the upgrade logic components fully worked out and then the following release work out the upgrades and the release after that add in a GUI based updater.

So at minimum it will be done over the course of 3 targeted releases I think. There may be other releases during this time as well, but this is what I am thinking at the moment.

[RedBearAK]

Even if you are quite skilled (I have no reason to doubt it) this sounds like something that is very ambitious and could take quite a while to implement. I'd like to see a "Stage Zero" as part of this, implemented in current installer. Shouldn't be difficult:

• Make sure installer makes a dated backup of the existing kinto.py. (I'm not aware of this already happening.)
• [Optional] Add a screen to the installer notifying user that kinto.py was replaced and their copy was backed up to the dated file.

This alone should make it possible for most users that that may have already been messing around with their config file in its current form to A) find the new backup file and potentially B) splice their changes back into the "new and improved" kinto.py file. Like I had to do until the Finder mods were merged.

Once I finally learned to use the right tool to do a side-by-side locked-together highlighted diff (VSCodium, if I recall), it wasn't too difficult even for me, even though I was still struggling to fully comprehend the entire file. That's actually the most tricky part, whether we're talking about kinto.py or a YAML file conflict that can't be automatically merged. Just using the right tool to make the task understandable.

I am also wondering if the suggested YAML format is stable and orderly enough that a GUI similar to the existing keyboard shortcut settings apps in most DEs could be constructed to let the user add/remove applications and tweak/add shortcuts in the "desktop" sort of way that many users are searching for, rather than manually editing the YAML files.

and the release after that add in a GUI based updater.

I think you're referring here to a GUI to help the user merge their custom changes back into a new release? So separate from what I just said about a possible future "Kinto Keyboard Shortcuts" app.

Also I think it might make more sense for users to look at the yaml config file because I could very consistently reference the physical keys that we will be remapping instead of remapping an already remapped modifier key. I can actually programmatically account for that so that users no longer have to carry all of that mental baggage themselves. I can even call the Super/Win key Cmd for Apple/Mac keyboard users in yaml.

One the one hand that sounds nice. On the other hand, the physical keys with remapped logical key codes will always have a problem: They will never match the keyboard shortcuts that are displayed in every DE's keyboard/shortcut settings app and every "Keyboard Shortcuts" panel in every application that has existing keyboard shortcuts, whether they are editable or not.

I feel like trying to make it "less confusing" within the config files by only using the physical key names would actually cause a lot more confusion outside the config file when trying to add a new shortcut or figure out why something isn't working. The fact that the modifier keys are remapped can't really be ignored. Nothing Kinto does makes much sense until the user understands that main concept. That the "Alt" key is now "Ctrl", etc.

If there is ever a GUI shortcut editor app for Kinto, the modifier remapping should be emphasized rather than hidden, with the GUI showing a sort of grid view of the editable "real" key codes that are being intercepted and remapped to what an application expects to see, and a non-editable automatic column showing what physical keys that corresponds to based on the modifier key remapping that is applied to the specific application (i.e., terminal vs GUI modifiers). Or maybe both editable, but each will automatically change the related item in the other column (physical vs logical). It would be like an instant-feedback learning tool for the user, to produce that, "Oh, I see!" moment as quickly as possible.

I hope that makes some sort of sense. In the end, Kinto is a keyboard remapper, and the sooner the user understands that, the better it should turn out when they try to add their own new shortcuts or fix something that isn't working.

Maybe a really simple little GUI can be constructed already that would just let the user enter a shortcut they are trying to implement and have it be automatically translated between physical and logical keys.

[rbreaves]

When I referenced a GUI for the updating I just meant a prompt that tells you an update is available and if you would like to update. It would do a practically everything I said in the background. If it succeeds then you get no error message at all, if not then it falls back to either your old config or goes with a new one based on the defaults. Nothing would really be lost however.

A GUI interface to work with adding, removing and editing hotkeys is a thought as well, but that certainly will not be in the 3 stage release and yes I am not referencing the time line for the installer that needs to be refactored as well. I am unfortunately having to travel and take care of multiple things things this week so I can't give any ETA on any of this, just whenever I am able to work on it pretty much and I still have some items on the Kairos project I am trying to finish up as well, and some of that work (or concepts) will go into Kinto's future installer as well.

[rbreaves]

I deleted my last comment.. just trying to think through what the remaps should look like when/if trying to reference the Cmd/Win or Alt key on varying keyboard types in yaml. It is a tricky thing to be honest..

Could just say L_Mod1, L_Mod2, L_Mod3 and R_Mod1, R_Mod2, R_Mod3 read left to right.. although on laptops there's usually only 2 out of the 3 of those modifier keys on the right but I guess ultimately the locations don't really change so the naming scheme could actually remain the same either way.

[RedBearAK]

When I referenced a GUI for the updating I just meant a prompt that tells you an update is available and if you would like to update. It would do a practically everything I said in the background. If it succeeds then you get no error message at all, if not then it falls back to either your old config or goes with a new one based on the defaults. Nothing would really be lost however.

That's what I thought. And it sounds fabulous.

A GUI interface to work with adding, removing and editing hotkeys is a thought as well, but that certainly will not be in the 3 stage release and yes I am not referencing the time line for the installer that needs to be refactored as well.

Yes, the theoretical full GUI settings editor would be much further down the line.

So I guess what you are wanting/intending to do is try to fully isolate the user from having to know (or care) about the modifier remapping. And reprocess the config file key names based on which keyboard is selected? Seems like that is putting a lot of work on your end. But it would theoretically result in something that wouldn't confuse the hell out of the end user. So... go for it, if you think it will be doable. That's a heck of a plan.

Ha, you deleted it. Having second thoughts about the complexity, I see. LOL. That's very rational.

But it's still a nice idea, if it's possible. If the modifier mapping was somehow completely invisible to the end user... Man, that would be neat.

What will be odd is that if a user changes their keyboard type to mac, win, ibm, or chrome manually then they might have to run a script to have the yaml file convert itself to be fully mac or win as far as key references throughout the file and that is completely doable. I can easily run a check to see if the file needs updating to align with what would be expected of that keyboard type or not. May even still be a regex/perl thing to do it, not sure.

What about auto-generating one file each for all keyboard types and hiding them in a sub-folder, then just swapping the main "user" file out whenever the keyboard changes. They would still need to be re-generated whenever the user file is modified, but then it becomes a simple file-swap operation to show the user the appropriate key names when they open the active user file. If the user file hasn't been modified versus the matching keyboard type in the sub-folder (most of the time it won't be), no need for any complex processing to take place in the background just because the user is switching keyboards all the time. The full reprocessing is only triggered when the file is modified.

user.yaml
user.yaml.d
    user.yaml-apple
    user.yaml-chromebook
    user.yaml-ibm
    user.yaml-windows
    user.yaml-winmac

[rbreaves]

Winmac is the same as Win - difference is in the driver config only. I think I still want to rename modifiers but use Left or Right prefixed to primary, secondary & tiertary.

[rbreaves]

More I think about.. it’d probably be a bad idea to rename the modifier keys in yaml vs maybe adding an ASCII diagram of the base modifiers for the two sections in comments. That’d probably be more helpful tbh.

And I’m not going to split configs based on keyboard types, that’d just introduce too much potential for configs to diverge imo or incur technical debt or people submitting incomplete PRs imo. Rather work out how to fully define those things in yaml or in the conversion process, when it can be.

[rbreaves]

Finally getting around to re-reviewing this. I think I will plan to make a new kinto.py config file that will simply parse yaml files in a config.d folder in real time. Will also design it so that each existing section/method/function will call for a filename with -custom.yaml on the end as well so that people can add their own changes.

I guess I should also allow for the ability to over-ride a default - cancel it out as well via the custom.yaml file, so really the default plus the custom needs to be fully parsed before being interpreted.

Mainly restructuring so that I can get rid of all the perl regex stuff I had been doing. I thought it was a smart idea at the time as I could keep everything together better - but really after working it all out had I split it up better instead so that users can modify things and have custom configs from the start that would have been smarter.

Now I will have to first fix the config file to flow better and then go into the kinto GUI installer, modify it and make it so I can install the app via CLI so that automated testing via CI can happen. Initially I will setup a staging branch that will test all changes against Ubuntu Budgie, but will keep it relatively simple - just install it and make sure the app continues to run. Later on I will add additional tests, unit tests, etc.

[RedBearAK]

@rbreaves

Are you sure you want to prioritize the refactoring of the config file over the transition from running Kinto as root to running as user? I feel like the issues with maintaining user changes to the config file are less of a problem than the issues caused by needing to run as root. The more knowledgeable users can always run Kinto manually as user, but then we lose the compatibility with the tray icon and GUI app, until those apps and the installer all get rewritten to support Kinto running as user.

I started this issue, so you know I'd be more than happy to see a more robust custom user config situation. But I think converting Kinto to run as user would provide a more solid base for moving ahead on rejiggering the user config setup.

[rbreaves]

Root vs user doesn't solve my automated CI Pipeline for testing problem.

Both are near equal priorities to me. The reason why I am thinking more of this refactor, for the config file, over root vs user, is that I want to get this project working with a CI Pipeline runner that can test the code any time something is merged into staging or the dev branch.

Root vs user does not prevent a routine testing to automatically occur, having a GUI only install sorta does, and I have no plans to bring back a CLI install option without refactoring the config file & I do want to bring back CLI installing for easier CI testing. It really solves 2 problems at once that makes this project more maintainable even if the root vs user still needs resolving.

The plan is to split up the defaults into multiple yaml files plus blank or templated yaml files for customizations in each category so that it is clear to users that you can modify this file safely - but not the others.

Example - read only defaults ~/.config/kinto/config.d/ conditional-modmap-gui.yaml conditional-modmap-terminal.yaml group-terminals.yaml app-general-browsers.yaml

~/.config/kinto/custom.config.d/ conditional-modmap-gui.yaml conditional-modmap-terminal.yaml group-terminals.yaml app-general-browsers.yaml

or I could place ~/.config/kinto/config.d/ in /opt/kinto/config.d even.

[rbreaves]

Also note the CI Pipeline won't be super extensive initially, it'll just make sure the app installs successfully and has no errors initially & I will slowly add in some unit type tests or fake input tests that'll ensure specific combos and edits occur as expected in text editor or certain programs. If any of those tests fail then I will know that a change has broke an existing test and deeper inspection needs to occur before merging into master.

I may have to add in a raspberry pi that initiates some of the input tests but I am not sure yet. It'd also be possible to add multiple distros for the CI pipeline as well. Ultimately it would just free me up to not being as concerned on needing to launch and test changes across multiple distros if the CI pipelines can just verify that nothing I care about breaks. I can't and won't be able to test everything, but I can at least ensure a higher level of quality with every release.

[RedBearAK]

@rbreaves

Both are near equal priorities to me. The reason why I am thinking more of this refactor, for the config file, over root vs user, is that I want to get this project working with a CI Pipeline runner that can test the code any time something is merged into staging or the dev branch.

I get what you're going for. If it works out I'm sure root-vs-user will be a relatively easy transition later.