Feature Request: Allow re-parsing without overwriting customizations

[infused-kim]

Sorry, but I have another feature request / suggestions... :)

My use-case is that I want to be able to edit my ZMK keymap and then generate a pretty representation of it without having to manually keep the ZMK keymap and graphics in sync.

Currently that's not possible, because reparsing overwrites whatever changes you have made in the yaml keymap.

My current workaround is to use a python script that makes my adjustments inside the keymap yaml. I imagine that I am not the only one who wants this, so perhaps that is a feature you could consider for the future.

And in the meantime, other users can use my solution as a workaround, which can be found here:

https://github.com/infused-kim/zmk-config/tree/chocofi/main/keymap_img

[caksoylar]

No need to be sorry, thank you for opening these issues! I am happy that it is being useful enough to someone to care about improvements :)

I can see this being not too hard if you only care about preserving non-default fields (like align for combos, or type for keys). In this case it might be possible to parse an existing YAML (keymap parse --update keymap.yaml -z corne.keymap), create a KeymapDrawer object, then update that Pydantic model with the newly parsed one (like we do for draw_config's if one is specified inside the yaml).

If you wanted to be able to remove field values or disallow changing existing fields with new values that would be less trivial and would require some decisions on how to handle certain cases.

I will be out for the weekend but I could have a look at this later, or I'd welcome any contributions if you are so inclined!

[caksoylar]

In general, being able to fully automate end-to-end would be very nice, although I am not sure how achievable it is in a practical sense. This feature would definitely help towards that goal. Also see this TODO item: https://github.com/caksoylar/keymap-drawer/blob/main/TODO.md#internal

I would say current approach for parsing is at a best effort basis where you get a "good enough" starting point to be manually edited, but I did try to give tools to the user with the configuration that you are taking advantage of. There's definitely a lot more to do on the parsing end 😄

https://github.com/jbarr21/keymap-display/blob/main/scripts/json2yaml is also related since they also need to do some post-processing parsed keymaps (although it is focused on QMK) due to the current limitations.

[infused-kim]

The parsing is actually really great. I have a fairly complex zmk config and everything is working. I'm super impressed.

And while total automation probably won't be possible (or be a lot of work), I think what I have now is a very reasonable workflow. The main thing I want to avoid is having to manually sync two definitions of the keymap. That is a lot of effort that I am not willing to do and the main reason I haven't been using keyboard-layout-editor.com.

But that's kind of the default workflow of your tool right now. You use the zmk config as a starting point, then clarify and customize the generated yaml and then if something changes in the keymap, you have to either manually adjust it in the yaml keymap or wipe your customizations and do them by hand again.

So my suggestion would be to find a way to do those customizations without manually editing the generated keymap file. Ideally all settings should be done in the config file and the tool generates the yaml file solely based on the config and the zmk/qmk config.

I can't imagine there are many people who would want to use this tool without a qmk or zmk config. There probably are some, but there are plenty of tools that serve their needs and don't require them to edit config files by hand, such as keyboard-layout-editor.com.

But there aren't any good tools to generate keymap images from zmk/qmk configs.

So my suggestion would be make that use cases the primary focus of your tool.

And I think that's not too difficult to do. There are really not that many things that users would want to customize:

• Key labels (Already done)
• Custom behavior labels (Already done)
• Combos (Could be done by using the zmk combo name in the config)
• Exclude layers
• Button highlights

That's basically it, I think. With those, I think it would be a VERY popular tool for most zmk users.

[infused-kim]

Another idea: it would be even better if you could define behavior key labels directly in ZMK.

This way there is only one config to maintain and it’s easier to remember to do it while you are adding stuff.

[caksoylar]

But that's kind of the default workflow of your tool right now. You use the zmk config as a starting point, then clarify and customize the generated yaml and then if something changes in the keymap, you have to either manually adjust it in the yaml keymap or wipe your customizations and do them by hand again.

So my suggestion would be to find a way to do those customizations without manually editing the generated keymap file. Ideally all settings should be done in the config file and the tool generates the yaml file solely based on the config and the zmk/qmk config.

Yes, I envisioned the most common use would be "parse once, update infrequently" but I do want to support frequent syncing with the keymap which was indeed the reason for customization with the config file. You are also right that parse-to-draw workflow is a big advantage of this tool compared to e.g. KLE; I appreciate you providing your thoughts!

Of course a solution to this issue (updating existing yamls) should go a long way to ease the pain points, even if parsing isn't perfect.

To touch on specific points:

Combos (Could be done by using the zmk combo name in the config)

Like I also mentioned in #8 this is pretty hard but there is room for improvement on current behavior. If you have any thoughts on how the configuration can be utilized for this I'd appreciate it. Getting an alignment hint from the ZMK combo name would be a pretty easy addition using the current implementation of the parser.

Exclude layers

If I understand you right this is possible with keymap draw --select-layers although it is selecting, not excluding.

Button highlights

I am planning to add this by remembering where momentary layer keys are, hopefully that should be soon. Supporting conditional layers can come later.

Another idea: it would be even better if you could define behavior key labels directly in ZMK.

That's certainly another way to do it, maybe with a special comment syntax. Right now the parser is relatively simple, it runs the C preprocessor (which removes comments), does nested brackets parsing, finds some nodes with special names keymap/behaviors/combos and parses inside them with regexes -- utilizing comments would require rethinking some of that but long term I plan to increase the robustness of parsing anyway.

[infused-kim]

I envisioned the most common use would be "parse once, update infrequently"

I think you are perhaps looking at it from a seasoned user's eyes. You probably have been using programmable keyboards for a long time and already have a very dialed in keymap that doesn't change very often.

But for that type of user your tool isn't very important anyways, because they likely already have the keymap memorized well. The user for whom your tool is most useful is someone who just started using programmable keyboards or someone who just switched to a new keyboard design and is in the process of figuring out what keymap they want. Those users are going to be tweaking the keymap a lot and would like to have an image to reference it.

Combos Like I also mentioned in https://github.com/caksoylar/keymap-drawer/issues/8 this is pretty hard but there is room for improvement on current behavior. [...] However this is hard to do during parsing since we don't know about the physical layout at this point.[...]

I think this could be solved by adding the layout info during the parsing step. In fact, if I were you, I would remove the entire yaml keymap and 2 step parse+draw workflow.

Keep in mind that, (imho) nobody would ever actually WANT to edit a keyboard layout in yaml files. It's only a last resort due to no other alternatives. ZMK and QMK users would prefer to maintain a small additional config file that tweaks the keymap image a little, but takes most of the info from their keyboard config.

And people who have more complex needs or don't use zmk and qmk have other tools that fulfil those needs.

So, my suggestion is to just make everything configurable in the config and only use the yaml keymap internally in the app.

Then it would be easy to integrate the zmk combos. The user could just use the name of the combo in the config like:

  raw_binding_map:
    '&mm_grescm': {t: 'ESC', s: '` ~'}
  zmk_combos:
    combo_tog_nav_word:
      alignment: bottom
      offset: 0.2
  zmk_keycode_map:
    AMPERSAND: '&'
    AMPS: '&'

[manuelherzog]

I support your idea of full automation and have already an idea i would like to try to contribute an solution i thought of already, and only need the time the next days/weeks to build it for my workflow. My steps would be

• add the config to exclude layers
• add the possibility to extend the generated yaml keymap, for combos it could be done that way, exclude layers list would be a no-brainer, pressed buttons would be either a layer -> keynumber map or perhaps for lt/mo and stuff it could be autoparsed
• add some easier possibility to reflect not only the US Layout, but have some presets for the local keymaps, you could sometimes be bound to. e.g. German, z and y swapped, other symbols on shifted Nums with same keycode, additional alphas ÄÖÜß, stuff like that ;)
• add a custom github action with config for keymaps, could be **.keymap as default, as in my opinion most people in need of autogenerated pics only have one keymap yet. It could be added in the same way as the zmk build action.
• add a second action that can be added to combine the keymaps and the zmk firmware in one zip for easier download

Does it make sense for you? @caksoylar is it in a way you would like to have it? or do you have already other plans?

I won't have the time the next days, but my brain is already working on it. So as it seems it could be done easily, as far as i understood the code right now, and this is one of my plans of a complete toolchain for myself, from metalanguage to zmk&qmk for multiple keyboards to autogenerated keymap pictures for all of them. I would be happy to contribute with some useful parts to the community.

[caksoylar]

I think there is a lot of benefit to preserving modularity between parsing/drawing: I know at least a few users who use and appreciate keymap drawer as a pure drawing tool that they can use programmatically and without going through a web app (like KLE), and preserve the source for their drawing in a user-editable data format (keymap YAML). Think of all the entries in https://keymapdb.com: Owners of all the keymaps there had to create these representations to share with others even though they themselves are familiar with them.

I think the downsides of this approach is very few as well (one of which is parsing not knowing the physical layout like mentioned): By adding some YAML dumping and loading code you get this intermediate representation for basically free. If you don't care about it at all, you can even pipe parse to draw like

keymap -c my_config.yaml parse -z .... | keymap -c my_config.yaml draw - >keymap.svg

I want to stress that I appreciate your thoughts and suggestions @infused-kim and I agree with the benefits of improving the end-to-end parse/draw workflow!

Then it would be easy to integrate the zmk combos. The user could just use the name of the combo in the config like:

This is a really good idea, it would be good to add support for such a zmk_combos field to the configuration.

[caksoylar]

Thanks for your thoughts as well @manuelherzog, below are a few notes:

add the config to exclude layers

Is there a significant benefit to enumerating layers to exclude rather than to include as in keymap draw --select-layers ...?

add the possibility to extend the generated yaml keymap, for combos it could be done that way, exclude layers list would be a no-brainer, pressed buttons would be either a layer -> keynumber map or perhaps for lt/mo and stuff it could be autoparsed

Like I mentioned in https://github.com/caksoylar/keymap-drawer/issues/7#issuecomment-1452740196, I would like to add a parsing feature to determine held keys which should be possible relatively easily.

add some easier possibility to reflect not only the US Layout, but have some presets for the local keymaps, you could sometimes be bound to. e.g. German, z and y swapped, other symbols on shifted Nums with same keycode, additional alphas ÄÖÜß, stuff like that ;)

I have been thinking about the best way to support international layouts but I haven't come up with a maintainable way to do that yet. We definitely don't want to maintain our own variant of keymap_extras; maybe we can somehow utilize ZMK locale generator? In any case, I guess this would be implemented as a modular include for zmk_keycode_map.

add a custom github action with config for keymaps, could be **.keymap as default, as in my opinion most people in need of autogenerated pics only have one keymap yet. It could be added in the same way as the zmk build action. add a second action that can be added to combine the keymaps and the zmk firmware in one zip for easier download

I dabbled with a test workflow already here, if you want to contribute an example workflow or a more complete workflow that people can point to with a uses: declaration I'd be happy to house it in the repo.

If you want to discuss each item further, feel free to open separate issues.

[infused-kim]

I want to stress that I appreciate your thoughts and suggestions @infused-kim and I agree with the benefits of improving the end-to-end parse/draw workflow!

You are welcome and thank you for considering them. I look forward to all your improvements. :)

[caksoylar]

@infused-kim Do you mind giving #13 a try? I explain the inheritance logic in the PR, I thought this would cover the most useful re-parse use cases. Please comment if you think of improvements to the logic from your use case.