Framework for keyboard information used GUI keymap editors

jackhumbert commented 6 years ago

There has been quite a lot of work going into GUI keymap editors and compilers recently (@pawnerd), and we'd like to make some changes to help these projects support all of the keyboards that QMK does without much trouble. We hope this will push support for keyboards using QMK back to qmk/qmk_firmware, rather than into a .json file, so it can take advantage of all of the QMK features, and get more support from the community.

The first thing we'd like to start is a standard for keyboard information that isn't necessary for compilation, but would be nice to have for developing keymaps, specifically through GUIs. The proposed list of information this would contain is:

Name
Shortname (used in QMK as the folder name)
Manufacturer
A webpage for more info (qmk.fm/ by default)
Physical layout options for the keyboard/keyswitches (some sort of KLE format?)

Down the road, expanding this format to include other things may be a good idea:

Hardware options available (backlight, RGB, etc)
Open pins available for modding
Teensy/Pro Micro selection
General build settings for DIY boards

I think the general understanding is that each will be a .json file, hosted on qmk.fm (eg qmk.fm/keyboards/.json).

In addition, it may be useful to have a list of all keyboards supported by QMK in a similar, more concise format, for easier access (eg http://qmk.fm/keyboards.json, a WIP), containing:

Name
Shortname

This would be generated from the individual files on each Travis build, and committed as things change.

Pawnerd commented 6 years ago

I am all in for this; I am currently swamped with other deadlines, but we can soon work something out that will work for any QMK keyboard.

In case anyone is wondering to which GUI @jackhumbert was referring: https://www.reddit.com/r/MechanicalKeyboards/comments/6tqiwf/photos_i_made_a_cloud_based_qmk_firmware_profile/ (summary: 'KBFirmware + QMK Flasher + cloud')

That-Canadian commented 6 years ago

I believe this already exists in the base "config.h" in the keyboard's folder. There you are supposed to define "VENDOR_ID", "PRODUCT_ID", "MANUFACTURER", "PRODUCT", "DESCRIPTION". And if there are different revisions, then in the revision folder you define "DEVICE_VER".

So you could pull that out? This information could also be put into the keyboard's README file, which is probably a better option. We could create a standardized header that all keyboards need to use in their README

jackhumbert commented 6 years ago

Yeah, the only thing that's really needed in this format is the physical layout/KLE data - the rest is just a convenience, and shouldn't really be a pain to add since it doesn't change. This will be accessed via an outside service that shouldn't need to scrape through the code.

One issue with using the config.h is that it isn't standardised (some of these things are defined completely in subprojects) - we could enforce a standard for this as well, though. I'm not sure we have any reason not to.

The readmes are being standardised for this now ;) I have a generator/scraper that pulls the first lines as the Names.

That-Canadian commented 6 years ago

That sounds fair, for the physical layout we could provide a KLE link, or a KLE .json? Depends on what you're looking to get out of it TBH.

I honestly think that the READMEs are the best location for the information to be standardized, easy to scrape too. You could easily add the KLE link in there, however if you want a KLE .json then that would obviously need to be a separate file, can't throw that in the README

jackhumbert commented 6 years ago

The KLE format is able to provide a lot of information that we don't really need - there might be some advantages to including some of it (on the GUI side), but I'm open to alternatives.

Yeah, it'd need to be the actual .json - we shouldn't force apps to make additional requests.

I'm thinking that a slimmed-down KLE may be the easiest way to generate these on the maker side (for new boards):

[{a:7},"","","","","","","","","","","",""],
["","","","","","","","","","","",""],
["","","","","","","","","","","",""],
["","","","","",{w:2},"","","","","",""]

There is a slight complication with keyboards supporting multiple layouts, and an even larger one with keyboards supporting different combinations (think of a keyboard that supports 2u & 2x 1u keys all over the keyboard). In either instance, I think defining the layouts like layout_60_ansi.json that match the layouts in #1609 is the way to go - custom layouts (not QMK-wide ones) would be possible as well. There might be a loss of customisability here, but new ones can always be added easily.

layout_<layout>.json could just contain the physical layout mentioned above, and the rest could be scraped from the readme/config file. If we structure it like that, it would save us a bit from redundancy.

jackhumbert commented 6 years ago

Thinking more about this, it might be a good idea to store the layout_<layout>.json files in layouts/<layout>/layout.json, and require all of the keyboards to use layouts in their rules.mk and <keyboard>.h files (in #1609 there's only the option to use the layouts - it isn't required) to be picked-up by the generator.

e: updated #1609 to include the layout.json files mentioned.

skullydazed commented 6 years ago

I've got the start to something like this in the QMK Compiler API now. Check out these URLs:

http://compile.qmk.fm/v1/keyboards

http://compile.qmk.fm/v1/keyboards/all

http://compile.qmk.fm/v1/keyboards/clueboard/60

http://compile.qmk.fm/v1/keyboards/clueboard/17,clueboard/66

http://compile.qmk.fm/v1/keyboards/amj40,amj60

I've been looking at the format that KLE uses and I just don't think it's a good fit for this. We'd take on an awful lot of technical debt to extend that format rather than breaking compatibility. I think the format I have here is pretty self-explanatory, (it's basically KLE with explicit coordinates and dictionaries instead of strings,) and will make it very easy for web and GUI authors to write their own visualizations.

Pawnerd commented 6 years ago

I think a new json format should be no problem at all. Although there already are multiple standards, such as KLE, we will require different features. If we want to add them later, it might cause trouble for people to understand that there are 'multiple KLE standards'. However, most people won't see any json if the gui's are built user friendly.

jackhumbert commented 6 years ago

@skullydazed and I worked through a near-complete format that I think we're just about happy with:

keyboards/<keyboard_folder>/info.json
Can use all of the keys in the JSON.

layouts/default/<layout>/info.json
Only able to use layouts key, all other keys will be ignored

Keyboard name (Clueboard)
Folder name (clueboard/60)
Manufacturer
VENDOR:PRODUCT:DEVICE_VER
URL (https://clueboard.co/parts/clueboard-66-pcb-25 or https://qmk.fm/keyboards/clueboard/66)
Processor
Bootloader
Maintainer - GitHub username of the maintainer, or "qmk" for community maintained boards.
Width - Width of the board in Key Units
Height - Height of the board in Key Units
Layout definitions
- Key: (All coordinates/lengths in Key Units, Degrees)
- X
- Y
- W: Width
- H: Height
- R: Rotation
- RX: Point of Rotation X
- RY: Point of Rotation Y
- KS: Key shape (define polygon vertices by X, Y origin at Top Left of Key)
  - [ {0, 0}, {1, 0}, {1, 1}, {0, 1}, {0, 0} ]
  - Default Shape: [ [0,0], [w,0], [w,h], [0,h], [0,0] ]
- Width: Total width in key units
- Height: Total height in key units
key_count: Total number of keys

All are inheritable/overridable like usual keyboard folders

Keyboard/Subfolder/Layout JSON:

{
  "keyboard_name": "Clueboard 66%",
  "keyboard_folder": "clueboard/66/rev3",
  "manufacturer": "Clueboard",
  "identifier": "FEED:6060:0001",
  "url": "https://clueboard.co/parts/clueboard-66-pcb-25",
  "maintainer": "skullydazed",
  "processor": "atmega32u4",
  "bootloader": "atmel-dfu",
  "width": 16.5,
  "height": 5,
  "layouts": {
    "KEYMAP": {
      "width": 16.5,  # Optional
      "height": 5,    # Optional
      "key_count": 72,
      "layout": [ { "w": 1, "x": 0, "y": 0 }, { "w": 1, "x": 1, "y": 0 }, { "w": 1, "x": 2, "y": 0 }, { "w": 1, "x": 3, "y": 0 }, { "w": 1, "x": 4, "y": 0 }, { "w": 1, "x": 5, "y": 0 }, { "w": 1, "x": 6, "y": 0 }, { "w": 1, "x": 7, "y": 0 }, { "w": 1, "x": 8, "y": 0 }, { "w": 1, "x": 9, "y": 0 }, { "w": 1, "x": 10, "y": 0 }, { "w": 1, "x": 11, "y": 0 }, { "w": 1, "x": 12, "y": 0 }, { "w": 1, "x": 13, "y": 0 }, { "w": 1, "x": 14, "y": 0 }, { "w": 1, "x": 15, "y": 0 }, { "w": 1, "x": 16, "y": 0 }, { "w": 1, "x": 0, "y": 1 }, { "w": 1, "x": 1, "y": 1 }, { "w": 1, "x": 2, "y": 1 }, { "w": 1, "x": 3, "y": 1 }, { "w": 1, "x": 4, "y": 1 }, { "w": 1, "x": 5, "y": 1 }, { "w": 1, "x": 6, "y": 1 }, { "w": 1, "x": 7, "y": 1 }, { "w": 1, "x": 8, "y": 1 }, { "w": 1, "x": 9, "y": 1 }, { "w": 1, "x": 10, "y": 1 }, { "w": 1, "x": 11, "y": 1 }, { "w": 1, "x": 12, "y": 1 }, { "w": 1, "x": 13, "y": 1 }, { "w": 1, "x": 14, "y": 1 }, { "w": 1, "x": 15, "y": 1 }, { "w": 1, "x": 0, "y": 2 }, { "w": 1, "x": 1, "y": 2 }, { "w": 1, "x": 2, "y": 2 }, { "w": 1, "x": 3, "y": 2 }, { "w": 1, "x": 4, "y": 2 }, { "w": 1, "x": 5, "y": 2 }, { "w": 1, "x": 6, "y": 2 }, { "w": 1, "x": 7, "y": 2 }, { "w": 1, "x": 8, "y": 2 }, { "w": 1, "x": 9, "y": 2 }, { "w": 1, "x": 10, "y": 2 }, { "w": 1, "x": 11, "y": 2 }, { "w": 1, "x": 12, "y": 2 }, { "w": 1, "x": 13, "y": 2 }, { "w": 1, "x": 14, "y": 2 }, { "w": 1, "x": 0, "y": 3 }, { "w": 1, "x": 1, "y": 3 }, { "w": 1, "x": 2, "y": 3 }, { "w": 1, "x": 3, "y": 3 }, { "w": 1, "x": 4, "y": 3 }, { "w": 1, "x": 5, "y": 3 }, { "w": 1, "x": 6, "y": 3 }, { "w": 1, "x": 7, "y": 3 }, { "w": 1, "x": 8, "y": 3 }, { "w": 1, "x": 9, "y": 3 }, { "w": 1, "x": 10, "y": 3 }, { "w": 1, "x": 11, "y": 3 }, { "w": 1, "x": 12, "y": 3 }, { "w": 1, "x": 13, "y": 3 }, { "w": 1, "x": 14, "y": 3 }, { "w": 1, "x": 15, "y": 3 }, { "w": 1, "x": 0, "y": 4 }, { "w": 1, "x": 1, "y": 4 }, { "w": 1, "x": 2, "y": 4 }, { "w": 1, "x": 3, "y": 4 }, { "w": 1, "x": 4, "y": 4 }, { "w": 1, "x": 5, "y": 4 }, { "w": 1, "x": 6, "y": 4 }, { "w": 1, "x": 7, "y": 4 }, { "w": 1, "x": 8, "y": 4 }, { "w": 1, "x": 9, "y": 4 }, { "w": 1, "x": 10, "y": 4 }, { "w": 1, "x": 11, "y": 4 }, { "w": 1, "x": 12, "y": 4 } ]
    }
  }
}

API:

{
  "generated_at": "2017-10-30 04:01:53 UTC",
  "keyboards": {
    "clueboard/66/rev3": {
      "keyboard_name": "Clueboard 66%",
      "keyboard_folder": "clueboard/66/rev3",
      "manufacturer": "Clueboard",
      "identifier": "FEED:6060:0001",
      "url": "https://clueboard.co/parts/clueboard-66-pcb-25",
      "maintainer": "skullydazed",
      "processor": "atmega32u4",
      "bootloader": "atmel-dfu",
      "width": 16.5,
      "height": 5,
      "layouts": {
        "KEYMAP": {
          "width": 16.5,  # Optional
          "height": 5,    # Optional
          "key_count": 72,
          "layout": [ { "w": 1, "x": 0, "y": 0 }, { "w": 1, "x": 1, "y": 0 }, { "w": 1, "x": 2, "y": 0 }, { "w": 1, "x": 3, "y": 0 }, { "w": 1, "x": 4, "y": 0 }, { "w": 1, "x": 5, "y": 0 }, { "w": 1, "x": 6, "y": 0 }, { "w": 1, "x": 7, "y": 0 }, { "w": 1, "x": 8, "y": 0 }, { "w": 1, "x": 9, "y": 0 }, { "w": 1, "x": 10, "y": 0 }, { "w": 1, "x": 11, "y": 0 }, { "w": 1, "x": 12, "y": 0 }, { "w": 1, "x": 13, "y": 0 }, { "w": 1, "x": 14, "y": 0 }, { "w": 1, "x": 15, "y": 0 }, { "w": 1, "x": 16, "y": 0 }, { "w": 1, "x": 0, "y": 1 }, { "w": 1, "x": 1, "y": 1 }, { "w": 1, "x": 2, "y": 1 }, { "w": 1, "x": 3, "y": 1 }, { "w": 1, "x": 4, "y": 1 }, { "w": 1, "x": 5, "y": 1 }, { "w": 1, "x": 6, "y": 1 }, { "w": 1, "x": 7, "y": 1 }, { "w": 1, "x": 8, "y": 1 }, { "w": 1, "x": 9, "y": 1 }, { "w": 1, "x": 10, "y": 1 }, { "w": 1, "x": 11, "y": 1 }, { "w": 1, "x": 12, "y": 1 }, { "w": 1, "x": 13, "y": 1 }, { "w": 1, "x": 14, "y": 1 }, { "w": 1, "x": 15, "y": 1 }, { "w": 1, "x": 0, "y": 2 }, { "w": 1, "x": 1, "y": 2 }, { "w": 1, "x": 2, "y": 2 }, { "w": 1, "x": 3, "y": 2 }, { "w": 1, "x": 4, "y": 2 }, { "w": 1, "x": 5, "y": 2 }, { "w": 1, "x": 6, "y": 2 }, { "w": 1, "x": 7, "y": 2 }, { "w": 1, "x": 8, "y": 2 }, { "w": 1, "x": 9, "y": 2 }, { "w": 1, "x": 10, "y": 2 }, { "w": 1, "x": 11, "y": 2 }, { "w": 1, "x": 12, "y": 2 }, { "w": 1, "x": 13, "y": 2 }, { "w": 1, "x": 14, "y": 2 }, { "w": 1, "x": 0, "y": 3 }, { "w": 1, "x": 1, "y": 3 }, { "w": 1, "x": 2, "y": 3 }, { "w": 1, "x": 3, "y": 3 }, { "w": 1, "x": 4, "y": 3 }, { "w": 1, "x": 5, "y": 3 }, { "w": 1, "x": 6, "y": 3 }, { "w": 1, "x": 7, "y": 3 }, { "w": 1, "x": 8, "y": 3 }, { "w": 1, "x": 9, "y": 3 }, { "w": 1, "x": 10, "y": 3 }, { "w": 1, "x": 11, "y": 3 }, { "w": 1, "x": 12, "y": 3 }, { "w": 1, "x": 13, "y": 3 }, { "w": 1, "x": 14, "y": 3 }, { "w": 1, "x": 15, "y": 3 }, { "w": 1, "x": 0, "y": 4 }, { "w": 1, "x": 1, "y": 4 }, { "w": 1, "x": 2, "y": 4 }, { "w": 1, "x": 3, "y": 4 }, { "w": 1, "x": 4, "y": 4 }, { "w": 1, "x": 5, "y": 4 }, { "w": 1, "x": 6, "y": 4 }, { "w": 1, "x": 7, "y": 4 }, { "w": 1, "x": 8, "y": 4 }, { "w": 1, "x": 9, "y": 4 }, { "w": 1, "x": 10, "y": 4 }, { "w": 1, "x": 11, "y": 4 }, { "w": 1, "x": 12, "y": 4 } ]
        }
      }
    }
  }
}

jackhumbert commented 6 years ago

These info.json files can now be accessed on the compiler API, like this: http://compile.qmk.fm/v1/keyboards/planck/rev4

Pawnerd commented 6 years ago

This is looking good! Perhaps there should be an method to define options as well. So, for instance, if a keyboard has back-lighting, it could look like this:

{
  "generated_at": "2017-10-30 04:01:53 UTC",
  "keyboards": {
    "clueboard/66/rev3": {
      "keyboard_name": "Clueboard 66%",
      "keyboard_folder": "clueboard/66/rev3",
      "manufacturer": "Clueboard",
      "identifier": "FEED:6060:0001",
      "url": "https://clueboard.co/parts/clueboard-66-pcb-25",
      "maintainer": "skullydazed",
      "processor": "atmega32u4",
      "bootloader": "atmel-dfu",
      "options": [
            {
                  "name": "backlight",
                  "label": "Turn LEDs on or off",
                  "type": "bool"
            }
      ],
      "width": 16.5,
      "height": 5,
      "layouts": {
        "KEYMAP": {
          "width": 16.5,  # Optional
          "height": 5,    # Optional
          "key_count": 72,
          "layout": [ { "w": 1, "x": 0, "y": 0 }, { "w": 1, "x": 1, "y": 0 }, { "w": 1, "x": 2, "y": 0 }, { "w": 1, "x": 3, "y": 0 }, { "w": 1, "x": 4, "y": 0 }, { "w": 1, "x": 5, "y": 0 }, { "w": 1, "x": 6, "y": 0 }, { "w": 1, "x": 7, "y": 0 }, { "w": 1, "x": 8, "y": 0 }, { "w": 1, "x": 9, "y": 0 }, { "w": 1, "x": 10, "y": 0 }, { "w": 1, "x": 11, "y": 0 }, { "w": 1, "x": 12, "y": 0 }, { "w": 1, "x": 13, "y": 0 }, { "w": 1, "x": 14, "y": 0 }, { "w": 1, "x": 15, "y": 0 }, { "w": 1, "x": 16, "y": 0 }, { "w": 1, "x": 0, "y": 1 }, { "w": 1, "x": 1, "y": 1 }, { "w": 1, "x": 2, "y": 1 }, { "w": 1, "x": 3, "y": 1 }, { "w": 1, "x": 4, "y": 1 }, { "w": 1, "x": 5, "y": 1 }, { "w": 1, "x": 6, "y": 1 }, { "w": 1, "x": 7, "y": 1 }, { "w": 1, "x": 8, "y": 1 }, { "w": 1, "x": 9, "y": 1 }, { "w": 1, "x": 10, "y": 1 }, { "w": 1, "x": 11, "y": 1 }, { "w": 1, "x": 12, "y": 1 }, { "w": 1, "x": 13, "y": 1 }, { "w": 1, "x": 14, "y": 1 }, { "w": 1, "x": 15, "y": 1 }, { "w": 1, "x": 0, "y": 2 }, { "w": 1, "x": 1, "y": 2 }, { "w": 1, "x": 2, "y": 2 }, { "w": 1, "x": 3, "y": 2 }, { "w": 1, "x": 4, "y": 2 }, { "w": 1, "x": 5, "y": 2 }, { "w": 1, "x": 6, "y": 2 }, { "w": 1, "x": 7, "y": 2 }, { "w": 1, "x": 8, "y": 2 }, { "w": 1, "x": 9, "y": 2 }, { "w": 1, "x": 10, "y": 2 }, { "w": 1, "x": 11, "y": 2 }, { "w": 1, "x": 12, "y": 2 }, { "w": 1, "x": 13, "y": 2 }, { "w": 1, "x": 14, "y": 2 }, { "w": 1, "x": 0, "y": 3 }, { "w": 1, "x": 1, "y": 3 }, { "w": 1, "x": 2, "y": 3 }, { "w": 1, "x": 3, "y": 3 }, { "w": 1, "x": 4, "y": 3 }, { "w": 1, "x": 5, "y": 3 }, { "w": 1, "x": 6, "y": 3 }, { "w": 1, "x": 7, "y": 3 }, { "w": 1, "x": 8, "y": 3 }, { "w": 1, "x": 9, "y": 3 }, { "w": 1, "x": 10, "y": 3 }, { "w": 1, "x": 11, "y": 3 }, { "w": 1, "x": 12, "y": 3 }, { "w": 1, "x": 13, "y": 3 }, { "w": 1, "x": 14, "y": 3 }, { "w": 1, "x": 15, "y": 3 }, { "w": 1, "x": 0, "y": 4 }, { "w": 1, "x": 1, "y": 4 }, { "w": 1, "x": 2, "y": 4 }, { "w": 1, "x": 3, "y": 4 }, { "w": 1, "x": 4, "y": 4 }, { "w": 1, "x": 5, "y": 4 }, { "w": 1, "x": 6, "y": 4 }, { "w": 1, "x": 7, "y": 4 }, { "w": 1, "x": 8, "y": 4 }, { "w": 1, "x": 9, "y": 4 }, { "w": 1, "x": 10, "y": 4 }, { "w": 1, "x": 11, "y": 4 }, { "w": 1, "x": 12, "y": 4 } ]
        }
      }
    }
  }
}

The compiler API should then have an option to set the value of definitions to inject something like this in the C-code: #define backlight 1 (where 'backlight' is the name of the option and '1' is the choosen value.

Keyboard specific code in clueboard/66/rev3 can then react on the choosen value of these definitions to turn on/off the backlight.

Probably too far fetched for the early stage in which this tool currently is, but this would definitely increase the value of the QMK compile tool.

jackhumbert commented 6 years ago

Yeah, @skullydazed and I talked about that a bit - there's some reworking of the rules.mk files to be done for that to happen. We'd like to be able to recognise that an option is available for a keyboard, even if it's not enabled at the rules.mk level, and one way to do that would be to assume that everything that's = no in the rules.mk file is available, but not enabled, and conversely, things not listed are not available for that keyboard.

Pawnerd commented 6 years ago

Awesome to hear how you guys are automating most of this stuff.

elliotboney commented 6 years ago

This is awesome! I was just thinking about this today when I was using a GUI for my keyboard layout and was of course losing my key place in keymap arrays.

+1!

Pawnerd commented 6 years ago

https://i.imgur.com/a2eG6bm.png https://i.imgur.com/sgCZjV1.png Working on it ;)

elliotboney commented 6 years ago

@Pawnerd wow, i need that now

kdb424 commented 6 years ago

Is this still being worked on?

Pawnerd commented 6 years ago

@kdb424 yes, it is online in beta at https://app.knops.io :). It currently only works with Windows.

Pawnerd commented 6 years ago

@elliotboney ^

elliotboney commented 6 years ago

@Pawnerd Need any help on the osx side of things? Or any other dev help?

jackhumbert commented 4 years ago

I think a lot of this has been implemented - the docs for this are here: https://docs.qmk.fm/#/api_docs

qmk / qmk_firmware

Framework for keyboard information used GUI keymap editors #1584