search

qmk / qmk_firmware

Open-source keyboard firmware for Atmel AVR and Arm USB families
https://qmk.fm
GNU General Public License v2.0
18.14k stars 39.05k forks source link

Split Firmware off from Keymaps #1120

Open exiva opened 7 years ago

exiva commented 7 years ago

I think the main qmk firmware and the keymaps should be put into their own repos and have a 'tap' setup sort of like Homebrew has? The current setup makes it extremely difficult to keep up with any changes to the base firmware since new keymaps and fixes to them pollute the commit log. Would also have the benefit that you could only 'tap' the keymaps for the keyboard you have, or could host on your own repo.

Just a thought I've had for a while now.

exiva commented 7 years ago

This might also help make #1038 easier to deal with.

jackhumbert commented 7 years ago

Having them in a branch of qmk_firmware may work, but this is interesting. What would you propose in terms of the tapping? Do you have a script in mind for this, or would it just be a git submodule thing?

Any sort of fancy thing like this would make everything more complicated for new-comers, though. Having everything in one (albeit large) repo does make it a lot easier to get started.

exiva commented 7 years ago

What would you propose in terms of the tapping? Do you have a script in mind for this, or would it just be a git submodule thing?

I'm not sure what would work the best, my instinct leads me to want to write something in python to manage it all, again, similar to homebrew. But maybe just a submodule would be easiest.

Any sort of fancy thing like this would make everything more complicated for new-comers, though.

I think it may be a safe assumption that if someone is looking for a new keymap or to make their own, they probably can handle an extra repo. At minimum though, keep board configs and a basic qwerty layout for each board in the main firmware repo, then put community keymaps in their own.

...Or, you could leave the main supported qmk boards in the firmware repo, and move 3rd party to external, etc. There's a few ways for something like this to go down, what's best though, idk.

PureSpider commented 7 years ago

My suggestion would be to include the main firmware as a submodule to the keymap repo. That way users can just checkout the keymap repo and don't need to worry about anything. If advanced users don't want that clutter, they can just use the firmware repo as a standalone or submodule in an own repo and add their own keymaps on top.

fredizzimo commented 7 years ago

Personally I prefer it the way it is now, at least for maintainers. When everything is in the same repository, it's easy to search where features are used, and even to make global refactoring. But if things were to be changed we should at least at the minimum still support building everything with a single make command to make sure everything compiles, and also make sure that we still build everything as a part of the continuous integration server.

The biggest problem with submodules is that in many cases, you have to make two pull requests. First for the actual QMK feature or fix, and then for the keymap that relies on it. You would also actually have to wait for the QMK to be merged so that you can point the kemap submodule to the right path and version. And if there are several people doing all this is parallel we will have many complicated merge problems. At least that's my experience with submodules, but there might be a better way to use them.

As for the changelog, that could indeed be a problem. The best way currently to get an idea what has been changed is probably to check list of merged pull request rather than the actual commit log. It's also possible to filter out all keymaps when you view the GIT log, at least by using the GIT command line.

The optimal solution would be if someone had time to maintain an easily readable list of changes on the wiki. That would not only include new features and bug fixes, but also code breaking changes that would require changes to existing keymaps, including instructions how to fix that.

jprMesh commented 7 years ago

What if all the core firmware development was on a separate branch that has "releases." The releases would get merged into master, which is where all the keymap stuff would be. So if someone wants to look at the firmware changes, they can just search the firmware branch in the commit log. I'm kinda new to large open source projects, so just let me know your thoughts.

jackhumbert commented 7 years ago

Thanks for bringing this up, and this is a good discussion, but we're going to stick with encouraging users to fork the entire repo for now!

piotr-dobrogost commented 7 years ago

Keymaps kept together with firmware really is a mess. I'm watching this repo and I'm really tired of "Added yet another keymap" commits. If you insist on having some keymaps together with firmware I think there should be one "default" keymap per keyboard type just to be able to quickly try it out and the rest should go into separate repo. I think this should be reopened and separate repo for keymaps should be created.

jackhumbert commented 7 years ago

I wouldn't mind having a few keymaps supported by the keyboard maintainer, then having the rest in some sort of contributor keymaps repo. I'd like for the make system to be able to support building keymaps/keyboards outside of the qmk_firmware folder - I think that's the only thing that would need to happen to jump-start this.

jackhumbert commented 7 years ago

There are some valid hesitations to using submodules, so I'd like to talk through some options with this. If we assume some will download/clone both repos, they could get a folder structure like this:

qmk_firmware/
 - keyboards/
    - <keyboard>/
        - keymaps/
    - <keyboard>/
qmk_firmware_contributor_keymaps/
 - <keyboard>/
     - keymaps/
 - <keyboard>/

Where Makefiles from the ck repo would reference ../../../qmk_firmware/. Is there a better way to do this?

algernon commented 7 years ago

I used to be on the "lets have one giant repo with a ton of keymaps" side, because:

But it also has downsides:

Having worked with Kaleidoscope, where there isn't even a repo that collects keymaps, I changed my mind, and now find the "default keymaps only" approach better, for a number of reasons:

We lose a bit on the discoverability side (but a WiKi can go a long way there), and one will need to clone two repos, but... that's not such a big deal, I found, and can easily be scripted and documented.

I would not, however, force qmk_firmware and qmk_firmware_contributor_keymaps into the same parent directory. There are people (myself included) who already maintain their keymap without dragging the whole of QMK with it. With the new system, I'd expect to be able to build my keymap by pointing its Makefile to where the rest of QMK is (it could default to ../qmk_firmware, but should be overrideable). I keep code I didn't write checked out under ~/src/ext/, while my own stuff is under ~/src/personal - I'd like to avoid symlinking stuff all over the place to please a picky build system.

Thus, as far as the build system goes, I'd suggest the following that _ck/<keyboard>/keymaps/<keymap>/Makefile would look for QMK in QMK_ROOT, defaulting to ../../../qmk_firmware, but if the user overrides that in, say config.mk, or the make command-line, it should be honored. This makes it easy to build a separate keymap outside of QMK, and outside of the contributor_keymaps repo too.

nicinabox commented 7 years ago

@algernon really hits the nail on the head.

For a long time I kept my keymaps on a separate branch, merging in upstream master. This has always felt awkward and hacky to me. The discoverability of other keymaps has made it easier to learn from other setups, but there's also a ton of noise from them.

Today I split my keymaps off to their own repo so they can be maintained easier. I keep qmk master up to date and I compile out of my keymaps repo. This way it treats QMK more like a dependency of the keymap.

I'm definitely in favor of this workflow and with some core support this could be a viable process.

jackhumbert commented 7 years ago

With the addition of #1609, it might be worth considering organising the contributor keymap repo with layout keymaps, rather than keyboard ones. There's also the argument that there shouldn't be any keymaps in the layout folders in the core repo, but I could see wanting some examples (maybe even default_<layout>) for each.

exiva commented 7 years ago

keeping a default map for each board in core would make sense.

jackhumbert commented 7 years ago

Each keyboard would have a default keymap, on top of the layout's default_<layout>.

I wrote a little bit about using layouts for this:

The layouts/default/ and layouts/community/ are two examples of layout "repositories" - currently default will contain all of the information concerning the layout, and one default keymap named default_<layout>, for users to use as a reference. community contains all of the community keymaps, with the eventual goal of being split-off into a separate repo for users to clone into layouts/. QMK searches through all folders in layouts/, so it's possible to have multiple reposistories here.

jackhumbert commented 7 years ago

1609 has been merged, so now it's just a matter of abstracting and moving the user keymaps into the layouts/community folder.

It may be worth considering something to restrict layout keymaps to certain keyboards with the KEYBOARD_<keyboard> flags, or even rules.mk settings.

drashna commented 6 years ago

Also, (once it's fully fixed), the version should be incremented for important changes only.

That would mean that docs, keyboards, layouts and user spaces wouldn't increment the version. Anything else would, and that would make it much easier to tell if serious commits have occurred.

Additionally, you can view the history of a folder, such as quantum or tmk_core. https://github.com/qmk/qmk_firmware/commits/master/tmk_core https://github.com/qmk/qmk_firmware/commits/master/quantum

leico commented 6 years ago

I support the way to make the layouts/community folder a different repository.

drashna commented 5 years ago

I've been thinking about this a lot, actually.

One of the issues that has cropped up now, is that there is the "userspace" feature. This makes this entire topic much more problematic.

It means that we'd need two submodules (one for keymaps, and one for userspace), or we'd need to reorganize the layouts and userspace into a single folder: Eg, we'd need to use layouts/default/, layouts/community and layouts/userspace. And then we use the layouts folder entirely for the submodule.

Which isn't a bad idea.

The only downside, is that we'd really need to enable layout support for EVERY keyboard, and have a layout name for every layout.

It would create a lot of work.

In the meanwhile, I've been trying to add Keymap: and Keyboard: to the commits (when squashing), so that it's easier to filter through.

nicinabox commented 5 years ago

I've been running an experiment over the last year keeping my keymaps in their own repo using a custom cli and treating qmk_firmware as a lib only.

It's been awesome. The implementation is a bit hacky due to having to copy keymaps in the lib, instead of including the lib in the keymaps, but it's been a ton easier for me to maintain a growing list of boards.

Prototyping this has made me want qmk to be setup as separate repos:

Splitting them up reduces noise in the main repo. It's a lot easier to focus on features, bugs, and support related to the firmware itself. Keyboards and layouts are still discoverable, and the CLI can use a real programming language, potentially making it much easier to maintain.

jackhumbert commented 5 years ago

I was thinking about this recently, and put this together as a template we can (eventually, on the qmk org) let people fork and customise: https://github.com/jackhumbert/qmk_firmware_embeded

It requires minimal changes to the QMK core (something we should be able to easily merge), and is mostly just cleaning up the paths (see the embedded branch in qmk/qmk_firmware). The Makefile in the root there is dead simple, and I think it would be pretty easy to support the scenario where someone wants to store all of their keymaps for multiple boards in one repo as well.

henrebotha commented 5 years ago

I'm also doing the separate repo + QMK as submodule + custom CLI thing. It's far more usable and useful than trying to deal with having my layout live in a subdir of QMK. I only have one QMK keyboard, so it's not even like that use case is limited to people with multiple layouts.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged properly or other activity occurs. For maintainers: Please label with bug, in progress, on hold, discussion or to do to prevent the issue from being re-flagged.

lowlevl commented 1 year ago

Hey,

I'd really want to have a way to use a separate repo for my keymaps, and that's something I partially achieved there, however I needed to supply #19679 to fix misbehaving Makefiles. I'm now at a roadblock because the qmk CLI that wanting to use the keyboards/ directory from $QMK_HOME for the generate-config-h, generate-keyboard-c, generate-keyboard-h and not the keyboards/ directory from $PWD.

Couldn't we just add a KEYBOARDS_DIR & KEYBOARDS_PATH to the Makefiles and a --keyboard-library-path to the qmk CLI ? I could free some time to do this.

EDIT: typos

piotr-dobrogost commented 11 months ago

How do things stand nowadays? I turned on github's notifications for this repo and I see the churn from adding/changing various keymaps is enormous.

keyboard-magpie commented 11 months ago

See:

https://docs.qmk.fm/#/ChangeLog/20230827?id=notable-changes

and

https://hackmd.io/@tzarc/Bysm5Vnan

piotr-dobrogost commented 10 months ago

@keyboard-magpie Do additions like this https://github.com/qmk/qmk_firmware/pull/22535 still go to this repository after split?

drashna commented 5 days ago

This got missed. KB PRs would still go to the main repo

Also, I think this can probably be closed, now