Removal of projects or separation of core library and project

[tmk]

I have plan to remove some of projects which I can't mainatain from repository, or move projects to another(each own) repository.

1) Most of converter projects will continue to be updated by hasu/tmk(me) because I still own keyboards needed to test firmware. Meanwhile most of keyboard projects can't be tested. hasu/tmk owns keyboard pojejcts; gh60, hbkb, hhkb, infinity, onekey and onekey_mbed. But they may be separated from core library.

2) If contributor of the project has a repository it should be center of the project. README of tmk_keyboard just has a link to the repository and tmk_keyboard has no code for the project. Place your repository link on this wiki page: https://github.com/tmk/tmk_keyboard/wiki/TMK-Based-Projects

Update: 2016/01 Moved some projects to orphan directory now. 649151c

[xauser]

I can offer to test the following keyboard projects: kitten_paw, kmac, lightpad, lightsaber, nerd and phantom.

[tmk]

Thanks. Probably I'll contact you when I make decision about this.

[p3lim]

What about turning it the other way around?

Currently the keyboards and converters are stored in directories within the firmware, but what about having each keyboard/converter it's own project (repository) and have tmk_keyboard be a submodule within those.

It will require some restructuring for sure, but it's one way of cleaning all of this up.

[p3lim]

I would also like to suggest that you make an organization instead of having all of this on an account.

This allows easier contribution and the possibility for teams and hosting keyboard/converter projects all in one public space instead of having everything scattered across different accounts/users (although still being possible for those who want)

[BenBergman]

An organization with a main repo for the core code and separate repos that reference the core (perhaps through subtrees or submodules) would be a great way to organize all the related projects. I think you can grant different permissions to different repos in an org, so you can give people access to maintain their own keyboard projects without letting them touch the core, if you wanted to do something like that.

[p3lim]

I think you can grant different permissions to different repos in an org, so you can give people access to maintain their own keyboard projects without letting them touch the core

That is correct, and exactly what I suggested :smile:

[xauser]

I wouldn't split up the tmk project in any kind of way. Having all drivers in one project is very convenient for me. Of course, no single person can test all incoming patches alone because of the required hardware, that's why I would assign a maintainer to each keyboard project/subtree. Patches to those projects/subtrees would go from maintainer to tmk project only.

[kejadlen]

A separate org with one main project with submodules for individual drivers makes sense to me.

[nemith]

A single repository can be beneficial as well. For example code refactoring means a single commit to touch every consumer of a library call vs a commit for each sub repository.

[p3lim]

A single repository can be beneficial as well. For example code refactoring means a single commit to touch every consumer of a library call vs a commit for each sub repository.

Which means one person (hasu) would essentially have to either update all of the drivers himself or validate and merge changes from others, equal amount of work.
If we have a main project treated as a submodule it won't be updated unless the repository owner/maintainer of the driver does so, which also means that nothing will be broken if the main project changes, and leaves the owner/maintainer to update the driver if necessary.

I wouldn't split up the tmk project in any kind of way. Having all drivers in one project is very convenient for me. Of course, no single person can test all incoming patches alone because of the required hardware, that's why I would assign a maintainer to each keyboard project/subtree. Patches to those projects/subtrees would go from maintainer to tmk project only.

Having one test driver in the main project could be used for both testing and maintaining what would be considered "the basic implementation" that drivers could refer to for changes. This test driver could also be heavily commented for this exact reason.
Which driver that would represent would be up to hasu, I'd guess the GH60 driver.

[BenBergman]

@p3lim ++

[p3lim]

@tmk If you end up going with the organization path and would want to use "tmk" for the name but still keep all your commits correct, I'd suggest first renaming your account to "hasu" (or anything for that matter, something that would be more personal), then create a new org with the name "tmk", finally transfering the repositories and restructuring everything in the org. This way you would still be the correct owner/commiter of all the commits, GitHub manages everything on their end.

If you need help with any of this, let me know, I've already experienced this myself.

[xauser]

I still don't like the idea of having submodules. You split up code that belongs together. You can always create integrity by branches, tags and test suites.

If you refactor core code you have to deal with all your dependencies, that's life. And if you can't test a patch yourself because of time or hardware, let others do it for you.

[p3lim]

You split up code that belongs together. You can always create integrity by branches, tags and test suites.

Sure, it belongs together, but in reality the core is treated like a library, and having the actual projects (the drivers) as subdirectories within is (in my opinion at least, seems like hasu would agree considering this issue exists) cluttered and.. weird.

[BenBergman]

I still don't like the idea of having submodules. You split up code that belongs together. You can always create integrity by branches, tags and test suites.

Tags would help a lot for identifying working commits for specific projects, but keeping all the projects in one repository kind of implies that the main branch will be maintaining all projects. Maintenance branches can work, but that is very nearly the same as having a separate repo for each project.

If you refactor core code you have to deal with all your dependencies, that's life. And if you can't test a patch yourself because of time or hardware, let others do it for you.

That is precisely the point of separated project repos. Worry about the core code and let each project deal with the core changes at their own pace. Using subtrees (instead of submodules), the core code can even stay with the projects so code that belongs together can still stay together for easy access or customization.

[domoaligato]

I think that it should be as hasu suggested in the OP and structured like cyanogenmod. have a look here. http://wiki.cyanogenmod.org/w/Doc:_the_cm_source

edit: BenBergman... correct. separate project repos for each hardware type and let them take care of themselves.

with that said... this project needs to cleanup some stuff in the core so you do not break the compiling steps. I had to update my keyboard_common.c last night for some weird reason.

[tmk]

Thanks all for your suggestions and discussion. To understand them all I needed time to learn about how 'organization' works and difference between submodule and subtree of git.

Orgnization is useful for team development but this tmk_keyboard is still my personal work and keyboard and converter projects(by other developers and including mine) is independent each other in terms of development. I think it is natural and useful that projects are hosted in their own repository and developed in their pace and preference. So I feel like organization is not needed at this moment at least. (that said I don't have much problem to convert my account 'tmk' to organization when its merit becomes clear to me.)

I'd keep 'tmk_keyboard' repository center of tmk firmware information and README(or wiki) of its 'master' branch will have hyper links to third-party project repositories. This is expected to form 'loosely coupled community' of tmk hopefully and I prefer it to github 'organization'. Related project owners can just let me know to add link on the README.

And I'm planing to offer core library(repository or branch?) to be refered hopefully from projects with subtree or submodule. The core library will be what just removed keybaord/ and converter/ directories from current tmk_keyboard repository, instead example project codes may be included perhaps.

In most cases project will have repository specific to a keyboard or converter but you can host several projects in one repository if you maintain many projects like xauser and I do. 'One repository per project' is very clean and useful for (pure) users but may not for developer. In the end it is your repository you can do anything you like! 'tmk' won't mind :) I'll also do what I want.

[tmk]

Problem is I frequently change core library and break build of some projects and people lose their time to build a broken project report it. And I also have to take time to fix it.

I think 'separaton of core library and projects code' resolves this problem. All projects don't necessarirly need to follow the latest core library, most of projects can stick to old stable commit of the library. So I can leave code of projects intact regardless of frequent change of core library. Now I can change core code without hesitation.

Now what to be considered is how projects already contributed are handled.

[p3lim]

You could perhaps have all work done in a different branch, then when you find that things are stable you can merge that to master and tag it with a version. Then the drivers simply specify which version they were developed for.

[bjd-pfq]

I hereby offer to test, from common usage, and special features upon request, Sun Type 5C keyboards under Linux.

[bgould]

@tmk I think that is a great decision. It will be much easier to keep organized.

I can maintain the NeXT converter project if that is helpful to you... I have some improvements to add to the matrix code that I originally contributed... as it happens I use my NeXT keyboard almost every day so I've become very familiar with it.

If I might make a suggestion, if you're not already considering this, it might be a good idea to restructure things so that other projects can just clone the tmk_keyboard core as a submodule of their project which would keep them pinned to a specific version until they choose to upgrade if they want to

That is essentially how I did for my https://github.com/bgould/arduino_tmk_keyboard project and it worked fine.

It might helpful for spinning off projects if the build system could be structured to support that sort of directory layout. I'd be glad to help test things out once you get started, if you need it.

[technomancy]

This sounds good to me. I added support for the Atreus in my personal fork a while ago and never got around to submitting it upstream. Now I've found a few things have broken. So I need to keep my Atreus-specific stuff either in a personal fork of this repository or in my own repository that brings this in as a library; the latter is what I prefer even though git submodules kinda suck.

[BenBergman]

@technomancy Check out git sub-trees. They are a bit more complicated to set up and update, but they solve a lot of the problems of submodules. You can keep the sub repo code in your project and make your own modifications while still keeping up with the upstream code.

[azhang]

I think https://github.com/clibs/clib, a package manager, might help with a lot of the issues brought up here. I've never used clib before, but it seems similar to https://github.com/npm/npm (node.js package manager) which I have a lot of experience with.

Organization is useful for team development but this tmk_keyboard is still my personal work and keyboard and converter projects(by other developers and including mine) is independent each other in terms of development.

I think Github organizations are useful for any collection of related code, not necessarily for team development. It's like having all the related projects in one easy to find place. Not really necessary if you use a package manager unless you're planning to break down the core lib further.

I'd keep 'tmk_keyboard' repository center of tmk firmware information and README(or wiki) of its 'master' branch will have hyper links to third-party project repositories. This is expected to form 'loosely coupled community' of tmk hopefully and I prefer it to github 'organization'. Related project owners can just let me know to add link on the README.

clib comes with a search function, which you can use in conjunction with namespacing to easily find all 3rd party tmk projects. For example, for browserify in npm: https://www.npmjs.com/search?q=browserify . You could set up a naming scheme like tmk-abc (e.g. tmk-gh60, tmk-nerd) and it'd organize itself. Any new project would automatically appear at clib search tmk.

And I'm planing to offer core library(repository or branch?) to be refered hopefully from projects with subtree or submodule. The core library will be what just removed keybaord/ and converter/ directories from current tmk_keyboard repository, instead example project codes may be included perhaps.

Sounds like a good example of a module dependency. Keyboard/converter project repos would just have project specific code, with the package.json file listing core tmk library as a dependency.

In most cases project will have repository specific to a keyboard or converter but you can host several projects in one repository if you maintain many projects like xauser and I do.

Considering different projects may have different tmk core version requirements, I would advise against this with or without a package manager. 1 project per repo would simplify dependencies significantly, and github issues would be clearly separated by project.

Problem is I frequently change core library and break build of some projects and people lose their time to build a broken project report it. And I also have to take time to fix it.

This would not happen with a package manager, since each project must specify dependency versions. You can use git-tag to freeze each version of the core lib, and if a project is stable with that version, it's not forced to change.

[p3lim]

In most cases project will have repository specific to a keyboard or converter but you can host several projects in one repository if you maintain many projects like xauser and I do.

Considering different projects may have different tmk core version requirements, I would advise against this with or without a package manager. 1 project per repo would simplify dependencies significantly, and github issues would be clearly separated by project.

What I think he meant was that some projects, like the Nerd keyboard, have different sizes, or like the GH60 which have several layouts, those may be part of the same project.

Good idea using a package manager tho'.

[technomancy]

This would not happen with a package manager, since each project must specify dependency versions. You can use git-tag to freeze each version of the core lib, and if a project is stable with that version, it's not forced to change.

You don't need a package manager to solve this; it's easy to avoid using git submodules. A package manager only makes sense if there's a nontrivial dependency graph. Maybe if we want to move the LUFA stuff out of the repo it would be a good fit, but certainly not for the problem currently under discussion.

[p3lim]

Got my keyboard running and thought I'd demonstrate a possibility of what we've discussed here.

I added tmk_keyboard as a submodule (subtrees are not easily done on Windows, requires a plugin) and slightly modified the Makefile to support this.

https://github.com/p3lim/keyboard_firmware

[xauser]

Are there any decisions done now on this topic? I have patches to several projects and don't know how to proceed?

[BenBergman]

I changed my project from being a fork to using git subtrees. Nothing really needs to change in the tmk_keyboard repo to allow other projects to take advantage.

I'm not sure if any progress has been made with the existing keyboard projects.

[p3lim]

@BenBergman You do however need to modify TOP_DIR in the Makefile when you're using submodules or subtrees. Other than that, yes, tmk_firmware doesn't need a single change for it to work, but the whole point of this was not only to allow drivers to use tmk_keyboard as a library/framework, but also the separation of the core and the drivers already present in the repository.

[tmk]

Sorry for late move. Yes, I think what Ben and p3lim did on their projects is exactly what I want.

I'm going to create new repository(or branch if possible?) whose name is like tmk_core so that project owner can submodule or subtree it without including unneeded projects and files(tmk_keyboard/keyboard/* and tmk_keyboard/converter/*).

tmk_keyboard will be still main repository for my projects(keyboards and converters) with subtreeing core library codes fromtmk_core.

I'll look into and try this today.

[p3lim]

@tmk Branches are possible, atleast submodules support it.

[tmk]

Created core branch which can be used as library with subtree or submodule. If branch is not useful for this purpose repository(named like tmk_core) can be created. https://github.com/tmk/tmk_keyboard/tree/core

And master_ng branch contains core library as subtree and my keyboard and converter projects. master_ng is going to take palce of current master branch later if it seems to works. https://github.com/tmk/tmk_keyboard/tree/master_ng

Both branches are not cleaned and fixed enough yet but I think the idea itself is clear to you.

For example, to import core library into project repository

$ cd your_project
...
$ git subtree add -P tmk_core --squash  https://github.com/tmk/tmk_keyboard.git core
or
$ git submodlue add -b core https://github.com/tmk/tmk_keyboard.git tmk_core

$ tree
README
Makefile
src
├── matrix.c
├── keymap.c
└── ...
tmk_core
├── common
├── common.mk
├── doc
├── ldscript_keymap_avr5.x
├── protocol
├── protocol.mk
├── README.md
├── rules.mk
└── tool

Am I missing something under discussion so far?

[p3lim]

If you end up going this path, I'd suggest you split up the repository to avoid a branch mess and confusion because of branches (not everyone is familiar with Git), having tmk_core (or whatever you'd prefer to name it) it's own repo, separate from the drivers you'll maintain yourself (as you suggested).

A further improvement upon that would be splitting drivers into their own repositories, preferably in an organization, to allow other maintainers to host all their drivers in a collaborative space, and allow for a cleaner commit log and easier merging. But that is entirely up to you.

[p3lim]

I'm going to bring this topic back up again.

@tmk, when you make changes to the core you seem to do the changes on master, then push the changes upstream to the core branch. If I were you I'd do it the other way around, making core the development branch.

The reason behind this is because of several points I've brought up in this discussion; when you make major changes you can, and will, break drivers.
Also, when you make changes and forget to push upstream to core (which is the case right now), any changes you've done is not available to those who use that branch in other projects/repositories.

[p3lim]

Right now you've got 3 sources for the core:

• The core branch
• The tmk_core directory in the master branch (which is the one you seem to develop with)
• The tmk_core repository

And none of them are the same, adding a lot of confusion.

[tmk]

I've tried both branch and repository for tmk_core for a while and I feel like repository is easier to work with. I'll remove core branch some later probably.

[p3lim]

If you indeed intend to use the tmk_core repository as the source for the core, please start using that as your base of development.

Currently the core repository (and branch in this repository) is 2 months and 25 or so commits behind the tmk_core directory in this repository (which is practically impossible to use in other projects).

[p3lim]

I just noticed you've made a wiki page for how you handle subtrees and the core: https://github.com/tmk/tmk_keyboard/wiki/Git-subtree

Development on the core in a separate repository simplifies this a lot, and it also allow people to easily pull the changes too.

---

Setup

One-time setup for the subtree in the driver repository (_tmkkeyboard in your case).

Add a new remote URL for the core repository git remote add -f core https://github.com/tmk/tmk_core

Add the core as a subtree in _tmkkeyboard: git subtree add -P tmk_core core master --squash

---

Updating the subtree

After you've made changes to the core (_tmkcore repository) you can pull the changes like this: git subtree pull -P tmk_core core master --squash

[xauser]

How are patches to the keyboard section of tmk_keyboard handled? I have patches for nerd, lightsaber and others in my tree. Will they be maintained in tmk_keyboard repo in future or removed from there? Users should be able to find the latest sources easy.

[p3lim]

How are patches to the keyboard section of tmk_keyboard handled? I have patches for nerd, lightsaber and others in my tree. Will they be maintained in tmk_keyboard repo in future or removed from there? Users should be able to find the latest sources easy.

Anything related to the drivers (anything within the keyboard and converter directories of this repository) are still being maintained in this repository. Hasu might change his mind later and separate them into different repositories, that's all up to him.

If it were up to me I'd make an organization and split the drivers into their own repositories.

[tmk]

@xauser, you can send PR to original author's repository or keep your own repository up to date so that people can clone. People can find the repository from this wikipage. (I think you are author of those projects, though.) https://github.com/tmk/tmk_keyboard/wiki/TMK-Based-Projects I will add link to that page in tmk_keyboard/README.md so that people can find the repositories easily.

This is my intention here and I hope it works.

[tmk]

@p3lim, thanks for subtree instruction. I don't get used to usage of subtree and can't feel comfortable wit it yet, to be honest. It will take time for me to learn new thing as always.

Yes, I want people to subtree(or submodlue) tmk_core repository instead of tmk_keyboard when they start their own keyboard project(driver) repository. They can add their repository URL on the wikipage.

[tmk]

Moved some projects to orphan directory now. 649151cbeb84

[vovcacik]

I am sorry but this is confusing. You are developing in tmk_core directory, but you want people to use the tmk_core repository which seems to be outdated. The tmk_core repo also states the the source code is to be taken from the core branch which has been deleted.

[tmk]

You can think tmk_repo is stable because it is tested here :p I'll update tmk_core repo occasionally. to be honest, git subtree is cumbersome to me and it prevents me from syncing the repo. I'm thnking about using submodule for tmk_core in this repo. submodule is probably easy to maintain.

[p3lim]

They are pretty much equally easy to maintain, only major difference is that submodule doesn't pollute the commit log.

[tmk]

I don't think so with my current knowledge. but I'll learn something anyway after trying submodule.

EDIT: I agree about commit log.

[p3lim]

The upside with subtrees is that they are transparent to the user, as in, users don't need to do anything extra. With submodules they need to clone with an additional parameter (git clone URL --recursive), and when pulling they need to update the submodule(s) as well.

[p3lim]

Here's a short guide to submodules like the one I did for subtrees.

---

Setup

One-time setup for the submodule in the driver repository (_tmkkeyboard in your case).

Add a new submodule on the path "tmk_core": git submodule add https://github.com/tmk/tmk_core tmk_core

Initialize and update the submodule: git submodule update --init

---

Updating the subtree

After you've made changes to the core (_tmkcore repository) you can pull the changes like this:

cd tmk_core
git pull

---

Here is a great (and complete) write-up:
https://medium.com/@porteneuve/mastering-git-submodules-34c65e940407