Closed mattab closed 8 years ago
+1 for this. We can then manage all Piwik plugins (Core + Premium) in unified way.
+1 @mgazdzik Adding "premium" plugins to composer.json won't work. If the repos are not public, people without access might get an error.
Instead of using composer we could also create a new console command to auto install all maintained plugins (which might contain premium plugins, that are ignored if there is no access given)
+1 for composer. It's tool which is using in many PHP projects so I think that in someway becoming standard in PHP world.
@sgiehl - on the other hand we don't want just anybody to download premium plugins I think. Also I rather ment those complete files rather for internal use of refactoring, testing, etc.
Can you explain step by step what is to be done and which commands are to be executed when having to use a branch for plugins? And maybe also what is to be done when merging? To see whether it makes it actually easier to work with branches...
Is there a reason why plugins in submodules are not in this repository?
I get that they are optional plugins, but we could have them in piwik/piwik disabled by default.
we don't need to bundle those plugins with core because they are optional. they are available via the Marketplace. Ultimately we would like to move more plugins to the Marketplace to keep core smaller, eg. #5341
Then why not removing the submodules and let developers checkout those plugins manually (or with a command or script)? PhpStorm can support all git repositories inside the project (e.g. I update with PhpStorm and it updates every plugin). The submodules brings no value except the initial installation.
Then why not removing the submodules and let developers checkout those plugins manually (or with a command or script)?
it will be error prone to clone all the repos.... maybe we could make it automatic? eg. maybe a new console command could init all repositories that the developer has access to and that we need to maintain as part of core?
For me removing submodules is a high priority because it's pissing me off every single day. Every time I pull master submodules are in detached state and I have to re-checkout master on most of them so that PhpStorm works again. It also messes up git rebase, and sometimes when I switch branch I end up committing submodule updates when I should not.
If we need to clone 8 repositories, that's doable. Here is a script:
git clone https://github.com/piwik/theme-PleineLune.git plugins/PleineLune
git clone https://github.com/piwik/plugin-SecurityInfo.git plugins/SecurityInfo
git clone https://github.com/piwik/plugin-TreemapVisualization.git plugins/TreemapVisualization
git clone https://github.com/piwik/plugin-VisitorGenerator.git plugins/VisitorGenerator
git clone https://github.com/piwik/plugin-CustomAlerts.git plugins/CustomAlerts
git clone https://github.com/piwik/plugin-TasksTimetable.git plugins/TasksTimetable
git clone https://github.com/piwik/plugin-LoginHttpAuth.git plugins/LoginHttpAuth
git clone https://github.com/piwik/plugin-QueuedTracking.git plugins/QueuedTracking
There would still be left ui-tests and the php tracker but at least that's a lot of time saved already.
And if we need to make sure those plugins are not broken, we don't need to check them out locally: we should use CI. #6542 Sounds like the solution.
For me removing submodules is a high priority because it's pissing me off
feeling the pain :+1:
Here is a script:
maybe we have to clone using SSH not the HTTPS otherwise it won't use the SSH key (when two factor enabled?)
git clone git@github.com:piwik/theme-PleineLune.git plugins/PleineLune
git clone git@github.com:piwik/plugin-SecurityInfo.git plugins/SecurityInfo
git clone git@github.com:piwik/plugin-TreemapVisualization.git plugins/TreemapVisualization
git clone git@github.com:piwik/plugin-VisitorGenerator.git plugins/VisitorGenerator
git clone git@github.com:piwik/plugin-CustomAlerts.git plugins/CustomAlerts
git clone git@github.com:piwik/plugin-TasksTimetable.git plugins/TasksTimetable
git clone git@github.com:piwik/plugin-LoginHttpAuth.git plugins/LoginHttpAuth
git clone git@github.com:piwik/plugin-QueuedTracking.git plugins/QueuedTracking
maybe we can create some command like development:git-init
or something better?
The reason why we had choosen submodules was to automatically include those optional plugins in every dev environment. I'm still thinking that it is necessary to do that. Otherwise no one would include them manually, even if it would be easy doable with console command. And of course everyone would maybe use another "version" of those repos. With a submodule (or subtree) the version is given. Without that some errors might not be noticed.
Another question why did we start to move optional and required parts into submodules? I'd prefer to do that with optional parts only. Required parts should be required using composer or something similar. The current way makes it difficult to deploy using git as some but not all submodules needs to be fetched, aswell.
I don't see so many problems with submodules that you probably do.
Small hint: To get all submodules to latest master I simply use:
git submodule foreach "git checkout master && git pull"
The reason why we had choosen submodules was to automatically include those optional plugins in every dev environment. I'm still thinking that it is necessary to do that. Otherwise no one would include them manually, even if it would be easy doable with console command.
for anyone who uses the Phpstorm then it can be configured to "git pull" all the repositories that Phpstorm finds in the project. So everyone will use the master branch of all plugins. If they don't use Phpstorm, maybe they could use the console ./console git:pull
which could be edited to also git pull all repositories within the path?
Another question why did we start to move optional and required parts into submodules?
only the PiwikTracker submodule is actually required... there was no other easy way to stay BC while moving it to its own repository. See discussion in #6870 and https://github.com/piwik/piwik-php-tracker/issues/1
Required parts should be required using composer or something similar.
definitely we'll do this whenever possible
The current way makes it difficult to deploy using git as some but not all submodules needs to be fetched, aswell.
it's true that the FAQ about Git deploy mentions php console git:pull
but I don't think currently it will fetch the PiwikTracker submodule. I think this is a bug (maybe git:pull could automatically init the submodule if not yet init)...
to automatically include those optional plugins in every dev environment. I'm still thinking that it is necessary to do that.
We have to keep in mind there are "contributors" and "contributors". People working everyday on Piwik will check out those plugins, as well as other plugins too.
Hm. I'm not sure about that. Another advantage of submodules: I'm sometimes switching between Piwik version tags to test issues. That might get more difficult when not having the submodules, as newer versions of those plugins might break older Piwik versions. With submodules automatically an older version of those plugins will be checked out.
We can still have some sort of version dependencies for plugins (e.g. using composer). I have to agree with what is said above that submodules cause more pain than good.
Can we use composer to require optional plugins? I don't think that is possible...
Following a discussion today, here is a suggestion: replacing submodules with git subtrees (≠ the whole process described in #6757)
We would still keep the current advantages of submodules:
We would gain:
submodule update
or git pull
in submodules: such plugins would be like if they were included in piwik/piwik
piwik/piwik
and it is also created in subtreesCons:
To sum up, it would be just like if those plugins are inside piwik/piwik
except they are also synced in a separate repo.
Thoughts?
+1 for using git subtrees for open source plugins that are released in the marketplace.
I would git-tree split all plugin-*
repos in our Piwik Github org, except plugin-PiwikDebugger
. That's even more than we currentl have submodules but I think they were not added as submodules because we forgot to?
maybe others have more thoughts?
I have done some tests with git subtrees and the script I mentioned, it worked as expected.
You can see an example here:
foo/
directory (so that repo would be the same as a Piwik plugin)I'm suggesting to try that with one of our plugins that we want always installed, for example VisitorGenerator
.
To make this work we would need a server to setup a github hook that would sync the subtree on each push: @mattab I guess that's possible?
We would also need to update the package script to remove the subtrees when doing a release. That's doable too.
The only problem I've identified for now is that branches in piwik/piwik would be synced to plugins, but never deleted when they are deleted from piwik/piwik. This is an issue identified in https://github.com/dflydev/git-subsplit/issues/6 and I'm sure we could implement that.
All good for everybody?
Oh one thing I just think about now: that would also synchronize the versions. So plugins setup as git subtrees would have the same versioning as Piwik itself :-/
I don't know if that's a good thing…
To make this work we would need a server to setup a github hook that would sync the subtree on each push: @mattab I guess that's possible?
We have this already for git diff emails, so no problem
So plugins setup as git subtrees would have the same versioning as Piwik itself :-/
That's not really a good thing since we want to manage and release them independently, there are less often actually changes so version number would less often increase but sometimes we want to release a new version before Piwik etc.
A little summary:
We cannot get rid of submodules at this stage. We're considering automating tracking of submodules to latest commit, see issue #8425
Maybe we could get rid of submodules using the upcoming open source tool described here: https://speakerdeck.com/fabpot/a-monorepo-vs-manyrepos ?
We likely won't get rid of submodules and if/when we do, we can reopen this issue or a new one
The goal of this issue is to investigate our uses of sub-modules and ultimately get rid of them to use a better way instead.
What do we use submodules for? SM or submodules are a feature of git that we use at Piwik to keep track of
Why getting rid of SM is important? Submodules are quite painful to use by themselves and especially with branches. This conflicts with our project to use a always-green
master
branch and adevelop
branch #5954 - because we need a master green at all times,Idea: replacing submodules with Composer
What if we would simply use Composer tool instead of submodules?
As a developer, maybe Composer could help me checkout all the core plugin repositories (they can be put in the
composer.json
sectionrequire-dev
). Each of those plugins eg. inplugins/CustomAlerts
would be a clone of the git repo (see this SO answer).Managing branches across many repositories Using phpstorm I can easily add the repos to Settings>Git and then:
Other ideas/brain dump
tests/PHPUnit/UI
submodule in another issue (out of scope?)What do you think of this?
RFC @tsteur @diosmosis @mgazdzik @mnapoli @vox3r @czolnowski @sgiehl @JulienMoumne @halfdan @quba @tzi