Tagging of project with sub-project fails

[esocode]

I have a Git project that contains sub-projects. If I try to tag it (with gradlew tag -Prelease -PbumpComponent=minor) I get the error message "Cannot create a release version when there are uncommitted changes". But the project is unchanged, git status yields "Your branch is up to date with 'origin/master'. nothing to commit, working tree clean".

The Gradle stacktrace:

org.gradle.tooling.BuildException: Cannot create a release version when there are uncommitted changes                               
        at net.vivin.gradle.versioning.VersionUtils.determineVersion(VersionUtils.groovy:82)                                        
        at net.vivin.gradle.versioning.VersionUtils$determineVersion.call(Unknown Source)                                           
        at net.vivin.gradle.versioning.SemanticBuildVersion.toString(SemanticBuildVersion.groovy:157)                               
        at net.vivin.gradle.versioning.SemanticBuildVersioningPlugin$_apply_closure1.doCall(SemanticBuildVersioningPlugin.groovy:40)

[vivin]

What version of the plugin are you using?

[esocode]

4.0.0

[vivin]

Not entirely sure why this would be happening. Previously I've seen some issues due to JGit incorrectly assuming that there are changes.

Any other information you might have?

[esocode]

You can reproduce the problem, the affected project is open source on github:

git clone https://github.com/esoco/sdack
cd sdack
git status (yields "nothing to commit, working tree clean")
 ./gradlew tag -Prelease (yields "Cannot create a release version when there are uncommitted changes")

[vivin]

@esocode I will try to take a look. I'm pretty sick with the flu so it may take some time.

[esocode]

One could guess from your profile picture ;-) Get well soon.

[esocode]

Any news on this?

[vivin]

Oops. I'm so sorry. I got caught up with a bunch of stuff -- I'm switching jobs at the moment and so I'm a bit busy with that.

On Tue, May 8, 2018 at 12:18 PM, eso notifications@github.com wrote:

Any news on this?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/vivin/gradle-semantic-build-versioning/issues/94#issuecomment-387512662, or mute the thread https://github.com/notifications/unsubscribe-auth/AADbHkgNYgF8DZt3jJsxzLKf38Bx74C_ks5twe-egaJpZM4Smfg_ .

-- [vivin.net http://vivin.net :: github http://github.com/vivin :: linkedin https://www.linkedin.com/in/vivin]

[esocode]

Job switched? Just a short reminder ;)

[vivin]

@esocode Yes :).

So I've looked into this and it might be due to the fact that you're using submodules. jgit apparently seems to think that esoco-*, gewt, and objectrelations directories are missing. I'll try to dig into it a bit more and see what I can find.

---

Ok, I looked into this and it appears to be a jgit problem. There's an existing issue tracking this bug, but the last comment is from 2015 and it seemed to suggest that this was a Mac OS X issue. But I was able to verify that it happens (at the very least) on Linux too (Linux Mint 18.3). I've added a new comment stating as much. I provided a link to your public git repo as well; I hope you don't mind.

---

Short version: you can get this to work for the time-being if you checkout all your submodules.

Long version: I have a fix so that the plugin so that it doesn't list unchecked-out submodule directories as missing. But the problem is that it won't track any changes in submodule directories either. To be frank, I am not sure if this plugin will work well with submodules or not because it was never really designed to take them into consideration. So I'm reluctant to commit that fix without fully understanding all the implications.

This is an issue in jgit that needs to be fixed, I think -- it needs to be able to understand that those directories aren't "missing"; they're just unchecked out submodules. There are some related issues in ajoberstar/grgit#60 and ajoberstar/grgit#80 that are waiting for the same jgit fixes.

[esocode]

Thanks for looking into this. I didn't consider that this could be caused by a dependency, although in hindsight this seems logical. Unfortunately it doesn't help to checkout the submodules, the error is still the same. The example above was only a simplified version to demonstrate the problem. You can check this by adding git submodule foreach git checkout master after cd sdack in the example.

[vivin]

@esocode Are you sure? I used git submodule init and then git submodule update to fetch all the submodules and the problem seems to go away.

However, I identified the actual issue in jgit (see here) and I'm going to see if I can commit a fix to take care of this.

[esocode]

Yes, I'm sure 😄 I also tried the init/update commands but always get cannot create a release version when there are uncommitted changes although git status reports no error.

[vivin]

Ah I think I know the issue -- looks like checking it out only works in jgit 5.0.1 (which is what I was using to test out the versioning plugin). So for the time being, I will push up a version that uses jgit 5.0.1 so that you can at least use the work-around (I'll try and push it tomorrow).

I fixed the actual issue on jgit and I will be submitting a patch soon.

[vivin]

@esocode btw, I tried the latest version of your repo with the latest version of jgit (from a local version of the plugin I made) and I get the "uncommitted" message in both cases. Not sure what changes you made, but now when I step through the code, jgit seems to think that a ton of your files are modified/missing (this happens after I do a submodule update). Oddly enough, I don't see that when I use the command-line jgit tool. I think this just might be the way this plugin behaves with submodules (i.e., it doesn't behave very well).

[esocode]

Yeah, there are unfortunately some tools that don't behave well with Git submodules. As a very prominent example, IDEA doesn't support them at all. Eclipse on the other hand handles them without problems.

The project structure of the sdack repo hasn't changed, only new commits have been added. I have no idea why jgit thinks something is changed, especially because Git isn't reporting any changes at all. Will your fix correct this behavior? Or does your latest comment suggest that it still occurs, even with the fix?

[vivin]

@esocode I think I know what is happening. It looks like when the submodules are checked out, the plugin compares the state of the submodule not to itself, but to the root. This is because subprojects are assumed to be in the same repo. So I think that's probably what's happening here. I don't know why this wasn't an issue before (or maybe it was and I didn't notice it since it was pretty late).

You did uncover a legitimate bug in JGit though, but unfortunately the resolution of that bug won't really help you here. I'm guessing it might be possible to detect whether the subproject is in a submodule, but as I said before, I'm not sure how tagging and versioning semantics work in that case. Do you now where that has been described in detail? For example, if you make a change in the submodule that is not committed, can you still tag the root repo? What about when you have nested submodules? How does that work?

I think a lot of assumptions change when dealing with submodules and that could make the plugin very confusing. I'll try to play around with this some more to see if I can figure something out.

[esocode]

Sub-modules in Git are actually trivial: they are just a folder that links to a certain commit in another repository. They are then just handled as normal git repos, recursively (even for more than 2 levels). Repos on any level can be handled like any "simple" Git project, including tagging. To update a submodule in a parent project, only the commit reference to the submodule needs to be updated. All other changes are made in the submodule project and won't affect the parent at all.

But tools need to be aware of that structure and handle the recursion correctly or otherwise they may interpret the submodule directories as a file of the parent instead of a repo to recursive into which I think may be the cause for the problem.

[vivin]

@esocode yes; I believe that is what is happening now. So a question. Would you expect this plugin to be able to push a tag for a submodule? Or, how does that work? I notice that in a submodule you're in a detached HEAD state (because I'm assuming the head of the parent project has been detached and is now pointing to the HEAD commit of the submodule).

Right now if there are any uncommitted changes in a project (including a subproject), the project won't let you do things like tagging. If I understand correctly, as you said, changes in a submodule don't affect the parent. But I'm interested in understanding what exactly happens when you tag a project with submodules. For example, let's say you have a project with submodules sub1, sub2, sub3, each pointing to commit1-sub1, commit3-sub2, commit5-sub3 (which are the current HEADs of those repos). I then tag the parent project and push up the tag. Now if I check out that particular tag in the future, will the submodules still be pointing to those particular commits, even if there have been new commits on those repos? The reason I ask is, let's say that you made new commits on a submodule, but haven't pushed them up. Then you tag the parent project, and then go back and modify the history on the submodule (maybe squash or something) and then push up those changes. If I check out the tag on the parent project, what commit will the submodule link be pointing to?

Sorry for all of these questions -- I'm trying to gain a better understanding so that I can make the plugin aware of submodules.

[esocode]

Regarding your first question: no, I neither expect nor want the plugin to tag submodules (see below for details). I just want to tag the parent project - the sub-projects are developed and therefore tagged individually.

Tags are completely independent from the submodule handling. Each project is handled on it's own with respect to tagging, committing etc. To apply git commands to submodules the command git submodule foreach <command> can be used, which is basically just a helper that visits all submodule folders and invokes the command. Using submodules is only a method to handle multiple (typically dependent) projects together. For example, the sdack project mentioned above combines several other projects which together form a stack of domain-specific frameworks. To use that stack I it is only necessary to clone the parent project with it's submodules (easiest achieved by using the --recursive flag).

A somewhat non-intuitive behavior of Git is that the submodules are in the detached head state after recursive cloning. That is probably because of the fact that a submodule is only a reference to a certain commit of the sub-project. Therefore, if a certain branch is needed, e.g. master, that branch must be checked out for all submodules - assuming that the branch has it's head at the same commits as that stored in the parent, else the parent would have uncommitted changes because the commits would not match.

But for tagging it should be completely sufficient to only clone the parent non-recursively. Git then reports the project as unmodified because the state of the (now empty) submodules is not considered, only the associated head commit.

[vivin]

That makes more sense. I've been reading a few articles about submodule best practices and they all generally advise against modifying submodules from within parent projects (for the reasons I mentioned earlier). They also say that you should use regular dependency-management if you can, instead of using submodules to establish dependencies. I can see that in your project, you only use the local submodule if the directory actually exists; otherwise you use an explicit version.

My original concern was that we cannot expect all users to follow these best practices, and I wanted to minimize the chances that they would shoot themselves in the foot. However, it appears there are multiple ways that things can go wrong when dealing with submodules and so it would be a tall order to enforce these best practices; plus that's not the job of this plugin anyway.

Maybe it's worth mentioning in the documentation that in the case of submodules, this plugin treats them as independent, separate projects (since that's what they are effectively, anyway) and will not try to enforce any sort of best practices based on the state of those submodules, with respect to their parents. This should at least discourage modifying subprojects in-place.

What do you think?

[vivin]

@esocode I've submitted a fix for the original issue, btw. You can see it here.

[esocode]

I fully agree that the plugin should not enforce any best practices whatsoever. It should simply tag a project with submodules as any other if Git reports it as unchanged.

In my case I'm using the plugin as an intermediate stage for automated deployment of new releases to bintray/jcenter. I do that for every single project separately (because they are developed separately). But because of the bug this doesn't work for the sdack project which contains the others as submodules and therefore I have to do the tagging and subsequent deployment manually.