[JENKINS-13810] Maven modules marked to wrong build when running concurrent job

[timja]

I have a Maven project build with one module, named jenkins-test. This module contains one test, which fails everytime (assertTrue(false) ). There is a problem when this project is executed with "Execute concurrent builds if necessary" turned on and when multiple jobs are started at same time. Multiple builds start and finish, but some of these jobs are completed succesfully and some are marked as failed. What I see in the build status page is something like this:

(Success) Build #180 (May 12, 2012 8:17:09 PM)
...
Module Builds
jenkins-test (didn't run)

(Success) Build #181 (May 12, 2012 8:17:12 PM)
...
Module Builds
(disabled) jenkins-test (didn't run)

(Unstable) Build #182 (May 12, 2012 8:17:19 PM)
...
Module Builds
jenkins-test (Unstable) #182 (Unstable) #183 (Unstable) #184

Logs show that the builds were executed properly (commit ids are right, and "[ERROR] There are test failures." is there). The next build then gets build number #187.

In my setup I'm using Gerrit and Gerrit Trigger to launch jobs, so when I push multiple commits into Gerrit in one push, Jenkins starts multiple jobs at same time. I don't know if you can duplicate this behavior without Gerrit, though.

Currently MavenModule and MavenModuleSet have their own build numbers, which are increment when a new module is built. These might get mixed up when multiple builds are executed concurrently. Multiple MavenModuletSets are built starting at the same time, and some how the numbering of MavenModules gets out of sync.

The commit in https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4 fixes this, but I haven't tested it other cases than mine and I don't think that I'm qualified to change something that is this deeply on the jenkins core The commit probably has some unnecessary bits in it too at the moment. So I don't think that it should be merged

---

Originally reported by jyrkip, imported from: Maven modules marked to wrong build when running concurrent job

• assignee: olamy
• status: Reopened
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

evernat:

Is it reproduced with a recent Jenkins version?

[timja]

jyrkip:

I've changed to freestyle project with Maven buildsteps at the end of February, and this was valid then. Jenkins version was 1.502.

[timja]

evernat:

Neither you or me are able to test this issue with recent Jenkins/Maven/Gerrit.
So resolving this issue as can not reproduce. Please reopen if reproduced.

[timja]

ifernandezcalvo:

I could replicate it on Jenkins 1.609, 1.625 and 1.642, with a simple project with 4 submodules.
I launch 10 build with the command

for i in {0..9}; 
do 
    echo $i
    curl http://127.0.0.1:8080/job/mv/build?delay=0sec
done

[timja]

aheritier:

Still valid with 2.46.x and maven-plugin 2.14

Just create a maven job to build maven itself (https://github.com/apache/maven) and configure it to "Execute concurrent builds if necessary"

Configure your instance (several executors on master) and launch several builds in a short period of time.

You'll quickly see several modules attached to same parent build by error

 

 

 

[timja]

aheritier:

olamy do you have in mind how are linked modules builds with the project build?

[timja]

aheritier:

olamy cc oleg_nenashev

The problem from my POV is that we don't/cannot pass an identifier to link a MavenModuleSetBuild and its MavenBuild(s) (We just trigger a build of each module from the Maven project)

https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/MavenModuleSetBuild.java#L712

And thus we hack to link them 

https://github.com/jenkinsci/maven-plugin/blob/master/src/main/java/hudson/maven/MavenModuleSetBuild.java#L518

WDYT ?

 

 

[timja]

olamy:

aheritier good catch! I reckon I never use concurrent build option

The fix by jyrkip seems to be ok https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4 

[timja]

aheritier:

good finding olamy you're the boss !!

but https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4 will requires an update of Jenkins core API and the maven plugin. I will discuss about it with oleg_nenashev but I'm not sure to have the ability to convince the community to update the Jenkins core for a need from the maven plugin  (which is really specific due to the use of a real Jenkins job for each Maven module - I don't see others cases where you could need such kind of coherency across jobs). And I'm not sure that I can implement a similar fix without the core part

 

[timja]

oleg_nenashev:

> I will discuss about it with Oleg Nenashev but I'm not sure to have the ability to convince the community to update the Jenkins core for a need from the maven plugin

I see no problem with that if it's implemented in a graceful way. After a quick review I do not see anything insane in https://github.com/jyrkiput/jenkins/commit/0d076e610b39b215ed34c35a5d4840e56183d5e4

[timja]

aheritier:

oleg_nenashev no the code itself seems ok (but it has no tests - TBD). But if we go in that direction it means that we have to update the Jenkins core and to do a new major release of the maven plugin which is targeting this core release at minimum (For now we are still compatible with 1.625.3)

[timja]

olamy:

aheritier OMG good luck!!   well it's only a new field with a buildNumber nothing really big change and should not break anything (IMHO)

[timja]

owenmehegan:

Is there any chance this is going to be resolved?

[timja]

olamy:

pr here https://github.com/jenkinsci/maven-plugin/pull/118 

I tried to avoid core changes but need more testing especially with large projects....

[timja]

olamy:

uhm I have an issue with core  

java.lang.IllegalStateException: JENKINS-27530: cannot create a build with number 47 since that (or higher) is already in use among [5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 30, 32, 33, 34, 35, 36, 38, 39, 41, 42, 43, 44, 48, 49]

Well I guess 47 is possible as not used
oleg_nenashev jglick any idea?

[timja]

oleg_nenashev:

Yes, I would expect core to handle such cases when parallel runs are enabled for AbstractProject. Unfortunately many APIs actually do not support it

[timja]

jglick:

No oleg_nenashev, parallel builds are normally fine, since build number allocation is done by a synchronized method on the project.

When this ISE occurs, something is badly broken. We just do not know what it is.

[timja]

jebeaudet:

Any chance of this being resolved at some point? We had a test job that had failures reported in another concurrent run, so the job was green, even though it had failures. Thanks

[timja]

JIRAUSER143774:

We also got this issue, a build was reported as red because of another job's failing test.

[timja]

JIRAUSER143781:

We've also been experiencing the same issue. Hope it gets fixed soon.

[timja]

jglick:

Use of maven-plugin has been strongly discouraged for years, so I would not expect any Jenkins developers to spend time investigating this. Switch to freestyle or Pipeline builds.

[timja]

[Originally related to: JENKINS-31322]