Maven Multi Module Builds

[krothu]

We have a large multi module build using the maven reactor and would really like to be able to see the modules broken down into seperate status bars.

Is possible to track which sub-module is being built as its own status bar from the parent, this would especially be helping in a multi module build as we use threading argmuments to build them. It will also allow use to see the final status of each module as we use "--fail-at-end". Potentially when we do a release, using the mavne-release-plugin we can also see the release status of each artifact.

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/2116782-maven-multi-module-builds?utm_campaign=plugin&utm_content=tracker%2F580790&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F580790&utm_medium=issues&utm_source=github).

[drekbour]

@jan-molak, Can I take a run at this one? I'll take a look at the un-merged code for #62 (matrix builds) for inspriation but it will take a few months for me.

[jan-molak]

@drekbour - of course you can! Thanks for offering your help :smile:

[drekbour]

Initial thoughts - there are several usecases here so I will implement multiple implementations of support

"Display Maven modules" : [Hide|InPlace|Expanded]

Hide Today's default behaviour. With a bit of bedding-in InPlace could be the default in future.

InPlace Multi-modules for code-seperation reasons (core/ui/installer/etc). These probably shouldn't be expanded as it is only the success of the group that matters but it would be nice to amend the lozenge title dynamically as they build or fail.

• Use of FAE means there might be >1 failure
• Use of parallel buids means there might be >1 module executing at a time

Expanded Multi-modules for aggregation reasons. Larger projects often need aggregator poms to group related items whilst still delivering multiple projects (example - https://github.com/apache/camel). These should be expanded with a lozenge per module

Not in scope

Per project I see both of the above being possible in the same project. This means you will want Expanded aggregator which may contain n-levels of nested projects, each of which might want Expanded or InPlace handling. It seems reasonable to honour a property added to the pom

 <jenkins.build.monitor.mode>InPlace</jenkins.build.monitor.mode>

The existence of this property would supercede the above UI settings

• I don't expect to implement the per-project in this request
• Although the code could easily support generic Matrix builds too I will explicitly constrain to Maven jobs

[krothu]

Would your solution be able to handle several of the modules being built at the same? We are using the Jenkins threading options to help to keep the build times down. Setting our Jenkins build "Goals and options" for our parent pom to "-T2 clean deploy --fail-at-end".

Just to let you know currently our multi maven parent pom has about 42 modules (we have never really tried going above -T4), and takes a long time.

Also our Jenkins builds and monitors are on large screens near our desks and are already limited on space. If the main build had some kind of indicator on the current module being built embedded would suit our use case.

[drekbour]

"Expand" appears to work just fine with threading and FAE. "InPlace" clearly has some issues here so I'll just take an opinionated decision on what it should display and if the community doesn't like it - this can be changed in future. In the case of multiple runners or failures, the active/failed module shown will be the one that would have stopped the build if it were not using threads and/or fae (Jenkins is aware of Maven's module ordering so can). Obviously I'll add "and N others" where appropriate

[drekbour]

PS. Read through #124 and #57 if working with large numbers of jobs

[drekbour]

@jan-molak For InPlace mode described here, I want to add a "subtitle" to the job which, in this case - shows the active Maven module during a build and the failed module when appropriate. Browsing though the many other issues and PRs I see people wanting similar "compound jobs" for other Jenkins features: dynamic (sub) title, nested lozenges or lozenges somehow structured together.

What is your opinion? I think it's time you decided on how to support higher-order builds (Maven modules, Matrix, Multi-job, pipeline etc) as the implementations of each of these are going to conflict terribly with each other.

[jan-molak]

@drekbour - yeah, I can certainly see the need for "higher-order jobs". I'm not sure what the answer should be though yet..

Let me think out loud for a moment :-)

It seems to me that the problem is worth splitting into two smaller ones:

• What the job is?
• How the job is represented/visualised?

So far it looks like each type of a higher-order job could require a different visualisation:

• a Cloudbees Folder "job" could be presented as either passing or failing depending on the worst status of a job it encapsulates
• if a job has downstream dependencies - it could be visualised as a pipeline
• a matrix job could be visualised as a ... well, a matrix.
• a multi-module job could be visualised in a similar fashion to a matrix job, maybe showing passing/failing colour blobs per module, or just highlighting the failing modules to avoid clutter.

It also looks to me that we might need to categorise each job depending on its traits*. This way we'd have a:

• SimpleJobView - a simple job with no downstream projects, no modules, regular project with no bells nor whistles
• MultiModuleJobView

to start with, and perhaps:

• CloudbeesFolderJobView
• PipelineJobView (or something like this ;) )
• MatrixJobView

and others, should people need them later?

A MultiModuleJobView could then have additional properties specific to that type of higher-order job, such as a list of modules and their passing/failing states. This could be then represented accordingly in the UI (the JSON representation of the Java object can have a type property multi-module or something like this).

What do you think?

---

* Or perhaps tag it instead? With, say: "MultiModule, Claimed, ShowingCulprits", or something similar? Categorising feels simpler to start with though

[drekbour]

It's a complicated space. I'll add some more.

• Parallel builds of "simple" jobs.
• Matrix builds do not mean 2 dimensional. You can have 1 to N dimensions.
• A pipeline (never used this plugin but am guessing) can have artifact A at stage 3 and artifact B at stage 1. Hence it quickly can look like a 2D matrix.
• All of these can be combined arbitrarily E.g. a matrix build of a multi-module project configured for parallel Jenkins builds. Not sure if you can pipeline a matrix build in practice but you should be able to.
• Many users have vast arrays of jobs - see the 200+ issue. Personally I am very happy the dynamic view implemented to show only running and failed jobs. This board now provides valuable information only. Combine a dynamic view (where jobs jump in and out of existence all the time) with all the above?
  • Ps. please integrate the hiding header/footer change!

The rabbit hole goes deep

So I wouldn't get hung up on implementation at all. It's right to think what is the job and how should it be (meaningfully) shown on a dashboard. What is reasonable to support and what not? I certainly like the idea of different options being available like the Expand and InPlace presented here for multimodule - each has a real usecase behind it. The "per project" mode is also a real usecase so thats three styles just for one feature. I think the same ideas apply to others - a matrix job on a small board might be simple enough to expand. A 5D matrix on a busy board cannot be expanded in any rational way yet these are the users who most need visibility.

I didn't (and still haven't) tried to think much further on this apart from liking choice, liking the subtitle mode in theory and realising we can have POM property control the board's choices here.