Coordinator project double triggering projects

[tzedar]

I have a main coordinator project that triggers other projects to run my tests. Sometimes I get the project triggered once and sometimes it triggers twice even though I have it configured only once. Below is the console output of the coordinator job showing that some jobs are triggered two times in a row.

Atomic Job: Clone Testing Repos - Android #1012 Completed, Result: SUCCESS Atomic Job ( Turn MFI Development Bridge ON ) Triggered Atomic Job: Turn MFI Development Bridge ON #37 Completed, Result: SUCCESS Atomic Job ( Physical Galaxy S6 Smoke Test MFI ) Triggered Atomic Job: Physical Galaxy S6 Smoke Test MFI #12 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy Note 5 API 23 Smoke Test MFI ) Triggered Atomic Job ( Emulator Galaxy Note 5 API 23 Smoke Test MFI ) Triggered Atomic Job: Emulator Galaxy Note 5 API 23 Smoke Test MFI #76 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy S3 API 17 Smoke Test MFI ) Triggered Atomic Job ( Emulator Galaxy S3 API 17 Smoke Test MFI ) Triggered Atomic Job: Emulator Galaxy Note 5 API 23 Smoke Test MFI #77 Completed, Result: FAILURE Atomic Job: Emulator Galaxy S3 API 17 Smoke Test MFI #8 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy S6 API 22 Smoke Test MFI ) Triggered Atomic Job ( Emulator Galaxy S6 API 22 Smoke Test MFI ) Triggered Atomic Job: Emulator Galaxy S3 API 17 Smoke Test MFI #9 Completed, Result: FAILURE Atomic Job: Emulator Galaxy S6 API 22 Smoke Test MFI #5 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy S6 API 23 Smoke Test MFI ) Triggered Atomic Job ( Emulator Galaxy S6 API 23 Smoke Test MFI ) Triggered Atomic Job: Emulator Galaxy S6 API 22 Smoke Test MFI #6 Completed, Result: FAILURE Atomic Job: Emulator Galaxy S6 API 23 Smoke Test MFI #3 Completed, Result: FAILURE Atomic Job ( Turn Non-MFI Development Bridge ON ) Triggered Atomic Job: Emulator Galaxy S6 API 23 Smoke Test MFI #4 Completed, Result: FAILURE Atomic Job: Turn Non-MFI Development Bridge ON #24 Completed, Result: SUCCESS Atomic Job ( Physical Galaxy S6 Smoke Test Non-MFI ) Triggered Atomic Job: Physical Galaxy S6 Smoke Test Non-MFI #13 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy Note 5 API 23 Smoke Test Non-MFI ) Triggered Atomic Job ( Emulator Galaxy Note 5 API 23 Smoke Test Non-MFI ) Triggered Atomic Job: Emulator Galaxy Note 5 API 23 Smoke Test Non-MFI #8 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy S3 API 17 Smoke Test Non-MFI ) Triggered Atomic Job ( Emulator Galaxy S3 API 17 Smoke Test Non-MFI ) Triggered Atomic Job: Emulator Galaxy Note 5 API 23 Smoke Test Non-MFI #9 Completed, Result: FAILURE Atomic Job: Emulator Galaxy S3 API 17 Smoke Test Non-MFI #4 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy S6 API 22 Smoke Test Non-MFI ) Triggered Atomic Job ( Emulator Galaxy S6 API 22 Smoke Test Non-MFI ) Triggered Atomic Job: Emulator Galaxy S3 API 17 Smoke Test Non-MFI #5 Completed, Result: FAILURE Atomic Job: Emulator Galaxy S6 API 22 Smoke Test Non-MFI #3 Completed, Result: FAILURE Atomic Job ( Emulator Galaxy S6 API 23 Smoke Test Non-MFI ) Triggered Atomic Job ( Emulator Galaxy S6 API 23 Smoke Test Non-MFI ) Triggered Atomic Job: Emulator Galaxy S6 API 22 Smoke Test Non-MFI #4 Completed, Result: FAILURE Atomic Job: Emulator Galaxy S6 API 23 Smoke Test Non-MFI #3 Completed, Result: FAILURE Atomic Job ( Turn All Bridges Off ) Triggered Atomic Job: Emulator Galaxy S6 API 23 Smoke Test Non-MFI #4 Completed, Result: FAILURE Atomic Job: Turn All Bridges Off #48 Completed, Result: SUCCESS

[ace-han]

At first sight, it appears that it will trigger the failed job one more time.

Interesting...

[tzedar]

It double triggers before the first job fails. I have 2 executors setup (1 for the coordinator job and 1 for the job that is supposed to be triggered.) When It executes the first emulator job uses 1 executor and 1 job is queued up. I also have retry failed jobs unchecked so it will not retry them.

[tzedar]

This does seem to only happen after a failed job. And it happens for every failed job. This kind of just snow balls into a lot of jobs getting queued up and taking forever to run all our tests. Is there anyway to get a eta on when someone can look into this? Otherwise I am going to have to find another plugin that comes close to accomplishing the same thing.

[ace-han]

@tzedar I could not reproduce your problem. Could you share your execution plan like below in a graph

[tzedar]

[ace-han]

@tzedar I did a copy of your config, still no luck

I am wondering if you are defining any other plugin parameters (NodeLabel Parameter Plugin) on those failure jobs so that it triggers twice...

or the job type is Multi-configuration project?

Otherwise, plz send me you job config $JENKINS_HOME/jobs/ (hope that isn't too big) as a zip, I will take a close look.

or you could pack a mini version config that could reproduce the problem as a zip to me thx.

[tzedar]

I am triggering parameterized jobs for each configuration. So each test has parameters that are passed from the coordinator job. This is completely random in failures. If a smoke test fails then the epic will not run but the next smoke test will trigger twice.

Can this be a configuration thing on my side like an issue where the job is blocking or non-blocking causing the double trigger?

[ace-han]

This bug-like behavior is only reported by you.

I need to identify whether it's a bug from this plugin or from mis-config. Since I could not reproduce on my local pc, I need to take a close look on you config.xml files of your job config like below

Send them to me, I think it's the fastest way to get this issue done

[tzedar]

I am working on setting it up on a smaller scale so I can send you my configuration files... I did notice one difference in our execution plans. The MFI configuration on mine is non-blocking but it is blocking on yours. Could this be the difference and the problem we are facing?

[ace-han]

Yes, I noticed that and retriggered, again no luck

[tzedar]

Is there another blocking job after the Note_5 that would be the job that gets triggered twice. I am still working on a smaller setup at my site but it may take a while.

[ace-han]

@tzedar okay, I could reproduce it now by adding a blocking job after Note_5

[tzedar]

@ace-han I have a smaller configuration setup that has the same issue. Do you still need the config files for that or are you good now since you can reproduce it on your side?

[ace-han]

@tzedar Thx for raising out this bug. It should be fixed in the latest develop branch

and here is package for the fix, you can manually replace it in your jenkins instance

coordinator-1.2.1-SNAPSHOT.jpi.zip

Hope this works.

[tzedar]

So far so good. I tested on my mock up and it fixed the issue. I just installed on the main system and we will see what happens tonight. Thanks for the quick fix.

[ace-han]

This should be fixed already.