[JENKINS-50595] build history disappeared after upgrade to 2.107.1

[timja]

After upgrade to 2.107.1 windows from 2.89.3, build history disappeared for jobs that have either been renamed during their life or jobs where the build.xml was updated via file system and configuration reloaded from disk. attached is an example of 3 days of missing build history for a job that was renamed in January.

 

 

 

---

Originally reported by schuchj, imported from: build history disappeared after upgrade to 2.107.1

• status: Open
• priority: Blocker
• resolution: Unresolved
• imported: 2022/01/10

[timja]

schuchj:

Also to note, the jobs that don't show the 3 days worth of build history, per configuration, where running when 'reload configuration from disk' was triggered for updates made to other jobs that where to begin their runtime within a few hours. The jobs updated outside the UI show no history although everthing is on the file system.

[timja]

oleg_nenashev:

Do you see any errors in the System logs? Maybe Build entries refuse to load due to the non-whitelisted classes (JEP-200) or maybe due to a regression with XML 1.1 introduction

[timja]

tfrearson:

Do all of your builds have a build.xml file in them (in the directory on the master)? I'm having a similar issue, and I believe that is the reason they aren't showing up in the build history. I'm not sure why the build.xml file isn't being generated yet though.

[timja]

schuchj:

nothing is being reported in the system logs. appears that the config.xml upgraded to 1.1, i reverted back my installation of Jenkins to the previous version I had on this server. Build history seems to now be 'lost'. decided to purge build history of impacted build jobs and start anew.

there is/are no build.xml files located in root of directory of impacted build jobs. build.xml does exist in build number directories of jobs not impacted by this issue.  how would these build.xml files 'disappear'?

[timja]

tfrearson:

decided to purge build history of impacted build jobs and start anew

Did this work? Do the affected builds now have a build history? I have a few job that are being affected and may have to do the same.

I'm not sure how those build.xml files stopped being created in each build. That's the correlation I noticed though, is that jobs with accurate build history had a build.xml file in each build dir, job with disappearing history didn't have one.

I'm surprised that no one else has reported this issue, not sure what to do next short of re-deploying the build master to see if that fixes it.

[timja]

schuchj:

I'll find out by tomorrow morning. Those jobs are scheduled to run this evening. I'll update tomorrow.  For this update, I had selected their 'upgrade automatically'. Is this how you updated? I'm wondering if it's an issue via their war, instead of using their windows installer package.

[timja]

tfrearson:

going to this latest release, yes. I used the "upgrade automatically". for the last release though, I switched from the weekly builds to the Stable build. I'm wondering if I had an issue somewhere in that process. 

[timja]

schuchj:

build.xml files are still missing when job completes. this is consistently happening to a group of build jobs, where other jobs (similiar jobs) are creating build.xml. Not able to determine why this group of jobs won't/don't create build.xml.

[timja]

schuchj:

creating new jenkins job, using 'old' job as template, is our current work around. build history (i.e., build.xml is present) shows with newly created config.xml.

[timja]

tfrearson:

That's interesting. I had tried that and the build.xml file was present in the first build but not in any subsequent builds. 

I just updated the TAP plugin as I noticed that only builds involving TAP reports were exhibiting this behavior. Unfortunately, the newer versions of the TAP plugin don't work for me so I'm going to downgrade and see if it is still working after that.

[timja]

jlafourc:

Hello,

 

We have exactly the same behavior. I see hold builds which have a build.xml inthe folder. New build folders do not have build.xml. This only happens on build that are generated with jenkins DSL and the seed plugin. 

 

Anyone knows what may be the cause ? 

 

Thx 

 

Julien

[timja]

ams:

We are also experiencing this issue in 2.107.3 as well. Is there any extra information that would be useful in addressing this bug?

[timja]

tfrearson:

ams jlafourc schuchj

I think to fix this issue, I had to install version 2.2.1 of the TAP plugin. I'm not sure if this fixes the issue or just puts a bandaid on things, and it wasn't the ideal solution as upgrading that plugin took away all test information other than pass/fail from my Slack reports. It seems to have kept the build history intact since I did that though, so I guess that's the better of two evils.

ETA: the reason I tried upgrading the TAP Plugin as a fix, is I eventually found that this issue only affected those job that were processing TAP results. Jobs that weren't were never affected by this bug. If you are noticing this happen, check if the jobs have TAP processing, and check your TAP plugin version.

[timja]

noamshochat:

We don't have TAP plugin installed and we also experienced this behaviour 

[timja]

divyanshumehta:

noamshochat
Check your system log you will find some errors 

[timja]

divyanshumehta:

I too face this issue the reason for the build disappearing is that the build.xml for the particular build is not being created at the end of the job. This mainly due to the JEP-200 changes to which several plugin have yet not made the changes. I am also facing this issue the error log for my case is
groovy.log

[timja]

d4rr3ll_sbg:

The solution for us was to uncheck the `Color ANSI Console Output` (xterm) in the Build Environment section.

This was on the only difference between our jobs that lost history and didn't, the overall job was essentially the same.

[timja]

tfrearson:

d4rr3ll_sbg I have that checked too actually but as far as I can tell it doesn't actually do anything for my jobs. I may revert the TAP plugin and uncheck that and see if the build history is still working. Good find there!

schuchj is this plugin being used in jobs affected here?

[timja]

swf:

We have the same issue with Jenkins 2.121.3.

The visible history of all jobs which are created using Job-DSL disappear after some time (next day). The folders already exist on the master and each build folder contains a build.xml. Jobs created using the OrgaScanner did not have this problem.

Please let me know if and which further details required to fix this.

 

[timja]

swf:

Any news here? This is actually a show stopper...

[timja]

swf:

OK, I digged a little bit deeper into this. 

I setup the same automatic job creation we use on another Jenkins instance using bleeding edge Jenkins release. So I have a job, which creates other Jenkinsfile-based pipeline jobs using job dsl. The created jobs have one difference: Some of them have a periodic trigger of an hour, some of them have no such trigger. So the first portion of the created jobs run every hour and the other ones must be triggered real manually.

Now the first interesting part: The periodic triggered jobs have no problem with disappearing history. All fine here. But the manually triggered jobs loose their history on the UI at least on the next day. Actually I did not figured out when exactly the history disappears from the UI.

And the 2nd interesting part is, that the whole problem disappears after I uninstalled the JobConfigHistoryPlugin! Can anybody verify this? 

[timja]

jlafourc:

We found out some of the problems where due to security hardening introduced in version 2.107.1. Everything is explained here :

https://jenkins.io/blog/2018/03/15/jep-200-lts/ 

You can find some useful information in the logs in order to know which classes you need to whitelist (although it's not a very good practice ...)