Build history destroyed with `7.0.0` upgrade: builds marked "1 Jan. 1970" and stuck

[dduportal]

Describe the bug

The 2 May 2024, the Jenkins infrastructure team upgraded the plugin datadog from 6.0.3 to 7.0.0 on ci.jenkins.io, the public controller used for the CI of Jenkins plugins.

This plugin upgrade resulted in all build history being partially destroyed: see https://github.com/jenkins-infra/helpdesk/issues/4080

The top-level symptoms are:

• Builds have a date set to "1 Jan. 1970"
• A LOT of builds, despite being finished since hours or days, are marked as "running" and are stuck: any tentative to stop it through the UI do not do anything (except showing a NullPointerException in the log with the job id).
• On each restart/reload of the controller, logs are spammed by XMLParsing errors (see https://github.com/jenkins-infra/helpdesk/issues/4080)
• The Jenkins UI shows a LOT (ci.jenkins.io cannot even shows all) of "Unreadable Data":

CannotResolveClassException: org.datadog.jenkins.plugins.datadog.traces.StepDataAction, CannotResolveClassException: org.datadog.jenkins.plugins.datadog.traces.StepTraceDataAction, CannotResolveClassException: org.datadog.jenkins.plugins.datadog.model.StageBreakdownAction, CannotResolveClassException: org.datadog.jenkins.plugins.datadog.model.CIGlobalTagsAction, CannotResolveClassException: org.datadog.jenkins.plugins.datadog.traces.IsPipelineAction

The problem is reported by other users in https://issues.jenkins.io/browse/JENKINS-66328?focusedId=446989&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-446989

To Reproduce

• Start a controller with LTS (2.440.3) and install datadog version 6.0.3
• Run a set of builds to generate an history
• Upgrade the plugin to 7.0.0
• Almost everytime (was able to reproduce 2 times on 3 retries) the symptoms appears again

Expected behavior

I would like the datadog plugin to either upgrade without any problem OR to not destroy my entire build history.

Screenshots

Environment and Versions (please complete the following information):

A clear and precise description of your setup:

• Controller running in a Linux Docker container based on the image jenkins/jenkins-lts-jdk17 (ref. https://github.com/jenkins-infra/jenkins-infra/blob/be148041dca2a13335af5260bae59198b16d9059/hieradata/clients/controller.sponsorship.ci.jenkins.io.yaml#L33-L34).
• Jenkins version is the current LTS: 2.440.3
• Installed plugins: https://github.com/jenkins-infra/jenkins-infra/blob/be148041dca2a13335af5260bae59198b16d9059/hieradata/clients/controller.sponsorship.ci.jenkins.io.yaml#L562-L604

Additional context

Same issue as https://github.com/jenkinsci/datadog-plugin/issues/393.

We are considering to stop using datadog plugin on the official Jenkins infrastructure because our build history is destroyed on almost each plugin upgrade.

The reason is that the build.xml files are modified by the plugin which makes them unparseable if there are any error or if we remove the plugin. There might be reasons to do the implementation that way but it start to really be a problem for administrators.

[legoGoat]

I was literally about to submit a post on this exact issue, too. 🙈

@dduportal Do you know more or less from what the date/time your oldest affected builds are? I'm trying to judge whether it is since a particular timestamp or over a particular period of time since now().

Because, honestly, it clears out the entire execution of the build, all its run data. This is horrible.

[nikita-tkachenko-datadog]

Hi @dduportal, I am very sorry this happened again. We are still looking into the root cause of this. In the meantime, would it be possible for you to provide the logs of your Jenkins' controller node? Thanks!

[nikita-tkachenko-datadog]

The 7.0.0 release of the plugin has been removed from the Update Center

[mjuszczak]

Also experienced this issue when updating Jenkins core + all plugins on Tues Apr 30th.

[nikita-tkachenko-datadog]

The issue has been fixed in https://github.com/jenkinsci/datadog-plugin/pull/425 released in v7.0.1

[dduportal]

Hi @dduportal, I am very sorry this happened again. We are still looking into the root cause of this. In the meantime, would it be possible for you to provide the logs of your Jenkins' controller node? Thanks!

Sorry I missed your comment. We don't have more than 2 days of retention today but I can check the backups if you are still interested (asking given #425).

[dduportal]

The issue has been fixed in #425 released in v7.0.1

Thanks for the fix!

[nikita-tkachenko-datadog]

I can check the backups if you are still interested

Thank you, I've managed to reproduce the issue locally so I don't need the logs anymore

[dduportal]

Closing the issue as fixed by 7.0.1, thanks!