search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.33k forks source link

Build CMSSW_12_4_1 #38550

Closed qliphy closed 2 years ago

qliphy commented 2 years ago

Following the plan as discussed at today's joint Ops meeting: https://indico.cern.ch/event/1176335/

cmsbuild commented 2 years ago

Request received. I will start to build the release after one of the following approve the issue: @perrotta, @smuzaffar, @qliphy, @dpiparo. You can do this by writing "+1" in a comment. You can also ask me to begin to build cmssw-tool-conf first ( Cannot be done for patch releases ). To do this write "build cmssw-tool-conf" in a comment. I will start to build cmssw-tool-conf and then wait for the "+1" to start the build of the release. CMSSW Branch: CMSSW_12_4_X Architecture: el8_amd64_gcc10(Production), slc7_amd64_gcc10, el8_aarch64_gcc10, el8_ppc64le_gcc10

qliphy commented 2 years ago

+1

cmsbuild commented 2 years ago

Release created: https://github.com/cms-sw/cmssw/releases/tag/CMSSW_12_4_1. The tag was created on top of branch: https://github.com/cms-sw/cmssw/tree/CMSSW_12_4_X

cmsbuild commented 2 years ago

Queuing Jenkins build for the following architectures: el8_ppc64le_gcc10, el8_amd64_gcc10, slc7_amd64_gcc10, el8_aarch64_gcc10 You can abort the build by writing "Abort" in a comment. I will delete the release, the cmssw and cmsdist tag, and close the issue. You can't abort the upload once at least one achitecture is being uploaded. If you are building cmssw-tool-conf first, I will wait for each architecture to finish to start the build of cmssw.

cmsbuild commented 2 years ago

The build has started for el8_amd64_gcc10 in cmsbuild01. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/build-release/3555/console CMSDIST Branch: IB/CMSSW_12_4_X/master, PKGTOOLS Branch: V00-34-XX

cmsbuild commented 2 years ago

The build has started for slc7_amd64_gcc10 in cmsbuild02. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/build-release/3556/console CMSDIST Branch: IB/CMSSW_12_4_X/master, PKGTOOLS Branch: V00-34-XX

cmsbuild commented 2 years ago

The build has started for el8_ppc64le_gcc10 in ibmminsky-1a. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/build-release/3557/console CMSDIST Branch: IB/CMSSW_12_4_X/master, PKGTOOLS Branch: V00-34-XX

cmsbuild commented 2 years ago

The build has started for el8_aarch64_gcc10 in olarm-99-102b. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/build-release/3558/console CMSDIST Branch: IB/CMSSW_12_4_X/master, PKGTOOLS Branch: V00-34-XX

cmsbuild commented 2 years ago

The build has finished sucessfully for the architecture el8_ppc64le_gcc10 and is ready to be uploaded. You can start the uploads by writing the comment: "upload all". I will upload all the architectures as soon as the build finishes successfully. You can see the log for the build here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-el8_ppc64le_gcc10/3557 Some tests ( runTheMatrix.py -s ) are being run, the results will be posted when done.

cmsbuild commented 2 years ago

The tests have finished for el8_ppc64le_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-el8_ppc64le_gcc10/3557

Tests results: 49 48 46 35 18 4 1 1 1 1 1 tests passed, 0 0 0 0 0 0 0 0 0 0 0 failed

cmsbuild commented 2 years ago

The build has finished sucessfully for the architecture slc7_amd64_gcc10 and is ready to be uploaded. You can start the uploads by writing the comment: "upload all". I will upload all the architectures as soon as the build finishes successfully. You can see the log for the build here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-slc7_amd64_gcc10/3556 Some tests ( runTheMatrix.py -s ) are being run, the results will be posted when done.

cmsbuild commented 2 years ago

The build has finished sucessfully for the architecture el8_amd64_gcc10 and is ready to be uploaded. You can start the uploads by writing the comment: "upload all". I will upload all the architectures as soon as the build finishes successfully. You can see the log for the build here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-el8_amd64_gcc10/3555 Some tests ( runTheMatrix.py -s ) are being run, the results will be posted when done.

cmsbuild commented 2 years ago

The tests have finished for slc7_amd64_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-slc7_amd64_gcc10/3556

Tests results: 49 48 46 35 18 4 1 1 1 1 1 tests passed, 0 0 0 0 0 0 0 0 0 0 0 failed

cmsbuild commented 2 years ago

The tests have finished for el8_amd64_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-el8_amd64_gcc10/3555

Tests results: 49 48 46 35 18 4 1 1 1 1 1 tests passed, 0 0 0 0 0 0 0 0 0 0 0 failed

perrotta commented 2 years ago

@smuzaffar build for el8_aarch64_gcc10 seems stuck since several hours: is there anything that can be done for it?

cmsbuild commented 2 years ago

The build has started for el8_aarch64_gcc10 in olarm-202a. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/build-release/3559/console CMSDIST Branch: IB/CMSSW_12_4_X/master, PKGTOOLS Branch: V00-34-XX

perrotta commented 2 years ago

Still stuck in the same point as this afternoon... If it does not converge by tomorrow we will upload all other architectures, letting el8_aarch64_gcc10 aside

smuzaffar commented 2 years ago

not really, it is now building cmssw which can take couple of hours. previously job failed as the nodes where it was running was disconnected from jenkins

Caused: java.io.IOException: Unexpected termination of the channel
  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)
  Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@766fba9c:olarm-99-102b": Remote call on olarm-99-102b failed. The channel is closing down or has closed down
perrotta commented 2 years ago

Ah, ok, great: thank you @smuzaffar! The log was stuck several hours in the same place also during the previous attempt... but let hope it can finish succesfully now

smuzaffar commented 2 years ago

cmssw buildis nearly finished , all 2450 libs are built. I think it will take half an hour more before it is ready to be uploaded

smuzaffar commented 2 years ago

we have 2 arm nodes. one is very slow where cmssw build can easily take 8 to 10 hours ( depending on how many externals we build). Previous build was running on slow node. Anyway, now it is running on the better node and build should be ready in half an hour to upload

smuzaffar commented 2 years ago

Ah, ok, great: thank you @smuzaffar! The log was stuck several hours in the same place also during the previous attempt... but let hope it can finish succesfully now

the build logs go to a log file that is why those are not visible in jenkins.

perrotta commented 2 years ago

Thank you again for the details!

smuzaffar commented 2 years ago

thanks to @iarspider who took care of failed release build process.

cmsbuild commented 2 years ago

The build has finished sucessfully for the architecture el8_aarch64_gcc10 and is ready to be uploaded. You can start the uploads by writing the comment: "upload all". I will upload all the architectures as soon as the build finishes successfully. You can see the log for the build here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-el8_aarch64_gcc10/3559 Some tests ( runTheMatrix.py -s ) are being run, the results will be posted when done.

perrotta commented 2 years ago

upload all

cmsbuild commented 2 years ago

Queing Jenkins upload for slc7_amd64_gcc10

cmsbuild commented 2 years ago

Queing Jenkins upload for el8_amd64_gcc10

cmsbuild commented 2 years ago

Queing Jenkins upload for el8_ppc64le_gcc10

cmsbuild commented 2 years ago

Queing Jenkins upload for el8_aarch64_gcc10

cmsbuild commented 2 years ago

The upload has started for el8_aarch64_gcc10 in olarm-202. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/upload-release/2398/console

cmsbuild commented 2 years ago

The upload has successfully finished for el8_aarch64_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-upload-release/CMSSW_12_4_1-el8_aarch64_gcc10/2398

cmsbuild commented 2 years ago

The upload has started for el8_amd64_gcc10 in cmsbuild01. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/upload-release/2399/console

cmsbuild commented 2 years ago

The upload has started for el8_ppc64le_gcc10 in ibmminsky-1. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/upload-release/2400/console

cmsbuild commented 2 years ago

The upload has started for slc7_amd64_gcc10 in cmsbuild02. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/upload-release/2401/console

cmsbuild commented 2 years ago

The upload has successfully finished for el8_ppc64le_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-upload-release/CMSSW_12_4_1-el8_ppc64le_gcc10/2400

cmsbuild commented 2 years ago

The upload has successfully finished for slc7_amd64_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-upload-release/CMSSW_12_4_1-slc7_amd64_gcc10/2401

cmsbuild commented 2 years ago

The upload has successfully finished for el8_amd64_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-upload-release/CMSSW_12_4_1-el8_amd64_gcc10/2399

perrotta commented 2 years ago

release-notes

cmsbuild commented 2 years ago

Generating release notes since CMSSW_12_4_0. You can see the progress here: https://cmssdt.cern.ch/jenkins/job/release-produce-changelog/ I will generate an announcement template.

cmsbuild commented 2 years ago

You can use this template for announcing the release:

Hi all,

The production release CMSSW_12_4_1 is now available for the following architectures:

el8_amd64_gcc10 (production) el8_ppc64le_gcc10 slc7_amd64_gcc10 el8_aarch64_gcc10

The release notes of what changed with respect to CMSSW_12_4_0 can be found at:

https://github.com/cms-sw/cmssw/releases/CMSSW_12_4_1

Following the plan as discussed at today's joint Ops meeting: https://indico.cern.ch/event/1176335/

Cheers, cms-bot

You can also click here to send the email.

cmsbuild commented 2 years ago

The release notes are ready: https://github.com/cms-sw/cmssw/releases/tag/CMSSW_12_4_1

cmsbuild commented 2 years ago

The tests have finished for el8_aarch64_gcc10 You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/auto-build-release/CMSSW_12_4_1-el8_aarch64_gcc10/3559

Tests results: 49 48 46 35 18 4 1 1 1 1 1 tests passed, 0 0 0 0 0 0 0 0 0 0 0 failed

cmsbuild commented 2 years ago

The cleanup has started for el8_aarch64_gcc10

cmsbuild commented 2 years ago

The cleanup has started for slc7_amd64_gcc10

cmsbuild commented 2 years ago

The cleanup has started for el8_amd64_gcc10

cmsbuild commented 2 years ago

The cleanup has started for el8_ppc64le_gcc10

cmsbuild commented 2 years ago

The workspace for el8_ppc64le_gcc10 has been deleted You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/cleanup-auto-build/CMSSW_12_4_1-el8_ppc64le_gcc10/2107

cmsbuild commented 2 years ago

The workspace for el8_aarch64_gcc10 has been deleted You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/cleanup-auto-build/CMSSW_12_4_1-el8_aarch64_gcc10/2105

cmsbuild commented 2 years ago

The workspace for el8_amd64_gcc10 has been deleted You can see the log here: https://cmssdt.cern.ch/SDT/jenkins-artifacts/cleanup-auto-build/CMSSW_12_4_1-el8_amd64_gcc10/2106