Release with newest LHAPDF for W mass sample production

[kdlong]

Dear experts,

We are investigated a workflow to manage the production of very large samples (~2 B events) for the W mass analysis. We understand that computing has some resources available to test running the GEN step on HPC resources. The resources are available now and are just waiting on us to have a workflow ready. From our side, we have gridpacks produced and validated, but we would very much like to take advantage of the most modern PDF sets that have just been released. This would require a new CMSSW release. The campaign (UL production) runs in CMSSW_10_6_20, but we're able to use CMSSW_12_X for the gridpack production, provided we work around the constraints of scram (https://cms-talk.web.cern.ch/t/problem-setting-up-cmssw-12-2-after-cmssw-10-6/9117/4)

In this sense, we are not super tied to the CMSSW version. To have the newest sets, we just need a release built with this PR merged (https://github.com/cms-sw/cmsdist/pull/7776) and that is guaranteed to be accessible at all sites and available long term for future production. Since the resources from computing are time sensitive, we would also want whatever be completed the most quickly. Does this make sense for a new 12_3 release, or a 12_4 pre-release? In particular, do pre releases satisfy the availability requirement?

Thanks,

Kenneth

Tagging people already involved in the discussion: @bendavid @mseidel42 @tyjyang @qliphy @jordan-martins

[cmsbuild]

A new Issue was created by @kdlong Kenneth Long.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign generators, core

[cmsbuild]

New categories assigned: core,generators

@mkirsano,@Dr15Jones,@smuzaffar,@alberto-sanchez,@SiewYan,@makortel,@GurpreetSinghChahal,@Saptaparna you have been requested to review this Pull request/Issue and eventually sign? Thanks

[qliphy]

In this sense, we are not super tied to the CMSSW version. To have the newest sets, we just need a release built with this PR merged (cms-sw/cmsdist#7776) and that is guaranteed to be accessible at all sites and available long term for future production. Since the resources from computing are time sensitive, we would also want whatever be completed the most quickly. Does this make sense for a new 12_3 release, or a 12_4 pre-release? In particular, do pre releases satisfy the availability requirement?

If urgently needed, ORP (@perrotta and me) can build 12_4_0_pre3 (originally scheduled on April/19) ahead this week. It should be distributed to and accessible via cvmfs. @smuzaffar and others may comment more on your requirement, i.e. "...accessible at all sites and available long term for future production"

[smuzaffar]

I think all cms sites get cmssw releases from CVMFS. So as soon as we deploy it under /cvmfs/cms.cern.ch then it should be available on all sites. As we never have deleted any thing from /cvmfs/cms.cern.ch, so it should remain available for long time.

[davidlange6]

But we should continue not using pre-releases in production... (otherwise what's the point in the naming scheme)

On Apr 11, 2022, at 4:33 PM, Malik Shahzad Muzaffar @.***> wrote:

I think all cms sites get cmssw releases from CVMFS. So as soon as we deploy it under /cvmfs/cms.cern.ch then it should be available on all sites. As we never have deleted any thing from /cvmfs/cms.cern.ch, so it should remain available for long time.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

[smuzaffar]

right @davidlange6 , we should not use pre-releases for production.

[bendavid]

Well in this case it's only the environment + externals which would be used, since the gridpack+LHE production doesn't actually link or call any CMSSW code...

[smuzaffar]

I would still suggest to backport https://github.com/cms-sw/cmsdist/pull/7776 to 12.3.X and build a new release. I know , in past production system was using https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML to get the valid releases and pre-releases are not shown there by default.

[bendavid]

The use of the alternative environment inside the gridpack is transparent to the production system (for better or for worse).

[davidlange6]

Ha ha - yes, indeed... definitely for the worse:)

On Apr 11, 2022, at 7:12 PM, Josh Bendavid @.***> wrote:

The use of the alternative environment inside the gridpack is transparent to the production system (for better or for worse).

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[kdlong]

From our side we're equally happy with CMSSW_12_3_X. Is the timescale to have a new release longer, though?

[qliphy]

From our side we're equally happy with CMSSW_12_3_X. Is the timescale to have a new release longer, though?

@kdlong CMSSW_12_3_0 was built last weekend. Now if https://github.com/cms-sw/cmsdist/pull/7776 gets merged and backported to 12_3_X, a new release CMSSW_12_3_1 can be built this week (with only difference from 12_3_0 on this LHAPDF change) if there is no objection.

[kdlong]

@qliphy sorry for the delay, it seems that the backport is made and accepted, from our standpoint having the CMSSW_12_3_1 release with it would be great.

[qliphy]

@kdlong Right, it is almost ready (also needs to wait for the IB test results). Will try to launch the build soon today or tomorrow.

[kdlong]

Thanks Qiang!

[qliphy]

@kdlong 12_3_1 is now ready.

[kdlong]

Hi all, is it possible to have this back ported to CMSSW_10_2? We would like to have a production for the low pileup which would use slc6.

[mkirsano]

Hello, I will look into this

[qliphy]

@kdlong 10_2_29 is now ready https://github.com/cms-sw/cmssw/releases/CMSSW_10_2_29

[kdlong]

Excellent, thanks a lot @qliphy and @mkirsano!

[kdlong]

Sorry, last request I promise, but if it's not too much trouble, having this back ported to 10_6 would be very useful for the Nano production. Is it possible? Thanks!

[kdlong]

hi all, just realised this never materialised. Is there any plans to have a new 10_6 release? Having the PDF sets updated in the CMSSW version used for the Nano would be very helpful.

[perrotta]

hi all, just realised this never materialised. Is there any plans to have a new 10_6 release? Having the PDF sets updated in the CMSSW version used for the Nano would be very helpful.

@kdlong we plan a 10_6 release kind of tomorrow: too early for another PR not yet submitted- As usual, I'd suggest you to prepare the backport, if you need it, and try to arrange with @cms-sw/generators-l2 to decide with them about a possible merging in a forthcoming release

[menglu21]

@kdlong is it urgent for ongoing production, but anyway it seems too tight for the new release which will happen today, please prepare the backport and we will work on it before next release plan

[kdlong]

Hi @menglu21 it's not urgent since this is mostly for making our private production much less error prone. But I think it would be good to have, since my understanding is it isn't a lot of effort. It requires a PR to cmsdist. In the past the gen integration experts (Mikhail) took care of this

[lviliani]

Can we close this one?

[cmsbuild]

cms-bot internal usage