Development musing

[sophiemiddleton]

During a conversation with @dnbrow01 we thought of an idea for testing code or mid-development running. That is to have a musing MDC2020... where ... is to be decided. This would always be a testing musing useful only to ensure everything is working. This avoid need to modify script paths or issues when changing primary fcl files.

[rlcee]

Currently the process of building and deploying musings in the official musing location requires tags, but it would be straightforward to include an option to build the head and call it "dev" or something like that.

But there is already something like this now. At every merge of Offline, we build the head of Production and Offline (i.e. make SimJob out of the head), and put that on the development cvmfs. We currently keep the last ~10 of these and purge older ones. You can setup the latest: muse setup HEAD or chose a specific build for more stability: muse setup main/412ec771 where "main" is the branch and the hash is the git hash of the head commit at the time. See ls -lt /cvmfs/mu2e-development.opensciencegrid.org/museCIBuild/main

[kutschke]

Do you always want both the head of Offline and the head of Production in this? Or will you sometimes want an older Offline with the head of Production?

[dnbrow01]

This was likely a conversation with Dave from LBL, who I copy here. I'm on the road, otherwise I would use his GitHub tag to include him - sorry.

Best Regards, Dave (KY) Brown

On May 19, 2023, at 7:46 AM, Sophie Middleton @.***> wrote:

CAUTION: This email originated from outside of our organization. Do not click links, open attachments, or respond unless you recognize the sender's email address and know the contents are safe.

During a conversation with @dnbrow01 we thought of an idea for testing code or mid-development running. That is to have a musing MDC2020... where ... is to be decided. This would always be a testing musing useful only to ensure everything is working. This avoid need to modify script paths or issues when changing primary fcl files. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[brownd1978]

my github id is brownd1978

Dave

On Fri, May 19, 2023 at 5:46 AM Sophie Middleton @.***> wrote:

During a conversation with @dnbrow01 https://github.com/dnbrow01 we thought of an idea for testing code or mid-development running. That is to have a musing MDC2020... where ... is to be decided. This would always be a testing musing useful only to ensure everything is working. This avoid need to modify script paths or issues when changing primary fcl files.

— Reply to this email directly, view it on GitHub https://github.com/Mu2e/Production/issues/256, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAH5763GHOKJBNYXUXS3H3XG5TRDANCNFSM6AAAAAAYHXUJTM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- David Brown @.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[sophiemiddleton]

I think it would be HEAD of Production, the Offline version could be the last official release though or HEAD, its mostly for Production development

[sophiemiddleton]

@rlcee can you make one of these development SimJobs for the current HEAD of production? And let me if I can also do that

[rlcee]

I think the process of building Musings on Jenkins, based on a record in MuseConfig, is all about having a specific audit trail of what was build. In this case I'm assuming there is no need for a permanent audit trail, so the Jenkins build isn't really necessary. If you agree, then it is pretty simple. You can create you own build area, checkout Production and set backing to the Offline you want. Make your mods then build and run "muse tarball". The only trick it is use "-r " here. That forces the tarball to be in release format, which is ready to install in Musings. I think using your own build area will speed up your development cycle. I'd suggesting thinking about how to indicate to users, who will see these releases at the top of their list, that this is not permanent. Maybe naming verisons like "MDC2020ydev" or temp, dev, scratch something so it is clear. I'd also like to ask you to clean these up when you are done. If you want even faster development, once you have your dev release on cvmfs, you can start a cvmfs transaction and just edit the files there, and publish, so for tweaks you don't even need the build step.

[kutschke]

For a name, how about rc0, rc1 etc for release candidate?

From: Ray Culbertson @.> Reply-To: Mu2e/Production @.> Date: Monday, May 22, 2023 at 9:15 AM To: Mu2e/Production @.> Cc: Rob Kutschke @.>, Comment @.***> Subject: Re: [Mu2e/Production] Development musing (Issue #256)

I think the process of building Musings on Jenkins, based on a record in MuseConfig, is all about having a specific audit trail of what was build. In this case I'm assuming there is no need for a permanent audit trail, so the Jenkins build isn't really necessary. If you agree, then it is pretty simple. You can create you own build area, checkout Production and set backing to the Offline you want. Make your mods then build and run "muse tarball". The only trick it is use "-r " here. That forces the tarball to be in release format, which is ready to install in Musings. I think using your own build area will speed up your development cycle. I'd suggesting thinking about how to indicate to users, who will see these releases at the top of their list, that this is not permanent. Maybe naming verisons like "MDC2020ydev" or temp, dev, scratch something so it is clear. I'd also like to ask you to clean these up when you are done. If you want even faster development, once you have your dev release on cvmfs, you can start a cvmfs transaction and just edit the files there, and publish, so for tweaks you don't even need the build step.

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Mu2e_Production_issues_256-23issuecomment-2D1557301647&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=Zi5IH-k2_JiK_Wipiv3pwwSXO9UNaHiLP8sisfPz__k&m=iZba_-2265Ic1pVwfcj2-BJJoSKWbfhhDnYahI0z0Nq6PsAzW6uId-4eY9QMlC6p&s=omsCY3ORVP80lFEGijxwte3xQ8wK4YS8TEMllHft5EI&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABHY5ZSC4HYPPVOIPZAL7WLXHNYITANCNFSM6AAAAAAYHXUJTM&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=Zi5IH-k2_JiK_Wipiv3pwwSXO9UNaHiLP8sisfPz__k&m=iZba_-2265Ic1pVwfcj2-BJJoSKWbfhhDnYahI0z0Nq6PsAzW6uId-4eY9QMlC6p&s=7_FTmVyXw-6xfT7y7d_FBedF-CzhoDj0Dc4QHbdMv4Y&e=. You are receiving this because you commented.Message ID: @.***>

[kutschke]

I would go through the extra step of always doing the work in the working clone rather than editing cvmfs in place. It does add a little overhead but it removes one possibility to forget to add something back to the working clone.

[brownd1978]

I agree that having everything in cvmfs is a better test. Since Offline is backing it should be fast to create these with just a new production.

On Mon, May 22, 2023 at 07:41 Rob Kutschke @.***> wrote:

I would go through the extra step of always doing the work in the working clone rather than editing cvmfs in place. It does add a little overhead but it removes one possibility to forget to add something back to the working clone.

— Reply to this email directly, view it on GitHub https://github.com/Mu2e/Production/issues/256#issuecomment-1557343995, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAH575JRX4DLDDICFHXW7LXHN3IJANCNFSM6AAAAAAYHXUJTM . You are receiving this because you commented.Message ID: @.***>

-- David Brown @.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[kutschke]

Clarifying me earlier content. If you are working towards MDC2020y, then the name of the first release candidate would be MDC2020y_rc0.

[kutschke]

A common progression is that MDC2020y would be a tag on the same commit as the final rc.

[brownd1978]

Can we implement this so that the cfg can refer to MDC2020y, not MDC2020y_rc0? Otherwise we’ll always need a final name change commit and tag before starting the official production.

On Mon, May 22, 2023 at 07:52 Rob Kutschke @.***> wrote:

A common progression is that MDC2020y would be a tag on the same commit as the final rc.

— Reply to this email directly, view it on GitHub https://github.com/Mu2e/Production/issues/256#issuecomment-1557363588, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAH576NNVECMSPOBIAQNTDXHN4TRANCNFSM6AAAAAAYHXUJTM . You are receiving this because you commented.Message ID: @.***>

-- David Brown @.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[kutschke]

Can we implement this so that the cfg can refer to MDC2020y, not MDC2020y_rc0? Otherwise we’ll always need a final name change commit and tag before starting the official production.

Offhand, I see two ways to do this. One is that, during development, we maintain MDC2020y as a link to the most recent rc.

Another is to embed logic in the POMS scripts that says: if MDC2020 exists use it; if it does not use the highest numbered rc; if there are no rc's, fail.

Does anyone see a way to automate the former? Other ideas?

[rlcee]

There are no if's in cfg. The cfg now has the SimJob version separate from the config field of the dataset names, so that could help (depending). Then you can play games with POMS production versus test submits, combined with variable precedence rules and include files, resulting in overrides or redirection. I am using some of this in pass1. I was eventually going to apply some of this to sim recommendations. If you want to hurry up and do that now, I'd probably want to take a day or so this develop the ideas into a plan. Another approach is to wrench the action out of the cfg and into bash scripts. But really I'm not sure of the exact problem we're solving - it might actually be fairly easy. One thing I think is wrong is to present MDC2020y in the published location, but not in the final form.

[sophiemiddleton]

I think just having MDC2020y as the current development, until we annouce a "release" is fine. Since this is just for testing and any datasets will be removed and retired once we know everything is working

[rlcee]

The point on using MDC2020y is that you have a temporary release in the place of, and with the name of, a permanent release. Any user just taking a look will confuse it for a published release. It looks like the SimJob name is separate from the dataset names in the cfg, so you could just point the SimJob to MDC2020y_rc0 for tests, and make that one change, one place, for production.

[kutschke]

I second Ray's comment. I don't like the idea of MDC2020y being time dependent. Someone could choose to run with it when it's still in test and their output will not have a robust audit trail. Has anyone asked Vito/Lucas/Marc if they have encountered this before and if they have a solution?

[rlcee]

I think there are some solutions, especially the "test submission" feature of POMS. Basically, you put everything you want for the final y configuration in rc0, then point the test submissions to a test cfg file, that test cfg includes the production cfg and overrides the SimJob setup and the output location. The only issue is that this means going over the cfg carefully and will probably require a few iterations, which will take some time. But then at least this factorization of test and production is complete. A downside is doing this under pressure will probably lead to a situation which will need more tweaking in the future, i.e. it is not a thoughtful design from the ground up. Also I may be forgetting some crucial detail that would derail this plan.

[brownd1978]

The key point is there should exactly one place to change production from using X_rc0 to X. More than that and you can’t really test it. This place would preferably live outside any repo, though that might not be possible today. Once fully tested, X_rc0 should ‘become’ X.

On Tue, May 23, 2023 at 12:31 Ray Culbertson @.***> wrote:

I think there are some solutions, especially the "test submission" feature of POMS. Basically, you put everything you want for the final y configuration in rc0, then point the test submissions to a test cfg file, that test cfg includes the production cfg and overrides the SimJob setup and the output location. The only issue is that this means going over the cfg carefully and will probably require a few iterations, which will take some time. But then at least this factorization of test and production is complete. A downside is doing this under pressure will probably lead to a situation which will need more tweaking in the future, i.e. it is not a thoughtful design from the ground up. Also I may be forgetting some crucial detail that would derail this plan.

— Reply to this email directly, view it on GitHub https://github.com/Mu2e/Production/issues/256#issuecomment-1560013954, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAH573AEELTBIKYURTAABDXHUGARANCNFSM6AAAAAAYHXUJTM . You are receiving this because you commented.Message ID: @.***>

-- David Brown @.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[rlcee]

Another solution is put these two details directly in the submission parameters, so you set up the cfg so that it requires the POMS process to define SIMJOB_CODE_VERSION and OUTPUT_DISK, then you send different parameters with production and test submissions.

[brownd1978]

I like that option. It’s consistent either the way we’ve setup the bash scripts.

On Tue, May 23, 2023 at 12:40 Ray Culbertson @.***> wrote:

Another solution is put these two details directly in the submission parameters, so you set up the cfg so that it requires the POMS process to define SIMJOB_CODE_VERSION and OUTPUT_DISK, then you send different parameters with production and test submissions.

— Reply to this email directly, view it on GitHub https://github.com/Mu2e/Production/issues/256#issuecomment-1560023674, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAH572JWW6H5WHHIS5UDC3XHUHBFANCNFSM6AAAAAAYHXUJTM . You are receiving this because you commented.Message ID: @.***>

-- David Brown @.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[sophiemiddleton]

Hi Ray,

I think I like your suggestion ,can you move forward to implementing it. We need to use it for the FlatPhoton campaign because we need to override some primary filters

[rlcee]

I think it is easy for you to make the rc0 release, and if it is in your hands you can control a faster turnaround. The sequence is

setup mu2e muse backing Offline v10_22_01 git clone Production edits muse tarball -r SimJob/MDC2020z_rc0 then take the tarball and unroll it in /cvmfs/../Musings

more edits can be done in the same area, retar, and just unroll the newest tarball on top of the existing Musing (or make rc1).

Then for setting the parameters in POMS, I think all you need to do is instead of setting release_v_o in the cfg, comment that out (not strictly necessary, just for clarity) and set it to "z_rc0" in the parameter list in the POMS test submission. We can talk on slack or email if you have questions about any of the details.

[sophiemiddleton]

OK, I'll get to this tomorrow first thing

[sophiemiddleton]

Hi Ray, I made the change and am trying to move the tarball to the Musing/SimJob directory while logged in as mu2epro:

[mu2epro@mu2egpvm01 SimJob]$ cp /mu2e/data/users/sophie/museTarball/tmp.yxWFRS6728/SimJob-MDC2020z_rc0-sl7-prof-e20-p040.tar.bz2 . cp: cannot create regular file ‘./SimJob-MDC2020z_rc0-sl7-prof-e20-p040.tar.bz2’: Read-only file system

[rlcee]

The mu2epro kerberos identities are not in the cvmfs account k5login. We could add them. For the moment, you can scp in from your account or scp from inside the cvmfs account out to mu2epro, using your kerberos identity.

[sophiemiddleton]

I tried: [mu2epro@mu2egpvm01 SimJob]$ scp sophie@mu2egpvm02.fnal.gov:/mu2e/data/users/sophie/museTarball/tmp.yxWFRS6728/SimJob-MDC2020z_rc0-sl7-prof-e20-p040.tar.bz2 . Permission denied (gssapi-keyex,gssapi-with-mic).

and same while logged in as "sophie"

[rlcee]

Are you using the cvmfs upload procedure, as here? ssh -X cvmfsmu2e@oasiscfs.fnal.go cvmfs_server transaction mu2e.opensciencegrid.org cd /cvmfs/mu2e.opensciencegrid.org/Musing scp sophie@mu2egpvm02.fnal.gov:/mu2e/data/users/sophie/museTarball/tmp.yxWFRS6728/SimJob-MDC2020z_rc0-sl7-prof-e20-p040.tar.bz2 . untar .bz2 rm .bz2 cd cvmfs_server publish mu2e.opensciencegrid.org

[sophiemiddleton]

I was not using that procedure. It worked now