Need to put merged keepup files into rucio/metacat

hschellman commented 7 hours ago

Here it is. The files are in bold right below

From: Schellman, Heidi Heidi.Schellman@oregonstate.edu Sent: Tuesday, September 10, 2024 4:29 PM To: Schellman, Heidi Heidi.Schellman@oregonstate.edu Cc: Schellman, Heidi Heidi.Schellman@oregonstate.edu; Steven C Timm timm@fnal.gov; Kirby, Michael 0000243ad0d37178-dmarc-request@LISTSERV.FNAL.GOV; dune-data-mgmt dune-data-mgmt@listserv.fnal.gov Subject: Re: Request to store/rucioize 1 set of merging files

[EXTERNAL] – This message is from an external sender Would it work if I moved it to persistent? We may be testing for a while. I turn in to a professor in 2 weeks.

On Sep 10, 2024, at 3:14 PM, Schellman, Heidi Heidi.Schellman@oregonstate.edu wrote:

Ok, so I have new files to store.

/pnfs/dune/scratch/users/schellma/merging/tostore/run0000028536_000000_20240910152808-local

Is a partial run. Rucio timed out on merge # 600 and I am resubmitting but it would be good for testing resubmission this way.

On Sep 10, 2024, at 2:04 PM, Steven C Timm timm@fnal.gov wrote:

[This email originated from outside of OSU. Use caution with links and attachments.] yes, my apologies it seems that these already dropped out of scratch. also I thought you'd initially made a wiki task on this but I don't see that. If you could please regenerate one set of merged tuples we will try again.

Steve Timm

From: Schellman, Heidi [Heidi.Schellman@oregonstate.edu](mailto:Heidi.Schellman@oregonstate.edu) Sent: Tuesday, September 10, 2024 9:05 AM To: Schellman, Heidi [Heidi.Schellman@oregonstate.edu](mailto:Heidi.Schellman@oregonstate.edu) Cc: Schellman, Heidi [Heidi.Schellman@oregonstate.edu](mailto:Heidi.Schellman@oregonstate.edu); Steven C Timm [timm@fnal.gov](mailto:timm@fnal.gov); Kirby, Michael [0000243ad0d37178-dmarc-request@LISTSERV.FNAL.GOV](mailto:0000243ad0d37178-dmarc-request@LISTSERV.FNAL.GOV); dune-data-mgmt [dune-data-mgmt@listserv.fnal.gov](mailto:dune-data-mgmt@listserv.fnal.gov) Subject: Re: Request to store/rucioize 1 set of merging files

[EXTERNAL] – This message is from an external sender Just a reminder on this. These may have dropped off scratch by now. I would like them stored officially so that I can test checking for duplicates/storage.

From: Schellman, Heidi Heidi.Schellman@oregonstate.edu Sent: Friday, August 23, 2024 11:00 AM To: Steven C Timm timm@fnal.gov Cc: Schellman, Heidi Heidi.Schellman@oregonstate.edu; Kirby, Michael 0000243ad0d37178-dmarc-request@LISTSERV.FNAL.GOV; Schellman, Heidi Heidi.Schellman@oregonstate.edu; dune-data-mgmt dune-data-mgmt@listserv.fnal.gov Subject: Re: Request to store/rucioize 1 set of merging files

[EXTERNAL] – This message is from an external sender

Ok, so can we either store or give me instructions to store? Files in

/pnfs/dune/scratch/users/schellma/merging/tostore/run0000028066_000000_20240821182319-local

On Aug 23, 2024, at 5:54 AM, Steven C Timm timm@fnal.gov wrote:

[This email originated from outside of OSU. Use caution with links and attachments.] Yes at the moment the ntuples and the artroot reco files are in the same namespace. We can use the fact that they are different data tiers to get them into different file families on tape eventually.

Steve

From: owner-dune-data-mgmt@listserv.fnal.gov [owner-dune-data-mgmt@listserv.fnal.gov](mailto:owner-dune-data-mgmt@listserv.fnal.gov) on behalf of Kirby, Michael [0000243ad0d37178-dmarc-request@LISTSERV.FNAL.GOV](mailto:0000243ad0d37178-dmarc-request@LISTSERV.FNAL.GOV) Sent: Friday, August 23, 2024 3:23 AM To: Schellman, Heidi [Heidi.Schellman@oregonstate.edu](mailto:Heidi.Schellman@oregonstate.edu) Cc: dune-data-mgmt [dune-data-mgmt@listserv.fnal.gov](mailto:dune-data-mgmt@listserv.fnal.gov) Subject: Re: Request to store/rucioize 1 set of merging files

[EXTERNAL] – This message is from an external sender

Hi Heidi, et al.,

Are we keeping ntuples and artroot reco files in the same namespace?

Kirby

On Aug 23, 2024, at 01:42, Schellman, Heidi [Heidi.Schellman@oregonstate.edu](mailto:Heidi.Schellman@oregonstate.edu) wrote:

[EXTERNAL] – This message is from an external sender

Can we catalog/store the root files in

/pnfs/dune/scratch/users/schellma/merging/tostore/run0000028066_000000_20240821182319-local

Steve was willing to provide instructions so I could do it.

What namespace should we use for these? Right now it is:

"namespace": "hd-protodune-det-reco”

Inherited from the parent files.

Michael Kirby (he/him/his) Senior Physicist

NPPS/Physics Department Brookhaven National Laboratory mkirby@bnl.gov Cell: +1 630 965 1456

dougbenjamin commented 7 hours ago

Rules created to bring the ATM CAF output files back to FNAL.

dougbenjamin commented 6 hours ago

all ATM CAF output files at DUNE_US_FNAL_DISK_STAGE

StevenCTimm commented 6 hours ago

Doug--note that Heidi was actually referring to files from keepup production in the above e-mail thread, not the CAF files.

StevenCTimm commented 3 hours ago

OK on the files in /pnfs/dune/scratch/users/schellma/merging/tostore/run0000028536_000000_20240910152808-local

I have declared the first 2 to metacat.. the metadata works so it is ok

so to declare the first one for instance the command is:

metacat file declare -f hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000600_lim000050_20240910T211511.root.json hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000600_lim000050_20240910T211511.root dune:all

and this is the response. U2sM5C63R8eaov2f hd-protodune-det-reco hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000600_lim000050_20240910T211511.root

StevenCTimm commented 3 hours ago

now all 8 of those files declared to metacat now have to make a metacat dataset and corresponding rucio dataset Doing all commands manually to figure out what's involved:

[dunepro@duneopsgpvm01 run0000028536_000600_20240910161127-local]$ metacat dataset create hd-protodune-det-reco:hd-protodune-det-reco_merged_tuple_28536 Dataset hd-protodune-det-reco:hd-protodune-det-reco_merged_tuple_28536 cteated with 0 files [dunepro@duneopsgpvm01 run0000028536_000600_20240910161127-local]$ rucio add-dataset hd-protodune-det-reco:hd-protodune-det-reco_merged_tuple_28536 Added hd-protodune-det-reco:hd-protodune-det-reco_merged_tuple_28536 metacat dataset files hd-protodune-det-reco:hd-protodune-det-reco_merged_tuple_28536 hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000950_lim000050_20240910T212327.root hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000750_lim000050_20240910T211759.root hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000700_lim000050_20240910T211656.root hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000850_lim000050_20240910T212117.root hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000900_lim000050_20240910T212216.root hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000600_lim000050_20240910T211511.root hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000800_lim000050_20240910T212024.root hd-protodune-det-reco:hd-protodune_detector_run0000028536_physics_standard_reco_stage2_calibration_protodunehd_keepup_root-tuple-virtual_merged_skip000650_lim000050_20240910T211602.root

StevenCTimm commented 3 hours ago

OK so I did these 8 with rucio upload but this is showing the limitations of the rucio CLI to get things to upload into a dataset, like Justin does, one needs to use the python binding With the CLI I have to insert them into the dataset afterwards and delete the individual file rules that are made.

So my conclusion after working this process is that this is something that could be done by the declad but we will need the new version installed since Marc gave us a feature request to make the data set name generated be more flexible than just scope:scope_run

StevenCTimm commented 3 hours ago

(also rucio upload is damned inefficient)..

hschellman commented 3 hours ago

definitely - for MC run is a very bad idea.

scope:metacatdataset is probably better.

On Sep 19, 2024, at 12:06 PM, Steven Timm @.***> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

OK so I did these 8 with rucio upload but this is showing the limitations of the rucio CLI to get things to upload into a dataset, like Justin does, one needs to use the python binding With the CLI I have to insert them into the dataset afterwards and delete the individual file rules that are made.

So my conclusion after working this process is that this is something that could be done by the declad but we will need the new version installed since Marc gave us a feature request to make the data set name generated be more flexible than just scope:scope_run

— Reply to this email directly, view it on GitHubhttps://github.com/DUNE/data-mgmt-ops/issues/720#issuecomment-2361959903, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA37DMFCU6YD2ESBEEIXGTZXMOCFAVCNFSM6AAAAABOQE7ZVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRRHE2TSOJQGM. You are receiving this because you authored the thread.Message ID: @.***>

StevenCTimm commented 3 hours ago

the rucio and metacat datasets are made now. Both are called hd-protodune-det-reco:hd-protodune-det-reco_merged_tuple_28536 and each contains 8 files, the same 8.

hschellman commented 2 hours ago

Great - I will (once I get back from talking to students, work on marking the files that went into that as merged in metacat).

On Sep 19, 2024, at 12:09 PM, Steven Timm @.***> wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

the rucio and metacat datasets are made now. Both are called hd-protodune-det-reco:hd-protodune-det-reco_merged_tuple_28536 and each contains 8 files, the same 8.

— Reply to this email directly, view it on GitHubhttps://github.com/DUNE/data-mgmt-ops/issues/720#issuecomment-2361976430, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA37DJL7YQ3WP3VAJHYYMDZXMOOXAVCNFSM6AAAAABOQE7ZVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRRHE3TMNBTGA. You are receiving this because you authored the thread.Message ID: @.***>

DUNE / data-mgmt-ops

Need to put merged keepup files into rucio/metacat #720