SBNDataPool1 clean-up

[mt82]

Dear all, Here is the link to the spreadsheet with the list of the disk paths and their size.

https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_spreadsheets_d_1tHPpwNtrGI46mrg7xGuyzjDVFo5JEjpcYW-5Fz4Y-2D9kQ8_edit-3Fusp-3Ddrivesdk&d=DwIFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=qHHNaX-FyUtD4tU3KqK1Mk3WgKA-Lvl1miRd5-369hijK9BUzO34NWsFLWcaqb_o&m=ZsBj7JCb5JShpx__0eWyJm2ofN6kdIW8kZpDOpXZXW8PCtE7Ee6_BxLVjvAWsqXY&s=hZ0UFzkOapbfBtvfEx_KE0pCR8ylj-WROOdnR8fXkUk&e= (Please refer to DU-SBN_FD-Datapool1 sheet)

It would be very important to identify all of those that can be safetly deleted. Their deletion will increase the efficiency of the furthcoming productions.

I would propose a reverse approach: consider all of these paths candidate to be deleted and let me know which ones you are interested in.

Thank you in Advance, Matteo

[mt82]

Dear all, I didn't get any feedback regarding the mail below. This means all the data listed in the spreadsheet can be safely deleted and I will proceed to delete them if there are no concerns.

If this is not the case please let me know as soon as possible.

Matteo

[mt82]

Thanks for the extra ping, Minerba! And also thank you Matteo!

Gray generated all the samples that I am using in my analysis, and I am checking with them if any are on this list. (I am using slimmed down Pandas Dataframes made from these events, so I am not exactly sure of their source.) If any of these were used for Gray's analysis, which should be a superset of what I am using, we as a collaboration would need to keep them indefinitely for reproducability of the analysis, I would think.

Matteo, if you could wait to hear from me once I've been in touch with Gray that would be great.

Thanks, and sorry for not checking this sooner.

Best, Jamie

[mt82]

from @brucehoward-physics Hi Matteo! I see /pnfs/sbn/data/sbn_fd/poms_production/2023A_ICARUS_NuMI_MC_RHC/reverse_horn_current/ on the list of things to be deleted. Please keep this area unless the stuff here has been saved somewhere else too! Wondering if there may be others we are using too :sweat_smile:

[mt82]

SBNDataPool1

Paths	Requester
/pnfs/sbn/data/sbn_fd/poms_production/2023A_ICARUS_NuMI_MC_RHC/reverse_horn_current/	@brucehoward-physics

[mt82]

Hi @PromitaRoy. I found these paths referring to Run2 (reprocessing!?). Which one is the most recent?

/pnfs/sbn/data/sbn_fd/poms_production/data/Run2_Rep1
/pnfs/sbn/data/sbn_fd/poms_production/data/Run2_DataReprocess_V2
/pnfs/sbn/data/sbn_fd/poms_production/data/Run2_Rep
/pnfs/sbn/data/sbn_fd/poms_production/data/Reprocessing_Run2
/pnfs/sbn/data/sbn_fd/poms_production/data/Reproc_Run2
/pnfs/sbn/data/sbn_fd/poms_production/data/Run2 

What about these paths in poms_test ,can they be deleted?

/pnfs/sbn/data/sbn_fd/poms_test/data/Run2_DataReprocess_V2
/pnfs/sbn/data/sbn_fd/poms_test/data/Run2_Rep
/pnfs/sbn/data/sbn_fd/poms_test/data/Run2_DataReprocess
/pnfs/sbn/data/sbn_fd/poms_test/data/Run2
/pnfs/sbn/data/sbn_fd/poms_test/data/Run2_Data
/pnfs/sbn/data/sbn_fd/poms_test2/data/Run2_Rep

[mt82]

/pnfs/sbn/data/sbn_fd/poms_test/data/Run2 is not Run2 reprocessing, but I think it can be deleted as it is supposed to test campaign. Rest all the test directories are for Run2 reprocessing and can be deleted. I started deleting a few, but could complete it. Few of them can be just empty folders, as I deleted them, but I saw /pnfs/sbn/data/sbn_fd/poms_test/data/Run2_DataReprocess is taking a lot of space. I can delete that if you want.

Regarding the actual production /pnfs/sbn/data/sbn_fd/poms_production/data/ the current one is /pnfs/sbn/data/sbn_fd/poms_production/data/Reproc_Run2 and /pnfs/sbn/data/sbn_fd/poms_production/data/Run2 is not reprocessing, it is the original Run2 campaign. Rest all should be deleted. I think (I hope) I have deleted most of them, if you want I can delete those which are still there. Let me know.

ohh also /pnfs/sbn/data/sbn_fd/poms_production/data/Reprocessing_Run2 this one is not run by me.

This is with an old version icaruscode_v09_72_00_06. We should discuss whether we want to keep this or delete this

[mt82]

from @jamiecd
Hi Matteo,

Thanks for your patience hearing back from me, and sorry it took me a bit of time. We have identified seven paths on your proposed list of files to delete, that we need to be saved for the dimuon analyses:

    /pnfs/sbn/data/sbn_fd/data_stage1/NuMIMinBias/run
    /pnfs/sbn/data/sbn_fd/data_stage1/OffBeamNuMIMinBias/run
    /pnfs/sbn/data/sbn_fd/data_stage1/OffBeamNuMIcommissioning/run
    /pnfs/sbn/data/sbn_fd/data_stage1/NuMIcommissioning/run
    /pnfs/sbn/data/sbn_fd/data_stage1/OffBeamNuMIMajority/run
    /pnfs/sbn/data/sbn_fd/data_stage1/NuMIMajority/run
    /pnfs/sbn/data/sbn_fd/poms_production/2023A_ICARUS_NuMI_MC_Nu_Phase2/tuned_signal_shape

Please do not delete the files at any of these paths! In general, we need everything in the paths /pnfs/sbn/data/sbn_fd/data_stage1/ and /pnfs/sbn/data/sbn_fd/poms_production/2023A_ICARUS_NuMI_MC_Nu_Phase2/. (All seven of the bulleted paths listed above fall into these two directories.)

Please let me know if you have any questions, and thank you again for your patience!

Best, Jamie

[mt82]

Hi Jamie, would be a problem if we moved them on tape?

Matteo

[mt82]

related issues: #50 #51 #52 #53