as announced last week, we will be doing a large-scale removal campaign of data saved in the cmgtools area of the cmst3 group and user space.
Just to be abundantly clear from the start: this will delete roughly 500 TB of mostly PAT/CMG-tuples, so if you think you might have something lying there, please read the following carefully.
There are two directories affected in this campaign:
/eos/cms/store/cmst3/user/cmgtools/
and
/eos/cms/store/cmst3/group/cmgtools/
I compiled a list of all subdirectories in these two paths that have things stored. You can find a list of these subdirectories in the two following files on my www (they are also attached to this message).
the format is:
where lastAccess is taken from os.path.getatime, which might not be the actual last access time, but some more or less useful/useless timestamp.
As far as where this data is from and which runperiod, reco version etc: most of this happened before my time, so I'm not entirely sure. All of it pre-dates MINIAOD, and only three directory names contain the string "13TeV" (namely in /eos/cms/store/cmst3/user/cmgtools/susy/). Only one directory contains the string 80X (in the same place). Most files are cmg-tuples and PAT tuples.
The procedure and timeline for the deletion is the following: By Tuesday next week, the 29th of January 2019, I will move all the files into a new subdirectory in each of the two folders called cleanup/. The data will remain exactly there for another week, until the 5th of February 2019. By the morning of the 6th of February, all the files will be deleted. Nothing will be archived, nothing is backed up, so those files will be gone for good.
If you want to keep some files: Please send me an email with the directories you want to keep from those two text files along with a good reason why you think you will need the data in the future. We can discuss what qualifies as a "good" reason, but note that all the space we will free up will be added to our local space to store data that is actually used on a daily basis, so removal has priority over archival.
If you know of somebody who might want to keep some data there but isn't subscribed to this github repository/issue, please do forward them this message.
Hi,
as announced last week, we will be doing a large-scale removal campaign of data saved in the cmgtools area of the cmst3 group and user space.
Just to be abundantly clear from the start: this will delete roughly 500 TB of mostly PAT/CMG-tuples, so if you think you might have something lying there, please read the following carefully.
There are two directories affected in this campaign: /eos/cms/store/cmst3/user/cmgtools/ and /eos/cms/store/cmst3/group/cmgtools/
I compiled a list of all subdirectories in these two paths that have things stored. You can find a list of these subdirectories in the two following files on my www (they are also attached to this message).
for the user/cmgtools directory: https://mdunser.web.cern.ch/mdunser/public/cleanup2019/fancy2019_output_du_eos_cms_store_cmst3_user_cmgtools_2019_01_22.txt
and for the group/cmgtools directory: https://mdunser.web.cern.ch/mdunser/public/cleanup2019/fancy2019_output_du_eos_cms_store_cmst3_group_cmgtools_2019_01_22.txt
the format is:
where lastAccess is taken from os.path.getatime, which might not be the actual last access time, but some more or less useful/useless timestamp.
As far as where this data is from and which runperiod, reco version etc: most of this happened before my time, so I'm not entirely sure. All of it pre-dates MINIAOD, and only three directory names contain the string "13TeV" (namely in /eos/cms/store/cmst3/user/cmgtools/susy/). Only one directory contains the string 80X (in the same place). Most files are cmg-tuples and PAT tuples.
The procedure and timeline for the deletion is the following: By Tuesday next week, the 29th of January 2019, I will move all the files into a new subdirectory in each of the two folders called cleanup/. The data will remain exactly there for another week, until the 5th of February 2019. By the morning of the 6th of February, all the files will be deleted. Nothing will be archived, nothing is backed up, so those files will be gone for good.
If you want to keep some files: Please send me an email with the directories you want to keep from those two text files along with a good reason why you think you will need the data in the future. We can discuss what qualifies as a "good" reason, but note that all the space we will free up will be added to our local space to store data that is actually used on a daily basis, so removal has priority over archival.
If you know of somebody who might want to keep some data there but isn't subscribed to this github repository/issue, please do forward them this message.
All the best, -marc
fancy2019_output_du_eos_cms_store_cmst3_group_cmgtools_2019_01_22.txt fancy2019_output_du_eos_cms_store_cmst3_user_cmgtools_2019_01_22.txt