Sanity checks for StorageConsistencyCheck

dmwm / PHEDEX

CMS data-placement suite

8 stars 18 forks source link

Sanity checks for StorageConsistencyCheck #990

Open nikmagini opened 9 years ago

nikmagini commented 9 years ago

Currently the StorageConsistencyCheck script does a simple comparison between the storage content and TMDB; a site admin blindly taking the output runs the risk of deleting valuable non-PhEDEx data which is classified as orphan by the script.

Examples of sanity checks to add:

exclude permanent non-PhEDEx directories like /store/group or /store/user from the list of orphans
exlcude files newer than X from the list of orphans (to avoid deleting files just produced, or temporary files still in use)

nataliaratnikova commented 9 years ago

In addition, SCC will qualify as orphans the files that belong to another PhEDEx instance.

joadiazcr commented 9 years ago

When we (Tranfer Team) ask for the full list of LFNs, we ask for files with the following conditions:

Files in the following directories(I think those directories only contain Production Files): /store/mc /store/data /store/generator /store/results /store/hidata /store/himc /store/lumi /store/relval
Files "OLDER THAN ONE MONTH"

nataliaratnikova commented 9 years ago

There are some Dev instance datasets under /store/data/GowdyTest10 . Not every site needs or uses them, but they become very useful when it comes to testing new storage instance, developing site specific plugins, etc.