Dataset manager

[guiguem]

This issue is to track progress on extending DIRAC's dataset management and metadata features.

Some motivations and goals for the dataset management project:

• Data organization
• Tracking status of dataset analysis and processing - What processing has been done? Which version of Katydid used, etc? What effects has this dataset been analyzed for?
• Provide a central location for comments that is closely associated with the data.
• Mass modify file metadata on a dataset basis with proper access control.
• Export lists of physical or logical filenames for each dataset and possibly filtering these, eg only getting .root files.
• Planning future data taking - Creating datasets for data that doesn't exist yet.
• Creating a general interface for adding and removing metadata and metadata fields for files and datasets

[ManyAngledOne]

Currently DIRAC datasets contain:

• Dataset ID
• Dataset name
• Metadata query - defines the files in the dataset (in our case this will be a query for files with metadata tags to certain run numbers)
• User ID
• Group ID - Associated with the user.
• Status - Refers to whether or not the LFNs are stored with the dataset ("Frozen" or "Static") or they are generated with each request using the metadata query ("Dynamic")
• Creation date
• Modification date
• Total size - Total size of all files associated with the dataset
• Hash - MD5 of the string of the list of LFNs for files associated with the dataset
• Number of associated files
• Annotation - A 512 character string for storing some information.*
• Mode - (?)
• List of LFNs - A list of LFNs of associated files. Only used when the the dataset is Frozen or Static.

*It seems as though only one annotation is allowed per dataset, despite there being a separate table for this information. This is because new annotations are added with REPLACE which will delete existing annotations which share the same primary key, in this case the dataset number.

The database which stores this information is defined here: https://github.com/DIRACGrid/DIRAC/blob/ce62643461be63dea717c8a3b96087e77f38137f/DataManagementSystem/DB/FileCatalogWithFkAndPsDB.sql#L286

DIRAC's dataset functions which use this data: https://github.com/DIRACGrid/DIRAC/blob/9f408b12ca57a92207e9c6a0defdf6efcdf81bc5/DataManagementSystem/DB/FileCatalogComponents/DatasetManager.py

[ManyAngledOne]

The dirac-dms-filecatalog-cli dataset command is (to my knowledge) the only interface to the dataset system. It's functionality as described by its help:

A set of dataset manipulation commands Usage: dataset add - add a new dataset definition dataset annotate - add annotation to a dataset dataset show [-l] [] - show existing datasets dataset status - display the dataset status dataset files - show dataset files
dataset rm - remove dataset dataset check - check if the dataset parameters are still valid
dataset update - update the dataset parameters dataset freeze - fix the current contents of the dataset
dataset release - release the dynamic dataset

https://github.com/DIRACGrid/DIRAC/blob/9f408b12ca57a92207e9c6a0defdf6efcdf81bc5/DataManagementSystem/Client/FileCatalogClientCLI.py#L1979

[ManyAngledOne]

We would like DIRAC to contain this information:

• Dataset Number
• Title
• Date Requested - creation date of the dataset
• Date Range Performed - date ranges when the runs making this dataset were performed (could be auto generated)
• Modification date
• Priority - Which datasets should be gathered first?
• Category
• Contact
• Performer - Who was the operator?
• Run Number Ranges - Can be represented as a metaquery.
• Description
• Acquisition Notes
• eLog Entry URL
• Analysis Status - Done, pending, etc
• Processing completed - Which tools have been used to process the runs.
• Data Quality - If there are a few bad runs in the dataset, flag the data quality as bad and add something to the analysis notes notes.
• Analysis Notes
• Automated Data Quality - If we determine some ways to automatically check for data quality.

[ManyAngledOne]

Currently DIRAC does not support modifying existing data sets, except replace the annotation, this is a critical feature to us.

edit: Apparently this is supported by the back end (at least there is code which appears to have accomplished this at one point), but the interface to it through the filecatalog-cli appears to be broken. The command is 'dataset update' but it doesn't seem to do anything.