search

DIRACGrid / DIRAC

DIRAC Grid
http://diracgrid.org
GNU General Public License v3.0
111 stars 173 forks source link

Dataset Management Features #3324

Open ManyAngledOne opened 7 years ago

ManyAngledOne commented 7 years ago

Hello, It would be useful if datasets could contain custom fields, similar to how files and directories can have custom metadata fields. Ideally, these fields would be capable of supporting list like data structures, but even without that, it would be good to have. It would also be useful to be able to query and sort on these fields.

Also, it would be good to be able to alter the metaquery of an existing dataset. There is a function updateDataset which appears to do that, but I could not get it to change anything in the dataset table. I'm probably not specifying the 'datasets' dictionary correctly. There is also the command 'dataset update' in the filecatalog-cli which doesn't seem to work either.

Some background: If these features exist, we would be interested in writing a DIRAC WebApp frontend for dataset management. This could be used to plan future data taking (by creating an empty dataset with fields describing the data that would go into it) and monitoring the analysis status of datasets among other things.

Thanks!

fstagni commented 7 years ago

Thanks for reporting. Maybe @atsareg or @andresailer can comment.

andresailer commented 7 years ago

Sorry, I have never used datasets. But again, if a command fails, maybe there is something in the log files? see also #3317

ManyAngledOne commented 7 years ago

This was what is going on in the logs as far as datasetUpdate: dataset_update_log.txt

I also noticed that the help for 'dataset update' doesn't ask for a new metaquery in 'help dataset' dataset update - update the dataset parameters

dataset_update in the filecatalog-cli doesn't seem to accept anything beyond a dataset name, but updateDataset asks for a "dictionary describing dataset definitions".