Closed QLeB closed 2 years ago
Because queryDatasets (https://github.com/LSSTDESC/gen3_workflow/blob/master/python/desc/gen3_workflow/gather_resource_info.py#L84) returns duplicates (https://pipelines.lsst.io/v/weekly/middleware/faq.html#why-do-queries-return-duplicate-results), the output from gather_resource_info have some duplicated metadata, i.e. several metadata for the same task.
As suggested in the above link, set() could be used to deduplicate.
closed by #44
jchiang87
commented
2 years ago jchiang87
commented
2 years ago
Because queryDatasets (https://github.com/LSSTDESC/gen3_workflow/blob/master/python/desc/gen3_workflow/gather_resource_info.py#L84) returns duplicates (https://pipelines.lsst.io/v/weekly/middleware/faq.html#why-do-queries-return-duplicate-results), the output from gather_resource_info have some duplicated metadata, i.e. several metadata for the same task.
As suggested in the above link, set() could be used to deduplicate.