openego / eGon-data

GNU Affero General Public License v3.0
10 stars 4 forks source link

Having multiple `Dataset`s ending in parallel tasks doesn't work #985

Closed gnn closed 1 year ago

gnn commented 1 year ago

More specifically, having multiple Datasets ending in parallel tasks doesn't work if those Datasets are in a module below the egon.data.datasets package. In that case the code removing the module name prefix from task ids and the code generating the final dataset task which updates the dataset version once all parallel tasks have finished interact in a way that generates non-distinct task ids so that tasks generated later clobber the ones generated earlier. This leads to spurious cycles and other inconsistencies and bugs in the graph.