Currently the Pipeline creates a task for each file as it run CSVLint/csv2rdf etc, which is good but can make the pipeline graph/user experience quite messy. SubDags should be able to group all of these tasks under one task in the pipeline for each step for better readability without sacrificing utility.
First thing to check will be how these SubDags interact with the XCom variables as these are used to pass around the Draft_ID so all the data is uploaded to the correct draft.
RedWalters
commented
1 year ago
Currently the Pipeline creates a task for each file as it run CSVLint/csv2rdf etc, which is good but can make the pipeline graph/user experience quite messy. SubDags should be able to group all of these tasks under one task in the pipeline for each step for better readability without sacrificing utility. First thing to check will be how these SubDags interact with the XCom variables as these are used to pass around the Draft_ID so all the data is uploaded to the correct draft.