Runtime validation is performed before_node_run. This means we validate only nodes which are loaded (e.g. inputs or intermediate outputs). We should also validate terminal nodes before saving them.
Context
Users expect all datasets being validated once.
Possible Implementation
Create a after_dataset_save or a after_node_run hook which checks if the dataset is a terminal output of a pipeline before validation
Galileo-Galilei
commented
3 months ago
Description
Runtime validation is performed
before_node_run. This means we validate only nodes which are loaded (e.g. inputs or intermediate outputs). We should also validate terminal nodes before saving them.Context
Users expect all datasets being validated once.
Possible Implementation
Create a
after_dataset_saveor aafter_node_runhook which checks if the dataset is a terminal output of a pipeline before validation