Runtime validation is performed before_node_run. This means we validate only nodes which are loaded (e.g. inputs or intermediate outputs). We should also validate terminal nodes before saving them.
Context
Users expect all datasets being validated once.
Possible Implementation
Create a after_dataset_save or a after_node_run hook which checks if the dataset is a terminal output of a pipeline before validation
Description
Runtime validation is performed
before_node_run
. This means we validate only nodes which are loaded (e.g. inputs or intermediate outputs). We should also validate terminal nodes before saving them.Context
Users expect all datasets being validated once.
Possible Implementation
Create a
after_dataset_save
or aafter_node_run
hook which checks if the dataset is a terminal output of a pipeline before validation