[core][experimental] Validate DAGs in a hierarchical manner

[kevin85421]

Description

https://github.com/ray-project/ray/pull/47261#discussion_r1727543342

Currently, most of the DAG validations happen during the compile process of ADAG. However, some validations should also be applied to traditional DAGs. Thus, it will be better if we can unity the codepath as follows:

• Step 1: Build a DAG.
• Step 2: Validate the DAG (some invalid cases for both DAG and ADAG).
  • For example, if a DAG node is not a descendant of the InputNode.
• Step 3:
  • Option 1: Execute (DAG)
  • Option 2: Compile (validate ADAG-only logics, e.g. NCCL) -> Execute (ADAG)

Use case

No response

[stephanie-wang]

This needs scoping.