Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines results in duplicate names.

mzient commented 3 months ago

Category:

New feature (non-breaking change which adds functionality) Refactoring (Redesign of existing code that doesn't affect functionality)

Description:

Prior to this change constructing exactly the same pipeline (e.g. by calling a function decorated with @pipeline_def) multiple times produced pipelines with different operator instance names and differently named operator instances and DataNodes. This PR changes that so that pipelines with the same structure have the same node names. This is achieved by:

Using a separate operator counter in each pipeline. When all nodes all instantiated within pipeline scope, no further action is required. This happens for the vast majority of cases.
If a name collision occurs (only possible when "stealing" a subgraph from another pipeline), an error is raised.

Pipelines that are defined without a "current" pipeline have distinct operator instance names.

Additionally, there were some problems with operator discovery. I rewrote it to a much simpler DFS.

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

pipeline_test.py: test_dangling_subgraph

[ ] Existing tests apply
[X] New tests added
- [X] Python tests
- [ ] GTests
- [ ] Benchmark
- [ ] Other
[ ] N/A

Checklist

Documentation

[ ] Existing documentation applies
[ ] Documentation updated
- [ ] Docstring
- [ ] Doxygen
- [ ] RST
- [ ] Jupyter
- [ ] Other
[X] N/A

DALI team only

Requirements

[ ] Implements new requirements
[ ] Affects existing requirements
[X] N/A

REQ IDs: N/A

JIRA TASK: N/A