Use new graph in Pipeline - Githubissues

NVIDIA / DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html

Apache License 2.0

4.98k stars 609 forks source link

Use new graph in Pipeline #5520

Closed mzient closed 1 week ago

mzient commented 2 weeks ago

Category:

Refactoring (Redesign of existing code that doesn't affect functionality) Breaking change (kind of)

Description:

This PR switches the Pipeline class to the new OpGraph as the main representation of the graph. The changes involve:

constructing the graph through OpGraph::Builder
constructing the executor with new graph (the lowering is hidden)
using Pipeline::GetOperator() instead of Pipeline::GetOperatorNode().op
explicitly adding preserve argument based on schema (new OpGraph doesn't look at the schema)
moved writing DOT files to the new graph (as a free function now)
removal of node ids from DOT files and deprecation of show_ids from The changes affect multiple tests that previously relied on access to the legacy OpGraph.

Breaking change:

save_graph_to_dot_file - the parameters are now keyword only, so we can remove the obsolete argument later. This function is neither documented nor used anywhere in our code base (it has no tests, either) and is used for debugging only.

Additional information:

Affected modules and functionalities:

Pipeline Tests

Key points relevant for the review:

Tests:

Multiple tests were modified to work with the new graph.

[x] Existing tests apply
[ ] New tests added
- [ ] Python tests
- [X] GTests
- [ ] Benchmark
- [ ] Other
[ ] N/A

Checklist

Documentation

[ ] Existing documentation applies
[ ] Documentation updated
- [ ] Docstring
- [ ] Doxygen
- [ ] RST
- [ ] Jupyter
- [ ] Other
[X] N/A

DALI team only

Requirements

[ ] Implements new requirements
[ ] Affects existing requirements
[X] N/A

REQ IDs: N/A

JIRA TASK: DALI-3974

dali-automaton commented 2 weeks ago

CI MESSAGE: [15768824]: BUILD STARTED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15768824]: BUILD FAILED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15810200]: BUILD STARTED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15810200]: BUILD FAILED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15827387]: BUILD STARTED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15831491]: BUILD STARTED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15831491]: BUILD PASSED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15925255]: BUILD STARTED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15927182]: BUILD STARTED

dali-automaton commented 2 weeks ago

CI MESSAGE: [15927182]: BUILD PASSED