subgraph TODO

[zheng-da]

Subgraph was proposed as a general mechanism for integrating many different backends. https://cwiki.apache.org/confluence/display/MXNET/Unified+integration+with+external+acceleration+libraries

As the project progresses, we get more people joined for the project. To have better collaboration, we like to maintain a TODO list here for better collaboration. As we get more tasks for this project, we'll update the list.

• [x] merge the default subgraph operator with CachedOp and make CachedOp a normal operator @zheng-da https://github.com/apache/incubator-mxnet/pull/11641
• [x] search for weight arrays in the subgraph, reorder their layout and cache the weight arrays with optimal layout. This task may need to be done for specific backends.
• [ ] enable graph partitioning in CachedOp and bind of the symbol executor.
• [x] infer shape/dtype/storage info before graph partitioning and pass nnvm::Graph that carries shape/dtype/storage info as an argument when creating a subgraph node.
• [ ] Customize memory planning. This task is now needed for MKLDNN because MKLDNN still uses CachedOp to execute operators inside a subgraph. It may be required by other backends as well.
• [x] Memory format conversion on subgraph boundary.
• [ ] Support control flow operators. More generally, support graphs with subgraph nodes.

There are two tasks that are MKLDNN-specific.

• [ ] Support the imperative mode. convert the format of outputs back to the default format.
• [x] Subgraph accuracy issue.

@reminisce @pengzhao-intel @TaoLv @ZhennanQin @ashokei

[pengzhao-intel]

Thanks, @zheng-da , the subgraph will be a unified interface between different implementations and make the whole architecture very clear.

We're very happy to co-work together to accelerate the progress :) And MKL-DNN would be the first backend with this wonderful bridge 🥇

[mbrookhart]

@zheng-da I think all we need for nGraph is

 enable graph partitioning in CachedOp and bind of the symbol executor.

  infer shape/dtype/storage info before graph partitioning and pass nnvm::Graph that carries shape/dtype/storage info as an argument when creating a subgraph node.

We'll review internally and try to assign an Engineer to those items.

[zheng-da]

@azai91

[azai91]

for task 2, what is the optimal layout for the weights?

[zheng-da]

@azai91 the optimal layout is defined by the operator. This is the tricky part of this task. How to determine the optimal layout without breaking the current API?