Support for complex architectures beyond sequential

[ssinhaleite]

Currently the only networks that are supported are sequential models. The chips in fact support branched architectures. The use of this feature is unfortunately limited by proper software support.

The target would be to support a range of network structures as listed below:

Two independent networks:

graph TD;
 A --> B;
 B --> C;
 C --> D;
 E --> F;
 F --> G

Two networks with merging outputs

graph TD;
 A --> B;
 B --> C;
 C --> D;
 E --> F;
 F --> G;
 G --> H;
 D --> H;

A network with a merge and a split

graph TD;
 A --> B;
 B --> C;
 C --> D;
 B --> F;
 F --> G;
 G --> H;
 D --> H;

A network with residual connections:

graph TD;
 A --> B;
 B --> C;
 C --> D;
 A --> E;
 D --> E;
 E --> F;

And finally a complex network structure.

graph TD;
 A --> B;
 A --> C;
 C --> D;
 C --> E;
 B --> D;
 D --> F;
 E --> F;

A special module/layer type that support branched data paths might be useful to try these types of architectures.

Current proposal: Use a Branched module that then has child sequential objects and merges their output at the end. The issues with such a construct is it doesn't support models that have two independent submodules that do not merge. So this is not the ideal solution.

[bauerfe]

@sheiksadique what's the current state of your graph extraction algorithm? Is it feasible to apply it to this issue?

[sheiksadique]

It should in principle be possible with NIR or NIRTorch utilities but it requires some work and isn't a simple plug and play. We will need to modify the current implementation (which assumes sequential everywhere) dramatically.