Knowledge Capture Blocks

[NareshPS]

Types of information capture blocks in seq2seq networks:

1- Essence Blocks 2- Relationship Blocks 3- Generalization Blocks 4- Specialization Blocks

Essence Blocks

They can be composed off:

• Feed-forward layers.
• Specialised information capture layers such as convolution, embedding etc.

Relationship Blocks

They can composed off:

• Attention Layers

Generalization Blocks

They generalize inputs into broad topics. Composition choices: ??

Specialization Blocks

They specialize topics into sub-topics. Composition choices: ??

[NareshPS]

Resources

• Blockwise self-attention for long document understanding. (Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih)