search

neurosim / DNN_NeuroSim_V1.4

Benchmark framework of compute-in-memory based accelerators for deep neural network (inference engine focused)
53 stars 12 forks source link

Transformer Support #1

Closed aagoksoy closed 1 year ago

aagoksoy commented 1 year ago

Is it possible to use Neurosim for Transformer networks? If not supported, which parts of the simulator should be changed? Is there any tutorial for such an addition?

neurosim commented 1 year ago

Hi Alper! Currently transformer networks are not supported in NeuroSim V1.4.

Using compute-in-memory to accelerate transformers is still in early stages. The main challenge is supporting new operations in transformers that don't exist in CNNs and MLPs. For example, calculating the Q, K, and V matrices in transformers require a matrix-matrix multiplication where both the input matrix and multiplicand matrix are different for each computation. This operation can be difficult to accelerate with compute-in-memory as re-programming the memory array frequently is undesirable for emerging non-volatile memories. Addressing this problem is an active area of research.

We are currently working on support for transformers in NeuroSim with the above solution which we plan to release in the next update.

-James

aagoksoy commented 1 year ago

thanks! that's helpful

HuiKou1 commented 3 months ago

@neurosim Hi! I noticed that your team have recently made progress on your NeuroSim work with Transformer models, specifically H3DAtten(23) and H3D-Transformer(24). Do you have any plans or timeline for this release? I am very excited to see this amazing work come to fruition.