memory compression question

[wangxdgg]

I have some questions when looking at the memory compression section, is the middle layer memory compressed here, or weight memory? How did you achieve such a high compression ratio.

[ThanatosShinji]

It's the activation compression, no weight memory here. When one activation tensor is consumed by all its consumers, its memory can be freed to the next layer's activation tensor.

[wangxdgg]

So can it be interpreted as peak activation memory usage？

[ThanatosShinji]

You can say it's the peak activation memory usage with dynamic activation allocation.

[wangxdgg]

OK，I got you, thanks a lot !