Problem:
Recently from dependencies of tgi-gaudi project some torch compile graph breaks were event out and it made some torch compiled graphs much bigger and more memory consuming which in some models could led to Device out-of-memory.
Solution:
Torch compiled graphs that wer causing Device OOM behaviour where related to having loops inside of them that where processing lots of tensors. Those functions with loops were excluded from torch compilation process.
Problem: Recently from dependencies of tgi-gaudi project some torch compile graph breaks were event out and it made some torch compiled graphs much bigger and more memory consuming which in some models could led to Device out-of-memory.
Solution: Torch compiled graphs that wer causing Device OOM behaviour where related to having loops inside of them that where processing lots of tensors. Those functions with loops were excluded from torch compilation process.