search

tenstorrent / tt-mlir

Tenstorrent MLIR compiler
https://tenstorrent.github.io/tt-mlir/
Apache License 2.0
72 stars 9 forks source link

ttrt deallocate_buffers #588

Closed nsmithtt closed 2 months ago

nsmithtt commented 2 months ago

Plumb through deallocate_buffers API to clear metal allocator between programs. Previously we were running OOM because of leaked buffers. Leaked buffers should largely be cleaned up after the allocate=false support is in #408.

It might still be needed even after #408 lands, because metal internally uses its allocator for some fast dispatch things, but we can revisit what policy to adopt.

nsmithtt commented 2 months ago

This pipeline on my other PR is failing because of this, see https://github.com/tenstorrent/tt-mlir/actions/runs/10674847934/job/29587241905

Always | FATAL    | Out of Memory: Not enough space to allocate 32768 B L1 buffer across 1 banks, where each bank needs to store 32768 B