Clay-foundation / model

The Clay Foundation Model (in development)
https://clay-foundation.github.io/model/
Apache License 2.0
262 stars 30 forks source link

Small memory buildup #157

Closed brunosan closed 5 months ago

brunosan commented 5 months ago

When doing very long runs, the memory footprint increases, which ends up crashing the run. Specially since we only save the weights this could be a problem for future long runs.

The plot below comes from the same run as #156

Screenshot 2024-02-18 at 23 33 20