search

dvmazur / mixtral-offloading

Run Mixtral-8x7B models in Colab or consumer desktops
MIT License
2.29k stars 225 forks source link

(Colab) Clear GPU RAM usage after running the generation code without restarting instance #28

Closed ScottLinnn closed 5 months ago

ScottLinnn commented 6 months ago

I found that if I want to try changes in the source code, I had to rerun the "Initialize model" cell to make the changes effective. However, if I run "Initialize model" cell then run "Run the model" cell, even stopping the execution of "Run the model" cannot release the GPU memory occupied, which makes it impossible to re-build the model which requires another whole bunch of memory. Restarting the instance would cost considerable time to resetup the environment, is there a way to resolve this without restarting? Thanks!

dvmazur commented 6 months ago

Hi! Try setting %load_ext autoreload and %autoreload 2. This will reload the code from the updates files automatically