search

sayakpaul / diffusers-torchao

End-to-end recipes for optimizing diffusion models with torchao and diffusers (inference and FP8 training).
Apache License 2.0
274 stars 8 forks source link

cleanup utility #31

Closed sayakpaul closed 2 months ago

sayakpaul commented 2 months ago

We should clean up the tmp directory as it can cause our node to drain out.

Check internal thread: https://huggingface.slack.com/archives/C043JTYE1MJ/p1726211997401189.

a-r-r-o-w commented 2 months ago

FYI: My latest run deleted everything from /tmp/aryan including compiled output code, previous triton files, etc. and killed my tmux sessions because tmux stores its session info in a random file like: /tmp/tmpj4txyw6vwandb.

Would not recommend this if you use /tmp for storing stuff like installers, etc. (my pyenv installer was in /tmp/aryan too which got nuked as well, but its fine since I didn't really need it any more)

sayakpaul commented 2 months ago

tmux sessions are fine. Installers should never go to /tmp. It should be under the /fsx volume.

For the compiled code output, we can always set TORCH_LOGS_OUT.