search

esp-rs / espup

Tool for installing and maintaining Espressif Rust ecosystem.
Apache License 2.0
224 stars 23 forks source link

espup uses large amounts of space in $TMPDIR #342

Closed codyps closed 1 year ago

codyps commented 1 year ago

Bug description

espup downloads and extracts data in $TMPDIR, resulting in ~1.2GiB of space usage. $TMPDIR is often a ram disk, so we're storing data to ram. We later copy this same data to other locations (directories in $RUSTUP_HOME, etc).

To Reproduce

Steps to reproduce the behavior:

  1. Point TMPDIR at a tmpfs (probably the default, if you're using linux with systemd)
  2. Run espup so that it downloads & installs a toolchain
  3. Use df -h to note 1.2GiB of tmpfs usage

Expected behavior

  1. Use less tmpfs space
  2. Minimally, don't extract the tar balls to $TMPDIR, instead extracting them adjacent to their final location so they can be moved into place
  3. Ideally, extract the file as a streaming operation as it is downloaded to avoid the use of $TMPDIR entirely.

Environment

Additional context

Systemd defaults RuntimeDirectory (which $TMPDIR points to) to 10% of available memory (https://www.freedesktop.org/software/systemd/man/logind.conf.html#RuntimeDirectorySize=). This means on a system with 2GiB of ram, we get 200MiB. With the current 1.2GiB of usage, espup users with linux + systemd defaults will need to ensure they have ~12GiB of ram.

SergioGasquez commented 1 year ago

Hi, thanks for opening the issue! I am not sure how to solve it, I will do some investigation on my side, if you have any suggestion, they are more than welcome!

3.Ideally, extract the file as a streaming operation as it is downloaded to avoid the use of $TMPDIR entirely.

This would be perfect, but I am not sure if there is a way to do this.

codyps commented 1 year ago

A good option here:

Download & extract files to the rustup tmp dir ($HOME/.rustup/tmp, $RUSTUP_HOME/tmp, etc) instead of $TMPDIR. This would mirror rustup behavior which folks probably expect. (even if they don't like it much. I have symlinks in place so that $RUSTUP_HOME/tmp ends up in $XDG_CACHE_HOME).

SergioGasquez commented 1 year ago

A good option here:

Download & extract files to the rustup tmp dir ($HOME/.rustup/tmp, $RUSTUP_HOME/tmp, etc) instead of $TMPDIR. This would mirror rustup behavior which folks probably expect. (even if they don't like it much. I have symlinks in place so that $RUSTUP_HOME/tmp ends up in $XDG_CACHE_HOME).

Good idea! Just implemented this, do you mind testing it?

cargo uninstall espup
cargo install espup --git https://github.com/SergioGasquez/espup --branch feature/update-tmp-loc
espup install
codyps commented 1 year ago

Placing temporary data in the final toolchain directory certainly does work, and it runs fine for me.

It does feel a little weird to use the toolchain directory instead of the $RUSTUP_HOME/tmp directory, because that directory already exists and is used for this purpose by rustup, but what's there now is a definite improvement over using $TMPDIR given the size of the temporary data.

SergioGasquez commented 1 year ago

Updated the branch to use $RUSTUP_HOME/tmp and simplified the code a bit!

codyps commented 1 year ago

Works for me 👍