gc: Determine CLI design for manual cleaning

[ehuss]

The current implementation from https://github.com/rust-lang/cargo/pull/12634 exposes a cargo clean gc subcommand to handle manually cleaning cache data. It is not clear what the final CLI design should be (and it is not clear exactly what the user scenarios are for when they would want to take manual control). This issue is tracking for determining what the CLI should look like. There are few different considerations:

• What should the actual subcommand be called? There were a few different considerations:
  • cargo cache is already in use by a third-party command
  • cargo gc (reserved by a third-party, but unused), was part of early proposals (cargo doesn't handle unstable top-level subcommands very well)
  • cargo clean gc — the current implementation
  • cargo clean — just fold the functionality into a single command which handles cleaning caches. @epage has concerns that this is overloading a subcommand used for different types of caches (local vs global, etc.).
  • cargo maintenance — just an idea stolen from git
• How does this evolve with cleaning target directories, global build caches, etc.?
• Which flags should be exposed? Currently it exposes a large number of low-level flags. Ideally I would like to only stabilize a small number of higher-level flags. But I don't know what the high-level user scenarios might be, so I'm uncertain exactly what to expose.
  • For example, one idea is to have a flag for "things that can be recreated" and another for "things that require downloading".
  • Also bikeshed the names of the options.

[epage]

Potential sources of prior of other tools with cache management

• poetry
• docker
• git
• nix, zig
• dune, etc)

Apparently buck2 and Turborepo grow unbounded

[epage]

Quick scan of brew

• autoremove
• cleanup
  • "Removes all downloads more than 120 days old. This can be adjusted with HOMEBREW_CLEANUP_MAX_AGE_DAYS."
• HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS (default 30 days)

One complaint that came up was "brew cleanup has not been run in 30 days, running now" ... and then proceeds to run an interminable process in the middle of you attempting to do something else." (mastadon)

[epage]

We should probably step back and enumerate what the required use cases are and the "if it works" use cases.

[juliusl]

I like the way pnpm approaches this,

https://pnpm.io/cli/store

Running pnpm store prune is not harmful and has no side effects on your projects. If future installations require removed packages, pnpm will download them again.

It is best practice to run pnpm store prune occasionally to clean up the store, but not too frequently. Sometimes, unreferenced packages become required again. This could occur when switching branches and installing older dependencies, in which case pnpm would need to re-download all removed packages, briefly slowing down the installation process.

[juliusl]

Regarding the concern of deleting crates that might still be in use, I like how rushjs asks you to pass an --unsafe flag to it's purge command,

https://rushjs.io/pages/commands/rush_purge/

 --unsafe    (UNSAFE!) Also delete shared files such as the package manager
              instances stored in the ".rush" folder in the user's home
              directory. This is a more aggressive fix that is NOT SAFE to
              run in a live environment because it will cause other
              concurrent Rush processes to fail.

[epage]

@juliusl

https://pnpm.io/cli/store

it sounds like pnpm store prune removes everything that isn't referenced, regardless of age or size.

In #13137 I bring up the idea to build on top of the work to track workspaces in #13136 so we pin entries not in current lockfiles.

To extend this to clean up everything has the risk is if a project is transient (e.g. removable media) or moved but a new command wasn't run to register the new location. If its manually done with a command, rather than part of the auto-gc, then that might be reasonable, especially if we swap the logic and have a --execute flag rather than a --dry-run flag.

Regarding the concern of deleting crates that might still be in use, I like how rushjs asks you to pass an --unsafe flag to it's purge command,

I don't think an --unsafe flag is as relevant. For any shared directories, we use filesystem locks to ensure consistent reads/writes.

Any "in use" concerns we have are more about "relevant to the user" and not "file descriptors are open" and is mostly relevant for slow networks/systems and offline usage (I don't want a rarely used dependency being removed just before I go on an airplane to do development offline).