search

getsolus / solbuild

The Solus package build system
https://getsol.us
Apache License 2.0
21 stars 12 forks source link

Allow custom cache directories #23

Open GZGavinZhao opened 2 years ago

GZGavinZhao commented 2 years ago

Currently solbuild can cache C, C++, and Rust builds. However, in the future, it's likely we'll need to cache more things. For example, the cargo registry (takes 1~2 minutes to pull the index), Go build cache, etc.

Therefore, I propose the ability for packagers to specify additional cache entries using /etc/solbuild/<profile>.conf. The structure can look like this:

# By default, sccache and ccache are provided.
[cache]
caches = ["ccache", "sccache", "cargo"]

[cache.cargo]
# Relative to the $BUILDUSER's $HOME directory, which should be enough for most conditions.
# This can be further discussed.
internalDir = ".cargo"
# For legacy builds. Require a full path.
legacyInternalDir = "/root/.cargo"

The benefits of this approach are that it does not affect existing cache (e.g. for the build server that doesn't want any caching), and it is very simple (we don't need to patch YPKG).

To achieve this, the cache directories' layout needs to be changed. Currently, we have /var/lib/solbuild/ccache and /var/lib/solbuild/sccache. For easier cache management, all build cache will be stored in /var/lib/solbuild/cache/<cache-name>. For example, the ccache directory will go to /var/lib/solbuild/ccache, and the cargo registry directory (in the example above) will go to /var/lib/solbuild/cargo. When deleting cache (solbuild dc), we can simply delete /var/lib/solbuild/cache.

If this is not possible, I suggest at least refactor the code so that in the future, we can add additional caches as easy as constructing a new struct (Cache) and add it to a global variable called caches, instead of writing 10 new methods by hand. I already implemented this locally and have been using it for a week, so I can verify this will not break anything. Unfortunately, I recently wiped my PC without pushing my commits to the remote, so I don't have them ready right now, but I am happy to rewrite them since they're still refresh in my head.

joebonrichie commented 2 years ago

We'll definitely want to cache more stuff in the future (the cargo cache comes to mind as you mentioned). This proposal makes general sense to me to avoid mindless copy pasta code of directories into the overlay.

For example, the ccache directory will go to /var/lib/solbuild/ccache, and the cargo registry directory (in the example above) will go to /var/lib/solbuild/cargo. When deleting cache (solbuild dc), we can simply delete /var/lib/solbuild/cache.

Assuming this is a typo e.g. /var/lib/solbuild/cache/cargo /var/lib/solbuild/cache/ccache

For compatibility we'll need to make sure that solbuild dc will also remove the cache of old directory locations if they exist (unless you can think of a better solution).

W.R.T the cargo cache specifically i'm not sure how to handle that. Personally i do not have the cargo cache in my home directory so there is nothing to cache from. It may make more sense to cache it from solbuild itself when building a package. Although, i am not sure if updating a very old cargo cache index is quicker than downloading the initial index from scratch. We may also need to think about caching similar network based tools such as npm, yarn, java build tools as well.

Feel free to send a PR for the cache refactoring for now, as that will cleanup a lot of copypasta, and we can work out details to cache networking based tools in the future.