search

freight-team / freight

A modern take on the Debian archive.
Other
107 stars 37 forks source link

Create repo with symbolic links #126

Closed RadxaYuntian closed 5 months ago

RadxaYuntian commented 2 years ago

Why recreate this PR (it is the same as #122)

We have found a race condition in freight so we are going to create a new PR to fix it. However, we didn't anticipate that freight's development will stay dormant for so long so we used master branch in the original PR. As such we have created a new branch to host this PR and keep master for our own use.

Background

We currently have an apt repo hosted on GitHub Pages. This was set up a long time ago and we are having multiple issues with it. Most importantly we are hitting GitHub Pages' storage quota.

Motivation of this PR

We ran out of space because we store identical copies of many packages for different distros, which is why we are very interested with freight since duplicated files are actually hard links, so no wasted spaces. Unfortunately git only supports symbolic links, and treat hard links as separate files that happen to have identical content. As such this PR adds the functionality to create an apt repo with symbolic links.

Compromise

We cannot only create symbolic link between VARLIB and VARCACHE, since each distro in VARLIB requires an copy of the package. As such we need to store all packages in a new location VARPOOL first, then both VARLIB and VARCACHE are symbolic linked to this location. The current implementation has the unfortunate side-effect with package removal: Remove from... also remove from VARPOOL? also remove from VARLIB? also remove from VARCACHE?
VARPOOL Yes Yes Yes
VARLIB No Yes No, since it is still pointing to VARPOOL
VARCACHE No No Will be recreated by f-cache

This basically means --keep option is always on for freight cache since VARCACHE is pointing at VARPOOL. It is possible to make VARCACHE pointing at VARLIB like the case with hard link, but that is equal to have --keep option always off. Since the switch is gonna be broken either way we settle with the current implementation to have less complex code, and deleting an existing package then rebuilding index is easier than adding a package after the fact.

Testing

Currently bats-core doesn't support loading test scripts, so we cannot have a file for generic tests and 2 files with different setup(): 1 for hard link and 1 for symbolic link. Instead the same tests are run twice with different init arguments.

3 tests are disabled for symbolic link repo: 1 for checking hard link is working, and 2 for --keep option which is broken here. All other tests passed.

Sample use case

Currently there is a trivial testing repo already set up with this PR.

RadxaYuntian commented 2 years ago

So the race condition turned out to be in our code (duh), but that part made me realize that our use case is quite different from other users. We ship mostly kernel images or bootloaders that do not link to other libraries, so different distros can share them without issue. However, once we need to ship something with external dependencies, we cannot guarantee the same packages that symlinked in different distros will be the same binary. Since we are not currently ship that kind of packages, I'll mark this as a draft for now.

RadxaYuntian commented 5 months ago

GitHub is looking to disable symbolic link on Pages, so this PR is losing its primary use case. We will probably move to aptly in the future, which seems more actively developed.