Named local repos, and local repos enabled by default

[jrray]

A case has been made that since the spk command is primarily a tool used by developers, and because day-to-day developer workflows generally involve working with the "local" repo, that the "local" repo should be enabled by default. In particular, the env and build commands, but perhaps all of the commands.

There is already a --no-local-repo flag available to disable this behavior on an as-needed basis if we were to make this change.

I further suggest adding the concept of multiple named local repos and/or having local repos be per-user, and that when the "local" repo is enabled by default, this means that the user's (quasi-)private local repo is the one being enabled, so that the user is not surprised by the sudden appearance of packages from some other user's activities on the same host. I cannot think of any real benefit for different users being able to see other user's local packages when using shared resources.

[rydrman]

From the meeting today:

• @rydrman the first part of enabling local is easy
• spk publish -r file:///tmp/myrepo can be used for on-the-fly repos
• the idea of namespaces in repos could solve this for both the local and remote repos
• spk/ns/<name>/pkg/...
• @jrray but there is the need to still fallback as well
• @rydrman namespaces would likely need to be dealt with as separate repos in the solver
• spk build package.yaml -r origin/$USER
• also relates to, #51 because it would start to become more useful to have this

[jrray]

A real world case of needing namespaced local packages hit us today.

I had built a new post release of a low level package I'm working on in CI, using our parallel build system, so across all our CI runners many of them had a single variant build in their local repo.

Later, another person had their CI build fail because it had found the local build of my package, but it wasn't there in the correct build flavor, so it failed with a "would have to build from source" kind of error. (This is related to #173.)

But even if it had found a build variant it needed of my WIP package, they could have published a new build that depends on an unpublished package.

[jrray]

I came up with this cheap and dirty SPI- and gitlab-specific workaround in spfs:

    /// Get the local repository instance as configured.
    pub fn get_repository(&self) -> Result<storage::fs::FSRepository> {
        // Temporary fix for giving pipelines on gitlab-ci isolated local repositories.
        if let Ok(id) = std::env::var("CI_PIPELINE_ID") {
            let isolated_path = self.storage.root.join("ci").join(format!("pipeline_{id}"));
            storage::fs::FSRepository::create(&isolated_path)
        } else {
            storage::fs::FSRepository::create(&self.storage.root)
        }
    }

With some preliminary testing it looks like it works and I don't expect anything to care about finding the extra ci directory when opening the "normal" local repo. I also don't think it would matter that these would all share the same runtime storage (inside the "normal" local repo).

@rydrman Please let me know if you can think of some reason why this would be hazardous to roll out internally. The issue of jobs picking up unwanted local builds is a ticking timebomb.

[rydrman]

Ya, As long as you don't have any other specifically configured or overridden pipeline storage values I don't see why not

[jrray]

The only gotcha so far is that this change needs to exist in both the spfs embedded in spk and the standalone spfs.

[jrray]

A more serious gotcha came up from the change in path to renders caused the overlayfs mount command to be too long. https://github.com/imageworks/spfs/pull/94 was created to try to solve this and there is some urgency in getting our CI out of this situation where it can either fail jobs because of unwanted local packages or fail jobs because there are too many layers.

[rydrman]

From the meeting today:

• what does interacting with namespaces look like for users at the command line (TBD)?