search

kata-containers / packaging

Kata Containers version 1.x packaging (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
119 stars 92 forks source link

Using packagecloud for durable package hosting #328

Closed ttx closed 3 years ago

ttx commented 5 years ago

We currently use the OpenSUSE Build Service to build packages for a number of Linux distributions where we are not in official package repositories yet. The built artifacts are distributed via the OBS mirror network.

One key limitation of that setup is that OBS branch-oriented model only stores one version of an artifact for a given series, rather than archive all past artifacts. That can make debugging hard, or even impossible in corner cases (example: https://github.com/kata-containers/runtime/issues/1081#issuecomment-452721605)

To solve that limitation we are considering using PackageCloud as a way to distribute Kata packages and have better retention of past built artifacts. This usage should be temporary, while we work on getting Kata containers properly packaged in all relevant distributions.

ttx commented 5 years ago

Current status: we contacted packagecloud a couple of weeks ago to get a non-profit account for Kata Containers. They replied (yesterday) favorably to our request, but have a bit of a backlog to process before setting up the account. The default plan for OSS projects comes with 250GB/month of bandwidth and 25GB of storage. In exchange for the free plan, they ask that we put a link in our project's README and our project's website linking back to packagecloud.io.

jodh-intel commented 5 years ago

@marcov, @jcvenegas - we're going to need your expertise here to work out how to get packages from OBS into packagecloud. We'll need to think about how we're going to handle the stable releases on packagecloud too. And we're going to have to change those installation guide URLs (again! ;) [*]

/cc @egernst, @nitkon, @Pennyzct, @alicefr.

[*] - To avoid any more future URL changes, I wonder if it makes sense to have some sort of redirect so that users will specify install URLs like:

... and that would redirect that to the correct packagecloud URL. I guess that's prolly going to be too much effort and that doesn't stop users specifying the packagecloud URL directly, but...?

marcov commented 5 years ago

you can use rsync to copy the OBS repository content. E.g.:

$ obs_project="home:/katacontainers:/releases:/x86_64:/master"
$ rsync -rlptH rsync.opensuse.org::opensuse-full-really-everything-including-repositories/opensuse/repositories/${obs_project} dest-directory --delete-after -hi --stats

Content change needs to be polled, either with rsync itself or fetching the repository metadata and comparing it with a local copy previously fetched, e.g. check for changes to the timestamp field in repomd.xml

jodh-intel commented 5 years ago

We'd have to be careful with rsync to avoid installs failing for users when the rsync itself was in progress. We could run something like the following on the packagecloud server:

today=$(date '+%Y-%m-%d')

repo="kata_repo"
new="$repo.$(TODAY)"

current=$(readlink -e "$repo")

# Seed the new repo with the contents of the current
cp -a "$repo" "$new"

# Update the repo to the latest from OBS
cd "$new"
rsync ...
cd ..

# Atomic update
ln -sf "$new" "$repo"

# Cleanup
rm -rf "$current"

That would minimise problems I think but won't rule them out entirely I guess.

marcov commented 5 years ago

you are right @jodh-intel, we should try to have the package copy done in an atomic way to minimize any inconsistent state.

Another thing to keep in mind is that packages on OBS may be rebuilt without changes to the package source being committed, e.g. in case a bug / security issue is found in the compiler or a linked library.

Ultimately the best thing to do is to run at least some packages installation tests on the supported distributions, using e.g. containers.

marcov commented 5 years ago

btw, if you are looking for volunteers I'd like to to start looking into this and set things up to use PackageCloud.

jodh-intel commented 5 years ago

Nice - thanks @marcov !

marcov commented 5 years ago

I started playing with PackageCloud and here's my findings.

Some potential drawbacks:

  1. The packages signing key that comes from OBS cannot be imported in PackageCloud. This is because (from what I can understand), PackageCloud requires packages to be signed with a subkey, while OBS uses master keys directly, without subkeys.

This means that, when setting up the new repository on your host, you need an extra step to manually import the OBS public key.

  1. Package Cloud provides a CLI tool to upload packages to their CDN that requires user input whenever is launched and I have yet to find a way to automate it.

  2. It supports a discrete set of distros, even if additional ones may be added if contacting support. RPM:

    
Listing distributions for rpm:

Enterprise Linux - Amazon Linux | CentOS | RedHat (el):

    Enterprise Linux 5.0 | CentOS 5 (5)
    Enterprise Linux 6.0 | CentOS 6 | Amazon Linux (6)
    Enterprise Linux 7.0 | CentOS 7 (7)

Fedora (fedora):

    14 Laughlin (14)
    15 Lovelock (15)
    16 Verne (16)
    17 Beefy Miracle (17)
    18 Spherical Cow (18)
    19 Schrödinger's Cat (19)
    20 Heisenbug (20)
    21 Fedora 21 (21)
    22 Fedora 22 (22)
    23 Fedora 23 (23)
    24 Fedora 24 (24)
    25 Fedora 25 (25)
    26 Fedora 26 (26)
    27 Fedora 27 (27)
    28 Fedora 28 (28)
    29 Fedora 29 (29)

Scientific Linux (scientific):

    Scientific Linux 5.0 (5)
    Scientific Linux 6.0 (6)
    Scientific Linux 7.0 (7)

Oracle Linux (ol):

    Oracle Linux 5.0 (5)
    Oracle Linux 6.0 (6)
    Oracle Linux 7.0 (7)

poky (Yocto Project Reference Distribution) (poky):

    2.0 Jethro (jethro)
    2.1 Krogoth (krogoth)

openSUSE (opensuse):

    openSUSE 13.1 (13.1)
    openSUSE 13.2 (13.2)
    openSUSE Leap 15.0 (15.0)
    openSUSE Leap 15.1 (15.1)
    openSUSE Leap 42.1 (42.1)
    openSUSE Leap 42.2 (42.2)
    openSUSE Leap 42.3 (42.3)

SUSE Linux Enterprise Server (sles):

    SUSE Linux Enterprise Server 11.4 (11.4)
    SUSE Linux Enterprise Server 12 (12.0)
    SUSE Linux Enterprise Server 12.1 (12.1)
    SUSE Linux Enterprise Server 12.2 (12.2)
    SUSE Linux Enterprise Server 12.3 (12.3)
    SUSE Linux Enterprise Server 15.0 (15.0)

If you don't see your distribution or version here, email us at support@packagecloud.io.


DEB:

Listing distributions for deb:

Ubuntu (ubuntu):

    4.10 Warty Warthog (warty)
    5.04 Hoary Hedgehog (hoary)
    5.10 Breezy Badger (breezy)
    6.06 LTS Dapper Drake (dapper)
    6.10 Edgy Eft (edgy)
    7.04 Feisty Fawn (feisty)
    7.10 Gutsy Gibbon (gutsy)
    8.04 LTS Hardy Heron (hardy)
    8.10 Intrepid Ibex (intrepid)
    9.04 Jaunty Jackalope (jaunty)
    9.10 Karmic Koala (karmic)
    10.04 LTS Lucid Lynx (lucid)
    10.10 Maverick Meerkat (maverick)
    11.04 Natty Narwhal (natty)
    11.10 Oneiric Ocelot (oneiric)
    12.04 LTS Precise Pangolin (precise)
    12.10 Quantal Quetzal (quantal)
    13.04 Raring Ringtail (raring)
    13.10 Saucy Salamander (saucy)
    14.04 LTS Trusty Tahr (trusty)
    14.10 Utopic Unicorn (utopic)
    15.04 Vivid Vervet (vivid)
    15.10 Wily Werewolf (wily)
    16.04 LTS Xenial Xerus (xenial)
    16.10 Yakkety Yak (yakkety)
    17.04 Zesty Zapus (zesty)
    17.10 Artful Aardvark (artful)
    18.04 LTS Bionic Beaver (bionic)
    18.10 Cosmic Cuttlefish (cosmic)

Debian (debian):

    4.0 etch (etch)
    5.0 lenny (lenny)
    6.0 squeeze (squeeze)
    7 wheezy (wheezy)
    8 jessie (jessie)
    9 stretch (stretch)
    10 buster (buster)

LinuxMint (linuxmint):

    16 petra (petra)
    17 qiana (qiana)
    17.1 rebecca (rebecca)
    17.2 rafaela (rafaela)
    17.3 rosa (rosa)
    18 sarah (sarah)
    18.1 serena (serena)
    18.2 sonya (sonya)
    18.3 sylvia (sylvia)
    19 tara (tara)
    19.1 tessa (tessa)

Raspbian (raspbian):

    7 wheezy (wheezy)
    8 jessie (jessie)
    9 stretch (stretch)
    10 buster (buster)

elementary OS (elementaryos):

    0.1 jupiter (jupiter)
    0.2 luna (luna)
    0.3 freya (freya)
    0.4 loki (loki)
    0.5 juno (juno)

If you don't see your distribution or version here, email us at support@packagecloud.io.

ttx commented 5 years ago

I was about to ask packagecloud again for the OSS account (no news from them since the last message) -- before I do that, please confirm that these are limitations we can live with :)

jodh-intel commented 5 years ago

Thanks for the info @marcov - as you say, sounds like most of those points can be dealt with. The CLI tool requiring user input is a bit... surprising though. What is the user-interaction prompting for ooi?

I thought the days of writing expect scripts to automate such unruly tooling were long over, but maybe not... ;)

@ttx - let's see what @marcov feels, but I think "we're good".

marcov commented 5 years ago

What is the user-interaction prompting for ooi?

When pushing a package on a repo, it is asking for what distro the package is. However, after reading some more documentation, you can push to a repository/disto/version and this does not require any user interaction.

I still need to do some test with non-x86 architectures, just to make sure multi-arch is handled properly. After this, we should be all set.

jcvenegas commented 5 years ago

@ttx do we have an account to start to push and to intial testing with kata repos?

ttx commented 5 years ago

@jcvenegas no -- the account apparently needs to be originally set up by packagecloud as an OSS account. Last month when I asked they were saying they had a backlog for doing that. I was about to ping them again for status but decided to pause until I get the greenlight that it would work for us from @marcov. Happy to do that now if @marcov confirms.

marcov commented 5 years ago

@ttx please go on. My overall feeling is that packagecloud is good enough for our use cases.

ttx commented 5 years ago

OK, I just pinged their support again...

jcvenegas commented 5 years ago

Thanks to the information provided by@marcov about how to do the rync to obs servers and useage of cli tool I tested the steps he provided by my own in a testing account.

I used two methods to download packages: 1)wget

wget --no-parent  -r https://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/stable-1.5/  --accept '*.rpm,*.deb' --reject '*.src.rpm'

Downloaded: 447 files, 907M in 26m 13s (591 KB/s)

2)rsync

rsync --progress -rlptH rsync.opensuse.org::opensuse-full-really-everything-including-repositories/opensuse/repositories/home:/katacontainers:/releases:/x86_64:/stable-1.5 "${PWD}/pkgs" --delete-after -hi --stats --exclude='**/*.src.rpm' --include='**/*.rpm' --include='**/*.deb' --include='*/' --exclude='*'

total size is 1.33G speedup is 1.00 in 15m19.821s

I feel more comfortable with rsync for two reasons:

The CLI is simple to use to do push I still need a way to say “I just want to keep the the latest rpm from 1.5.x) to avoid use more storage than we will have.

Testing) I uploaded packages for fedora and Ubuntu

And tested if I can query for different versions and install an specific kata version, the answer is yes

I think same apply for the rest of the distros.

jodh-intel commented 5 years ago

Any update on this folks?

ttx commented 5 years ago

Oh sorry I did not update that. The account was fully set up, with @egernst @marcov and @jcvenegas holding the keys. I think it can be closed?

jodh-intel commented 5 years ago

Ah - thanks @ttx!

Before we close this though, are we at the point where we can switch (new) users to using the packagecloud URL (https://github.com/kata-containers/documentation/issues/466)?

Speaking of that URL, could someone share it please as I cannot see it listed here? :)

jodh-intel commented 5 years ago

Huh, whilst searching around obs, I came across:

https://software.opensuse.org/package/kata-runtime

... which lets you download all versions of the runtime since 1.1.1...?!?

/cc @egernst, @jcvenegas.

grahamwhaley commented 5 years ago

/cc @marcov who I suspect knows what/where/why that is?

marcov commented 5 years ago

About the current status, I'm going to submit soon a tool handle the synchronization from OBS to Packagecloud. This is supposed to be run periodically (e.g. every 24 hours), so it should be deployed somewhere. I think @jcvenegas is looking into this.

After that's done, we can switch the doc to the new URLs.

Huh, whilst searching around obs, I came across:

https://software.opensuse.org/package/kata-runtime

That's just some crawling of the OBS projects to have a user-friendly interface for installation. It's picking up packages from the master and stable version, and probably some outdated packages too :)

jodh-intel commented 5 years ago

Looks like we might be able to recreate old releases (on packagecloud) from these crawls though?

marcov commented 5 years ago

That's defintely possible, if there's any interest in doing it.

One minor issue is that the list of distros we build have a bit changed over the time, so the version vs distro matrix would need to be documented.

Also we need to consider that the total space is 25GB, and if we are going to include on Packagecloud all architectures, we may run out of space.

E.g. at the moment binary packages takes ~ 730MB per release, so considering 4 architectures we would have just enough space for 8 releases.

marcov commented 5 years ago

One thing I was wondering that maybe @egernst can anwer:

jodh-intel commented 5 years ago

Any update on this folks? Do we have a rough ETA for when we could start updating the install docs to point to a packageloud URL?

marcov commented 5 years ago

Thank you @jodh-intel for the ping. We basically need to deploy kata-pkgsync somewhere, so that it could periodically synchronize packages from OBS to Packagecloud.

I can take care of this, but I am not familiar enough with the cloud infrastructure we have here. AFAIK @jcvenegas is using Azure for periodic OBS builds, so maybe that would be an option.

egernst commented 5 years ago

@jcvenegas can you provide a recommendation for @marcov