Keep older versions in OBS

[matpen]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

I have noticed that, whenever a new podman (or buildah) release comes out, the PPA is updated with the new one, but the old ones are removed. This makes it hard to pin versions in install scripts.

Is it possible to keep older versions in the PPA (at least a few back)?

Steps to reproduce the issue:

1. Install by pinning version: apt-get -yq install podman="1.5.0-2~ubuntu18.04~ppa5"

2. Notice that, after a release, the above code fails with E: Version '1.5.0-2~ubuntu18.04~ppa5' for 'podman' was not found

(Environment details omitted because not relevant)

[baude]

@lsm5 ptal

[lsm5]

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

@baude @mheon once we have a new release, we stop supporting the older ones, yes?

[mheon]

Yeah, we don't generally do backports to older versions

[lsm5]

@matpen any reason you need to pin podman version?

[matpen]

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

I think that it can be easily accomplished by using file names. At least this is what I see on the kernel ppa and gcc ppa.

For example, I used to install gcc 7 on Ubuntu 16.04 like this:

add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get -yq update
apt-get -yq install gcc-7 g++-7

@matpen any reason you need to pin podman version?

When using podman in production, things might break on system update. For example, when upgrading from 1.5.0 to 1.5.1 the default pod hostname changed: I am making use of this information inside the containers, so the "old" config will not work with the "new" podman.

For this reason I prefer to keep podman at a defined version, which I tested on my development environment, until I have the time to upgrade.

@baude @mheon once we have a new release, we stop supporting the older ones, yes?

Unfortunately I have never setup a PPA myself, so I cannot provide direct experience. But one possible way to support both approaches could be to simply deploy as follows:

• publish one podman package, which is the "default" one and always points to the latest release;
• publish one "versioned copy" of the same filename;
• on the next release, replace the default, but keep the old versioned copy.

[lsm5]

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

I think that it can be easily accomplished by using file names. At least this is what I see on the kernel ppa and gcc ppa.

For example, I used to install gcc 7 on Ubuntu 16.04 like this:

add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get -yq update
apt-get -yq install gcc-7 g++-7

If upstream isn't supporting older releases, I'd rather not commit to this either.

There's another request to publish multiple PPAs where one has only the latest release, while the others have RCs/nightlies/hourlies. Perhaps you could use those (once published) to test upcoming versions?

When using podman in production, things might break on system update. For example, when upgrading from 1.5.0 to 1.5.1 the default pod hostname changed: I am making use of this information inside the containers, so the "old" config will not work with the "new" podman.

@mheon can correct me if I'm wrong, but the old behaviour was a bug. So, you should keep the new one.

[matpen]

If upstream isn't supporting older releases, I'd rather not commit to this either.

This has nothing to do with supporting old version. The requirement here is that an older version is still available through the convenience of the PPA, not that bugfixes are backported. In fact, this is exactly what should not happen (at least not without changing the version number).

There's another request to publish multiple PPAs where one has only the latest release, while the others have RCs/nightlies/hourlies. Perhaps you could use those (once published) to test upcoming versions?

This is not always an option: when the version is released, the PPAs are rotated, and it is not always possible to keep the production servers at the same pace.

the old behaviour was a bug. So, you should keep the new one.

In a perfect world, I would obviously always want the latest bugfree version. But in the real world, a new version will have incompatibilities. In this particular case the change, while definitely being an improvement, would break my containers.

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

I just remembered that the docker project maintains exactly the envisioned setup:

[ ... ]
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
[ ... ]
sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io

[lsm5]

Hmm, I think I can set this up, but quite likely won't be on the current launchpad ppa, but some other location where I have more control. I'll try and get back on this asap.

[afbjorklund]

I couldn't get any 1.5 nor 1.6 version to work properly on Ubuntu 16.04, so ended up having to rebuild v1.4.4 from source (since I had deleted the local .deb package)

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

@lsm5 Any update on this?

[lsm5]

@lsm5 Any update on this?

Not yet, but hopefully once other distractions are outta the way ..

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@lsm5 What should we do with this?

[vrothberg]

The cross-distro packaging is now happening in the public OBS instance -> https://build.opensuse.org/project/subprojects/devel:kubic:libcontainers. The links to older package versions should be static and can hence be used in install scripts.

@lsm5 @sysrich what do you think?

[sysrich]

There are multiple approaches we could do

1. Versioned package names (eg. podman1.5 , podman1.6, etc) in the existing OBS Project.

This would be the easiest for the described usecase, but would be a significant pain for many users as there wouldn't be a simple 'apt-get podman' option..and how would people know when to move to a new version? Such things aren't a concern when talking about gcc versions but I dont think podman can be really compared to a core system compiler.

It would also be a pain for us to maintain and build, with a never ending growing amount of packages with potential build issues.

2. In RPM-land we have a concept of a 'maintenance repository' which keeps all previous versions of packages under the original name. I have NO IDEA if such things as possible with debs, but if they are OBS should be able to support the extra metadata..but it would require the modification of scripts to take benefit of such a feature, such as on openSUSE where "zypper in podman" would have to be "zypper in podman=1.5" to guarantee you only get podman version 1.5.

If I was @lsm5 I wouldn't like doing either regularly, but I could see Option 1 being useful as an exception to the norm for cases like the 'old version 1.4.4 only works on still in-support LTS distro release'

[vrothberg]

@lsm5 WDYT? Feel free to close.

[matpen]

Feel free to close.

I do not think that closing this is a good idea. In fact, from the point of view of any serious deployment, it is a very urgent issue: it pretty much makes the repos unusable. As @afbjorklund said, the only viable alternative is to build from source, which can prove tricky for some sysadmins.

Is there any technical reason why the solution offered by the docker repos is not applicable in this case?

[vrothberg]

do not think that closing this is a good idea. In fact, from the point of view of any serious deployment, it is a very urgent issue: it pretty much makes the repos unusable.

Can you elaborate a bit more on that? We provide a repository with updated and fresh builds but don't have a good approach for keeping builds around.

[lsm5]

In RPM-land we have a concept of a 'maintenance repository' which keeps all previous versions of packages under the original name. I have NO IDEA if such things as possible with debs, but if they are OBS should be able to support the extra metadata..but it would require the modification of scripts to take benefit of such a feature, such as on openSUSE where "zypper in podman" would have to be "zypper in podman=1.5" to guarantee you only get podman version 1.5.

@sysrich could you please share some info/examples on setting up a maintenance repository?

@matpen in docker's case, they probably maintain the entire repo by themselves, which I'd rather not. I guess a maintenance repo that @sysrich mentioned could address your case, we can check if that's doable for debs too. Also, for debs, I will keep the packaging sources available for older versions, even though the older debs themselves might get wiped out.

[matpen]

Can you elaborate a bit more on that? We provide a repository with updated and fresh builds but don't have a good approach for keeping builds around.

I think that the problem has been outlined sufficiently in this issue: since there is no way to pin a version, sysadmins only have the following options in production:

• upgrade podman each time they reinstall or upgrade a system (which is dangerous - see for example here)
• not use the repos and instead build from source (out of reach for some, and 10x more work in any case).

In other words, unless using the second options, sysadmins have no control on which version of podman they are running, which makes maintenance a nightmare.

in docker's case, they probably maintain the entire repo by themselves, which I'd rather not

@lsm5 could you please expand this? How are you maintaining the repo? Seems like something is preventing it to enable a feature which is otherwise available via apt-get (version pinning).

we can check if that's doable for debs too. Also, for debs, I will keep the packaging sources available for older versions, even though the older debs themselves might get wiped out.

This should be the case, as with many other repos version pinning is always available: apt-get install package=1.2.3. I would like to give more advice, but for that I need more knowledge about the particular infrastructure (see my previous question).

[rhatdan]

@lsm5 Any progress on this issue? Really should not be an upstream issue, but I guess there is no other place to report it.

[lsm5]

@lsm5 could you please expand this? How are you maintaining the repo? Seems like something is preventing it to enable a feature which is otherwise available via apt-get (version pinning).

@matpen see: https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman . The debian build cares about podman_1.8.0~2.dsc and podman_1.8.0~2.tar.gz. Once it finds a new version, it gets rid of the old debs. If it finds multiple dsc and tar.gz (not the _service:blah ones), it'll refuse to build anything for debian, so that's why I append .old for older files.

@lsm5 Any progress on this issue? Really should not be an upstream issue, but I guess there is no other place to report it.

Nothing yet. I'll investigate if maintenance repos can be setup or nag @sysrich for help. Also IIRC, @matpen had some job running on their side to download and cache builds so maybe that can be shared with the community if possible for whoever needs version pinning.

[sysrich]

@lsm5 sorry for the delay in getting back to you - busy week during Hackweek..

https://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.maintenance_setup.html is the documentation on how to set up a maintenance project

https://openbuildservice.org/help/manuals/obs-reference-guide/obs.glossary.html#obs.glos.maintenance_project

https://openbuildservice.org/help/manuals/obs-reference-guide/obs.glossary.html#obs.glos.incident

The project needs to have some attributes set, such as OBS:MaintenanceIdTemplate and OBS:MaintenanceProject (See https://build.opensuse.org/attribs/openSUSE:Maintenance for examples)

Besides that I'm not an expert, the maintenance process is one I've done my best to avoid as much as I can in my life with my love for rolling releases :)

[matpen]

see: https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman

I understand now: you are using a third party service which automates the build, and discards older versions automatically. I scanned their docs, but could not find anything useful. Maybe its worth asking them via IRC?

Also IIRC, @matpen had some job running on their side to download and cache builds

Uhmm this was not me... unless I am missing something.

[lsm5]

Thanks @sysrich!

I understand now: you are using a third party service which automates the build, and discards older versions automatically. I scanned their docs, but could not find anything useful. Maybe its worth asking them via IRC?

@matpen would you be willing to help us out in setting this up? I can add you as a co-maintainer..

Also IIRC, @matpen had some job running on their side to download and cache builds

@matihost did you have some job for caching builds? Is that something you could share publicly?

[matpen]

@matpen would you be willing to help us out in setting this up? I can add you as a co-maintainer..

Not sure how much time I will be able to spare, but I am willing to help!

[alitvak69]

@lsm5 @matpen Any updates on the topic? At this point, I can no longer download the source from SuSe build service (no download links) for old source packages. Ubuntu Packages for 1.8.1 rc2 are no longer being built either (no debs packages in testing repo)

[lsm5]

@alitvak69 sorry, there were some dependency issues on the builder image I used for CI. Fixed now, 1.8.1-rc2 is available at https://build.opensuse.org/package/show/devel:kubic:libcontainers:testing/podman . There are some build issues for 32-bit which I've already informed upstream about.

@matpen let me know once you have an obs/opensuse account. I'll add you as co-maintainer on the testing project first, once things are setup there, we can replicate it on the stable project.

[matpen]

@lsm5 I registered on openSUSE and can now login to OBS. My username is the same as here on GitHub. Let me know if you need further info to set things up.

[lsm5]

I found out from the OBS admins that we can see the older files by going to the Revisions tab for a package and clicking Browse Source for a particular revision. Should be doable via CLI as well, but I don't know the exact commands atm. So, I'll no longer be having any .old files in there.

[sysrich]

@lsm5 you mean like osc co devel:kubic:libcontainers:stable podman -r $REV ?

[medyagh]

I also like this to happen, this broke minikube docker file, because we were pinning the podman version but now we can no longer re-build the image

[afbjorklund]

I was able to find the 1.8.2~2 package in https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman?rev=137 eventually.

But I don't see how to download the podman_1.8.2~2.tar.gz file with the debian directory from there ? So will have to some guessing of the actual packaging, anyway.

The quoted repositorites seem to be outdated.

Vcs-Git: git://github.com/lsm5/libpod.git

I was looking at https://gitlab.com/lsm5/podman, but it only seemed to have the latest ?

Like the "debian" branch: https://gitlab.com/rhcontainerbot/podman/-/commits/debian

[alitvak69]

You need to sign up with suse build service and login with your account. Then you will be able download .gz

On Sun, May 24, 2020, 9:56 AM Anders Björklund notifications@github.com wrote:

I was able to find the 1.8.2~2 package in https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman?rev=137 eventually.

But I don't see how to download the podman_1.8.2~2.tar.gz file with the debian directory from there ? So will have to some guessing of the actual packaging, anyway.

The quoted repositorites seem to be outdated.

Vcs-Git: git://github.com/lsm5/libpod.git

I was looking at https://gitlab.com/lsm5/podman, but it only seemed to have the latest ?

Like the "debian" branch: https://gitlab.com/rhcontainerbot/podman/-/commits/debian

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/containers/libpod/issues/3924#issuecomment-633243169, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABWJN3DIZRLVAXVTUBX3S43RTEYQTANCNFSM4IS7M6KA .

[afbjorklund]

Same story with 1.9, as with the previous 1.5 series (had to rebuild v1.8.2 from source)

Would it be possible to host the stable podman releases somewhere, like crio is doing ?

[afbjorklund]

You need to sign up with suse build service and login with your account.

Will do, but I do think that it would have been a lot easier to use git for this.

And I still have to find some suitable hosting, for my podman deb packages.

[lsm5]

There's experimental work to build deb packages using rpm spec files. I would really like to take our Kubic project in that direction, but not sure when I can find sufficient time for it. I'll probably play with it in the Kubic unstable project whenever I can and migrate the changes to testing and stable projects once they are ready.

I welcome anyone willing to volunteer toward this effort and I can help set them up with access.

[afbjorklund]

There's experimental work to build deb packages using rpm spec files. I would really like to take our Kubic project in that direction, but not sure when I can find sufficient time for it. I'll probably play with it in the Kubic unstable project whenever I can and migrate the changes to testing and stable projects once they are ready.

Not sure that this would help with getting the packages accepted upstream, so that would probably mean a difference betwen the "system" packages and the "vendor" packages - just like with Docker.

Then again, maybe that is the way to go in order to make the latest and greatest available faster. I was more requesting something similar to the RHEL packages, that would actually move a bit slower...

[rhatdan]

@lsm5 Any update on this?

[matpen]

My bad actually. I have been busy at work lately, and could not spare time to dedicate to the issue. But I have not forgotten: I have access to OBS and will have a good look asap.

[lsm5]

Then again, maybe that is the way to go in order to make the latest and greatest available faster. I was more requesting something similar to the RHEL packages, that would actually move a bit slower...

IIUC, podman is now available in debian unstable and hopefully will make its way into the other repos as well. That itself might be the right slower moving build to recommend.

[afbjorklund]

I was able to (re)build my own packages for ubuntu of podman and cri-o and to find apt hosting:

• https://bintray.com/afbjorklund/podman

• https://bintray.com/afbjorklund/cri-o

Good that it is available in Debian (even if 1.6), and I guess the Mac/Win will be back eventually.

[lsm5]

I'm gonna close this one. AFAIK, @siretart is working toward enabling podman on debian as well as ubuntu default repositories. And hopefully once those are available, I won't need to maintain debs on OBS anymore. So, this may no longer be needed anyway and don't think anyone's going to work on this meanwhile.

@matpen please reopen if you disagree.

[matpen]

@lsm5 Sorry I did not manage yet to find the time to work on this one. Anyway, having podman available from the official repos would make it even easier, both for the end user and for the maintainers. Great if this works out!

[siretart]

@siretart is working toward enabling podman on debian as well as ubuntu default repositories.

Done:

• https://tracker.debian.org/pkg/libpod
• https://launchpad.net/ubuntu/+source/libpod

And hopefully once those are available, I won't need to maintain debs on OBS anymore.

You may want to keep your existing work for earlier Distributions. I'm not sure whether I can commit to providing backports for debian 10 or Ubuntu 20.04 or earlier. I imagine the kernel to be particularly challenging.

I wonder whether flatpack or snapd might be an interesting option for distributing podman for earlier distro releases...

[alitvak69]

Unfortunately many people use LTS distros for ubuntu, i.e. 18.04 and 20.04 so unless ubuntu effort can be backported we would need to keep OBS or some other distro longer. IMHO snapd is not a good idea.

[vrothberg]

Reopening due to #8971

[matpen]

Following up from https://github.com/containers/podman/issues/8971, which has been closed in favor of reopening this issue (thank you @vrothberg).

@lsm5 I renew my intention to have a look at the OBS configuration, if you are still willing to support my progress and be patient while I try to find the time for this.

[afbjorklund]

@vrothberg : You might want to rename this to OBS then, since podman is no longer using PPA repositories ?

I think we will have to set up our own package cache, so that we don't have to rebuild from source all the time:

https://github.com/kubernetes/minikube/issues/9552