RFD 165 Discussion

[cburroughs]

"RFD 165 Security Updates for Triton/Manta Core Images"

[trentm]

There is no mechanism to apply security updates to the origin images.

Could we not bump triton-origin-image.git/package.json#version to "1.0.2" and update the pkgsrc versions (at https://github.com/joyent/triton-origin-image/blob/master/images.json#L10-L15) to newer locked versions that have the security updates. Then spin a new build of the origin images, publish those, and then update components to use the new BASE_IMAGE_UUID?

[cburroughs]

Manually updating the locked version would apply some -- but not all -- security updates. For example, on minimal-multiarch-lts@15.4.1, pkg_add -U curl-7.56.1 will also update openssl (because curl depends on openssl and the openssl version is not locked). However, packages that are not dependees of the origin image set (such as libarchive in this example) would not have their security updates applied. We would still need to do something like pkgin full-upgrade for them, which would upgrade everything anyway.

[rmustacc]

Hi Chris,

Thanks for putting this together. I have a couple of different thoughts on this.

Regarding the question of 'Is it feasible to keep pkg audit clean?' I think the answer is that we can try; however, the answer may be no. pkg audit will report all the vulnerabilities, but that doesn't always mean that a fix is available. I know that jperkin has pointed out that the reason for that is to make sure that a user can understand and audit it.

I don't think we want to fully unpin all packages, there may be some amount of pinning that is still required to distinguish between multiple versions of a package that are present in a repository. I think it'll ultimately be the same effect as what you're proposing, but for some packages, there'll still be some amount of version bump. An example (and I know we don't manage node this way) is that if you install nodejs you'll get node 10.x today on 18.4.0 base64; however, you may want to be on the 4.x, 6.x, 8.x train.

One other concern around full-upgrade is that it may mean you get more than a security fix. That may be fine, but it seems like we still have the full analysis problem, which is then potentially amplified as everything will now have to use new versions of software which may or may not change behavior. This system will probably be easier to manage than the component model; however, I think that it doesn't actually change the amount of effort in testing and required analysis.

Historically, we've also had newer versions of software sometimes make it back into an LTS release for reasons that aren't related to security. So that may also trigger this same upgrade analysis question. That said, if I'm understanding it, the origin image that everything is being based on would need to be rebuilt manually and the act of addressing the security issues and spinning a new one is a manual process so it's unlikely that this would happen without someone being aware. Is that right?

I'm not sure if this is all very clear or not. Let me know if I can clarify some of the above. Thanks for working on this problem.

Thanks Robert

[trentm]

pinning

I think I'm sold on unpinning the (small) set of packages that are included in the origin image. The set is small, e.g. for triton-origin 18.1.0:

            "coreutils-8.26",
            "curl-7.59.0",
            "gsed-4.4",
            "patch-2.7.6",
            "sudo-1.8.22"

and I find it hard to imagine that there is a dependency on those in one of our images that would be broken by an upgrade of those. And as Chris pointed out, each component needs to opt in to a new triton-origin-image anyway via BASE_IMAGE_UUID in their Makefile.

rm's node package example is fair, but we don't have that issue for any of the packages installed in the triton-origin image. pkgsrc packages installed by the individual component build would likely still pin to a particular version for reproducible builds.

origin-image version string

Currently the image version of the origin-image is yoinked out of the package.json field in triton-origin-image. As of this writing there are 5 triton-origin-multiarch-15.4.1@1.0.1 images on updates.joyent.com. To push out updated images with non-conflicting versions we would need a new scheme.

Or we could have to manually maintain a rule that we don't bless a triton-origin-image build of a version that is already "released". https://github.com/joyent/triton-origin-image#how-to-use-triton-origin-images states that one should use one of the released images that are in the "release" channel. There should only be one for a given version there. (That "should" is manual, because there is no updates.jo/IMGAPI mechanism to avoid (name, version) conflicts. Nor does https://github.com/joyent/triton-origin-image#releasing-triton-origin-images describe watching out for that.)

However, I don't object to X.Y.TIMESTAMP if that helps things. Similar issues have been discussed in RFD 68 as you know.

How often to update?

It is a balance between these issues:

• keeping up with security updates in the origin image pkgsrc packages
• adding to maintenance burden updating Triton and Manta components using these origins
• (minor) having a large number of triton-origin image in used by the latest of all Triton images on the usbkey. If they all use a different version, then we lose any size savings benefit of a shared origin.

What if we updated:

1. approx. every 6 months; or
2. earlier if there is a new security issue from pkg audit that was deemed important to any Triton/Manta services

?

[jclulow]

I find it hard to imagine that there is a dependency on those in one of our images that would be broken by an upgrade of those

I would note that we've experienced at least one regression in coreutils in an image we've shipped; the details appear in joyent/node-manta#298.

[cburroughs]

I don't think we want to fully unpin all packages, there may be some amount of pinning that is still required to distinguish between multiple versions of a package that are present in a repository. I think it'll ultimately be the same effect as what you're proposing, but for some packages, there'll still be some amount of version bump. An example (and I know we don't manage node this way) is that if you install nodejs you'll get node 10.x today on 18.4.0 base64; however, you may want to be on the 4.x, 6.x, 8.x train.

In cases where the the difference between versions is such that each major version is a unique package such as nodejs, (or python, openjdk, postgresql, apache) I agree that it is almost always preferable to specify the unique package (python27) and not a meta-package (python). In the case of specifying python27 vs python27-2.7.15 in general I'd lean towards just python27 but I'm sure there are use cases for each and plenty of upstreams with idiosyncratic versioning practices. In this RFD I only meant to suggest un-pinning the half dozen or so packages that are in triton-origin images.

One other concern around full-upgrade is that it may mean you get more than a security fix.

(@jperkin can correct me if I mischaracterize something here. ) I don't doubt that there have been backports that a customer insisted on, are not strictly security fixes, or cases where the only choice from upstream is taking a security fix along with other changes. However, past the 3 month mark Joyent is the one doing all of the maintenance. I'm afraid "too many backports" is not the problem we are likely to be blessed with. Here is the rate of 2017Q4 backports as an example https://github.com/joyent/pkgsrc/commits/joyent/feature/backports/2017Q4

That said, if I'm understanding it, the origin image that everything is being based on would need to be rebuilt manually and the act of addressing the security issues and spinning a new one is a manual process so it's unlikely that this would happen without someone being aware. Is that right?

Regardless of how much attention anyone pays to the triton-origin-image repo, there would still be a CR for each individual component to use the new image.

However, I don't object to X.Y.TIMESTAMP if that helps things. Similar issues have been discussed in RFD 68 as you know.

:-) OS-5798 is getting closer!

approx. every 6 months; or

I think that's a reasonable first stab. Once there is less variance in what we are shipping, "at the same time as sdcnode security update" might also be reasonable.

[jperkin]

Realistically speaking, with the current size of the pkgsrc team, the number of backports will be incredibly low and driven solely by requests or critical issues (e.g. a Heartbleed level event).

I think @jclulow's concern was that an upgrade of a package to fix a security issue may also include unrelated changes that may affect the behaviour of the package. We try to avoid doing that but sometimes it might be unavoidable. Trying to only pull in the security fix without other changes can lead to an even worse situation (e.g. Debian's past issues).

[trentm]

Having some changelog report of what package version changes there have been from one triton-origin image build to the next might be useful for CR'ing and debugging should issues arise.