cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
300 stars 115 forks source link

Incorporate "full" Paketo build and run Bionic stack images instead of the to-be-deprecated "full-cnb-cf" ones #483

Closed emalm closed 3 years ago

emalm commented 4 years ago

The Paketo buildpacks are consolidating on a single io.buildpacks.stacks.bionic stack ID for all Ubuntu Bionic-based cloud-native buildpack stacks, and will soon deprecate the full-cf and full-cnb-cf stacks that present the org.cloudfoundry.stacks.cflinuxfs3 stack ID. The Bionic-based full stack is functionally identical to the cflinuxfs3 stacks, as they are both Bionic with the cflinuxfs3 set of mixins included, so there should be no impact to buildpack apps that currently run on cf-for-k8s.

cc @kvedurmu, who can provide additional context as needed.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175053873

The labels on this github issue will be updated when the story is started.

kvedurmu commented 4 years ago

Thanks @emalm. Images are published to the following location: Build Image: index.docker.io/paketobuildpacks/build:full-cnb Run Image: index.docker.io/paketobuildpacks/run:full-cnb

emalm commented 4 years ago

FYI @paulcwarren as this is a good candidate for a change to get into cf-for-k8s before its v1.0 release.

paulcwarren commented 4 years ago

@kvedurmu should we consuming these buildpacks from index.docker.io/paketobuildpacks or gcr.io/paketo-buildpacks. To date we have been consuming them from gcr.

kvedurmu commented 4 years ago

You should switch over to the dockerhub locations. We currently publish to both dockerhub and GCR for the full-cf, base, and tiny images for backwards compatibility, but plan on eventually moving away from GCR. Since the full stack is new, we only publish it to dockerhub.

Recommended stack locations can be found here - https://paketo.io/docs/stacks/

paulcwarren commented 4 years ago

More comments on story.

paulcwarren commented 4 years ago

@emalm this doesnt look trivial after all I am afraid. Unless you can spot something we missed.

This fails running the CATs detect suite. Java and php apps fail to start. Both with permission issues attempting to write to the filesystem during startup (example below). In the new run image files are owned as user cnb/1000, whereas with the current run image they are owned by user vcap/2000. Eirini sets up a pod security context, hardcoding the RunAsUser to vcap/2000. Hence, issues if anything attempts to write to those areas of the fs.

Looks like we'll have to chase this one down the stack and ask Eirini to make this configurable and possible to make it part of the OPI DesireLRP API (although, as simple configuration parameter would do us).

cwlbraa commented 4 years ago

This doesn't need to be configurable on the LRP API. K8s knows how to run containers based on their images and images encode UID. K8s even knows how to prevent users from running as UID 0 with runAsNonRoot. Eirini already sets that up. A startup configuration parameter on OPI or the LRP controller would not be sufficient here: most docker images still wouldn't run as UID 1000.

The simplest and most correct solution here is for Eirini to stop hardcoding runAsUser: 2000 and to instead only hardcode runAsNonRoot: true.

emalm commented 4 years ago

FYI @herrjulz @mnitchev re Eirini changes

mike1808 commented 4 years ago

One caveat of not hardcoding the UID in the PodSpec is that the container should expose a user using UID and not username. E.g in the Dockerfile instead of:

RUN adduser --uid $UID --disabled-password $USER
USER $USER

you need to do:

RUN adduser --uid $UID --disabled-password $USER
USER $UID
acosta11 commented 3 years ago

With the consumption of the VCAP UID fix in the Eirini 2.0 release through https://github.com/cloudfoundry/cf-for-k8s/commit/53fda0d5b1c4dce74a4fcd39c80c5da7a58b4908, this issue should now be resolved.