search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
301 stars 115 forks source link

kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: #659

Open naveenspen14 opened 3 years ago

naveenspen14 commented 3 years ago

Hi Team,

We are installing cf-for-k8s on vmware environment. We are getting the below error through kapp.

$ kapp deploy -a cf -f cf4k8s_setup/cf-for-k8s-rendered.yml -y Target cluster 'https://:6443' (nodes:master-0-cf4k8s01, 4+)

Changes

Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri (cluster) bionic-stack ClusterStack 0/1 t 5h - - reconcile ok - ^ cf-buildpack-store ClusterStore 0/1 t 5h - - reconcile ok - ^ defaults.webhook.kpack.io MutatingWebhookConfiguration - 5h update - reconcile ok - ^ istiod-istio-system ValidatingWebhookConfiguration - 5h update - reconcile ok - ^ validation.webhook.kpack.io ValidatingWebhookConfiguration - 5h update - reconcile ok - cf-workloads-staging cc-kpack-registry-service-account ServiceAccount - 5h update - reconcile ok - ^ cf-default-builder Builder - - create - reconcile - - kpack webhook-certs Secret - 5h update - reconcile ok -

Op: 1 create, 0 delete, 5 update, 2 noop Wait to: 8 reconcile, 0 delete, 0 noop

11:40:25PM: ---- applying 1 changes [7/8 done] ---- 11:41:26PM: create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging

kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: Saving record of last applied resource: Updating resource builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: admission webhook "validation.webhook.kpack.io" denied the request: validation failed: invalid value: “cf4k8s”/cf-default-builder: spec.tag (reason: BadRequest)

The interesting part is, cf api <> is a success and but couldn't able to push any apps.

Also, we not seeing any build packs after installation. Do we need to install build packs separately through packeto.

cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/177859822

The labels on this github issue will be updated when the story is started.

jimconner commented 3 years ago

The cf-default-builder container image is uploaded to your specified registry as part of the deployment. From the error message you've got, I'm guessing that the registry didn't like the request that it got sent due to quoting within the spec.tag... invalid value: “cf4k8s”/cf-default-builder: - I'm guessing here, but it doesn't look right ot me that 'cf4k8s' is quoted, but the bits after the slash is not. Maybe something quoted in the manifest that shoudn't be?

naveenspen14 commented 3 years ago

Thanks, Jimconner.

I'm still getting the same issue after removing the quotes to repository_prefix: cf4k8s.

7:45:30PM: ---- waiting on 2 changes [305/308 done] ---- 7:45:30PM: ongoing: reconcile clusterstack/bionic-stack (kpack.io/v1alpha1) cluster 7:45:30PM: ^ No failing or successful conditions found 7:46:28PM: ongoing: reconcile clusterstore/cf-buildpack-store (kpack.io/v1alpha1) cluster 7:46:28PM: ^ No failing or successful conditions found 7:46:30PM: ---- waiting on 2 changes [305/308 done] ---- 7:46:30PM: ongoing: reconcile clusterstack/bionic-stack (kpack.io/v1alpha1) cluster 7:46:30PM: ^ No failing or successful conditions found

kapp: Error: Timed out waiting after 15m0s

one of the observations is, the pod ccdb-migrate-6tkjr is in a completed state but an error with the volume mount.

Events: Type Reason Age From Message

Normal Scheduled 11m default-scheduler Successfully assigned cf-system/ccdb-migrate-6tkjr to k8s-worker-2-cf4k8s03 Normal Created 10m kubelet, k8s-worker-2-cf4k8s03 Created container istio-init Normal Started 10m kubelet, k8s-worker-2-cf4k8s03 Started container istio-init Normal Pulling 10m kubelet, k8s-worker-2-cf4k8s03 Pulling image "index.docker.io/istio/proxyv2:1.7.3" Normal Pulled 10m kubelet, k8s-worker-2-cf4k8s03 Successfully pulled image "index.docker.io/istio/proxyv2:1.7.3" Normal Started 10m kubelet, k8s-worker-2-cf4k8s03 Started container istio-proxy Normal Created 10m kubelet, k8s-worker-2-cf4k8s03 Created container istio-proxy Normal Pulling 10m kubelet, k8s-worker-2-cf4k8s03 Pulling image "cloudfoundry/cloud-controller-ng@sha256:5ee75f427b8859eb35e7c9449992ccd4fb4c3dbd37db95d1ffac02a35db12553" Normal Pulled 10m kubelet, k8s-worker-2-cf4k8s03 Successfully pulled image "cloudfoundry/cloud-controller-ng@sha256:5ee75f427b8859eb35e7c9449992ccd4fb4c3dbd37db95d1ffac02a35db12553" Normal Created 10m kubelet, k8s-worker-2-cf4k8s03 Created container run-migrations Normal Started 10m kubelet, k8s-worker-2-cf4k8s03 Started container run-migrations Normal SandboxChanged 10m kubelet, k8s-worker-2-cf4k8s03 Pod sandbox changed, it will be killed and re-created. Normal Pulling 10m (x2 over 11m) kubelet, k8s-worker-2-cf4k8s03 Pulling image "index.docker.io/istio/proxyv2:1.7.3" Normal Pulled 10m (x2 over 10m) kubelet, k8s-worker-2-cf4k8s03 Successfully pulled image "index.docker.io/istio/proxyv2:1.7.3" Warning Failed 10m kubelet, k8s-worker-2-cf4k8s03 Error: cannot find volume "default-token-zh9k6" to mount into container "istio-init"

kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE cf-blobstore cf-blobstore-minio-6d9d86dff5-wtljx 2/2 Running 0 14m cf-db cf-db-postgresql-0 2/2 Running 0 14m _cf-system ccdb-migrate-6tkjr 0/2 Completed 0 11m_ cf-system cf-api-clock-dc89dfc98-9c7q5 2/2 Running 0 11m cf-system cf-api-controllers-6464964cc7-966zr 3/3 Running 0 14m cf-system cf-api-deployment-updater-78cf895cc-bfjnj 2/2 Running 0 14m cf-system cf-api-server-5c58f95fb7-hs7rn 6/6 Running 0 11m cf-system cf-api-worker-64957dc6d4-fglrd 3/3 Running 0 11m cf-system eirini-api-59c8f57956-mllgc 2/2 Running 0 14m cf-system eirini-app-migration-f6t5t 0/1 Completed 0 14m cf-system eirini-event-reporter-595b7fd969-8djhm 2/2 Running 0 14m cf-system eirini-event-reporter-595b7fd969-9nhpk 2/2 Running 0 14m cf-system eirini-task-reporter-54d4b685d4-58bwl 2/2 Running 0 14m cf-system eirini-task-reporter-54d4b685d4-vgwcc 2/2 Running 0 14m cf-system fluentd-7kbcl 2/2 Running 0 14m cf-system fluentd-l5fkt 2/2 Running 0 14m cf-system fluentd-ppv8x 2/2 Running 0 14m cf-system fluentd-rqc25 2/2 Running 0 14m cf-system fluentd-sk4pq 2/2 Running 0 14m cf-system instance-index-env-injector-5fff98685b-b2rd5 1/1 Running 0 14m cf-system log-cache-backend-759d9b7797-mp8rf 3/3 Running 0 14m cf-system log-cache-frontend-c68f7f45f-gd2tc 3/3 Running 0 14m cf-system metric-proxy-5b48fbcb56-795jm 2/2 Running 0 14m cf-system routecontroller-69586ffd46-w8msq 2/2 Running 0 14m cf-system uaa-7bbdbff88f-gw596 3/3 Running 0 14m cf-workloads restart-workloads-for-istio1-7-3-6qbk5 0/1 Completed 0 14m istio-system istio-ingressgateway-4ls6r 2/2 Running 0 15m istio-system istio-ingressgateway-5vs4v 2/2 Running 0 15m istio-system istio-ingressgateway-jp49t 2/2 Running 0 15m istio-system istio-ingressgateway-lrk92 2/2 Running 0 15m istio-system istiod-89bc798f5-vd7zd 1/1 Running 0 15m kpack kpack-controller-5c9b8fcc97-c6pjw 2/2 Running 0 14m kpack kpack-webhook-555bf54bc4-ccp7h 2/2 Running 0 14m

kubernetes version: v1.18.4 cf-for-k8s version: v3.0.0 os: centos 7 kernel version: 5.9.1

cf buildpacks Getting buildpacks...

buildpack position enabled locked filename stack No buildpacks found

Here I'm using artifactory docker repository as an app registry.

Kindly help me.

naveenspen14 commented 3 years ago

Request you to help me on this. 2:11:26PM: fail: reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging 2:11:26PM: ^ Encountered failure condition Ready == False: (message: stack bionic-stack is not ready)

kapp: Error: waiting on reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: Finished unsuccessfully (Encountered failure condition Ready == False: (message: stack bionic-stack is not ready))

naveenspen14 commented 3 years ago

Looks like issue is with connectivity with docker hub.

Namespace cf-workloads-staging Name cf-default-builder Kind Builder Status conditions:

Namespace (cluster) Name bionic-stack Kind ClusterStack Status buildImage: {} conditions:

Now I have changed app_register to docker hub.

app_registry: hostname: https://hub.docker.com/ repository_prefix: "hub.docker.com/cf4k8s" username: "**" password: "*****"

12:58:02AM: ongoing: reconcile clusterstack/bionic-stack (kpack.io/v1alpha1) cluster 12:58:02AM: ^ No failing or successful conditions found 12:58:02AM: fail: reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging 12:58:02AM: ^ Encountered failure condition Ready == False: (message: stack bionic-stack is not ready) 12:58:02AM: debug: CommandRun: end (10.519357212s)

kapp: Error: waiting on reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: Finished unsuccessfully (Encountered failure condition Ready == False: (message: stack bionic-stack is not ready))

Appreciate your help.

jimconner commented 3 years ago

message: 'Get "https://index.docker.io/v2/": read tcp 10.244.181.113:54198->52.55.43.248:443: read: connection reset by peer'

Do you have a firewall or proxy blocking your access to Dockerhub or something like that? cf-for-k8s needs to push the cf-default-builder image up to the registry that you defined, and it would appear that it can't get a connection to Dockerhub.

Hope that helps.

naveenspen14 commented 3 years ago

Thanks, Jim for the details. Docker registry details are working fine while tested through hack scripts. But the same is failing through kapp.

cat registry.yaml

app_registry: hostname: https://index.docker.io/v1/ repository_prefix: XXXXXX username: "****" password: "***"

[myhome@k8s hack]$ bash validate-registry-access.sh registry.yaml WARNING: The hack scripts are intended for development of cf-for-k8s. They are not officially supported product bits. Their interface and behavior may change at any time without notice. registry_host -> https://index.docker.io/v1/ username -> ** repo -> nvn4u81 docker_tag -> XXXXXX/cfk8s-test-delete-me password -> ** logging into dockerhub with username and password WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /home/myhome/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded building tiny test docker image... docker push-ing XXXXXX/cfk8s-test-delete-me to test push access... Confirmed push access to dockerhub registry

any other alternative, instead of pushing the cf-default-builder image to the registry. Install only on the K8s environment.

jimconner commented 3 years ago

Hi Naveen. When I was trying out cf-for-k8s a few months ago (haven't touched it for a while), I used harbor as my registry. I did this because my broadband speed is limited and because I don't have a paid-for dockerhub account and therefore was hitting the container pull limits that they introduced back in November. My deployments of cf-for-k8s and harbor were running on top of minikube.

Here's the notes I made from when I set harbor up... Maybe there'll be something in here that helps you. 

Generate trusted cert for Harbor using LetsEncrypt Certbot:
certbot certonly --manual --preferred-challenges dns \
    -m jim@mydomain.org.uk -vvv --agree-tos -d harbor.mydomain.org.uk \
    --work-dir ~/tmp/cert/ --logs-dir ~/tmp/cert/logs --config-dir ~/tmp/cert/config
# do the DNS TXT record dance to make LetsEncrypt trust us and generate the cert
cd ~/tmp/cert/config/live/harbor.mydomain.org.uk
cp fullchain.pem tls.crt
cp key.pem tls.key
kubectl create secret tls harbor --cert tls.crt --key tls.key # Store the secret in kubes as tls and call it 'harbor'. We use the secret for deploying harbor via helm

Install Harbor on top of minikube:
# https://github.com/goharbor/harbor-helm/
helm repo add harbor https://helm.goharbor.io
helm install helm-harbor harbor/harbor \
    --set expose.type=loadBalancer \
    --set expose.tls.auto.commonName=harbor.mydomain.org.uk \
    --set domain=harbor.mydomain.org.uk \
    --set externalURL=https://harbor.mydomain.org.uk \
    --set expose.tls.certSource=secret \
    --set expose.tls.secret.secretName=harbor

Default User/Pass: admin/Harbor12345

Log in at https://harbor.mydomain.org.uk # Yay for valid certs
Create user 'cf-for-k8s' ... Password: OhNoTheInternetKnowsMyPasswords
Create project for 'cf-images'. We'll use this for the kubes images. Add cf-for-k8s user as a member. 'Maintainer' permissions seems to work.
Create a project for 'cf-for-k8s'. We'll use this for apps. Add cf-for-k8s as a mebmber as before.

cf-for-k8s:
Follow steps for creating cf-values file : https://github.com/cloudfoundry/cf-for-k8s/blob/develop/docs/getting-started-tutorial.md

In my cf-values.yml I had the following for app_registry

app_registry:
  hostname: https://harbor.mydomain.org.uk/v2/
  repository_prefix: "harbor.mydomain.org.uk/cf-for-k8s"
  username: "cf-for-k8s"
  password: "OhNoTheInternetKnowsMyPasswords"
naveenspen14 commented 3 years ago

Thanks, Jim. I will try the same.

Birdrock commented 3 years ago

@naveenspen14 Were you able to resolve your issue?

naveenspen14 commented 3 years ago

Hi Jim & Birdrock,

I couldn't resolve this issue in VMware proxy environment. Except for docker-registry connectivity, others are working fine. Due to this issue, we couldn't be able to push any apps. But it worked seamlessly on Tencent cloud. Currently trying on AWS.

drpdishant commented 3 years ago

@jimconner I have installed Harbor using helm on Kubernetes, but somehow the blob access is not working with it, I checked pushing images, its successful but its show error while accessing blob

docker push core.registry.openxcell.dev/cf-for-k8s/alpine
Using default tag: latest
The push refers to repository [core.registry.openxcell.dev/cf-for-k8s/alpine]
b2d5eeeaba3a: Layer already exists 
received unexpected HTTP status: 500 Internal Server Error

Due to this I am getting error in kapp deploy

3:42:59PM:  ^ Encountered failure condition Ready == False:  (message: POST https://core.registry.openxcell.dev/v2/cloudfoundry/cf-default-builder/blobs/uploads/: UNKNOWN: unknown error; map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir Path:/storage/docker/registry/v2/repositories/cloudfoundry/cf-default-builder]])

kapp: Error: waiting on reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging:
  Finished unsuccessfully (Encountered failure condition Ready == False:  (message: POST https://core.registry.openxcell.dev/v2/cloudfoundry/cf-default-builder/blobs/uploads/: UNKNOWN: unknown error; map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir Path:/storage/docker/registry/v2/repositories/cloudfoundry/cf-default-builder]]))