Open zhoufenqin opened 1 year ago
The builder message shows 'could not find buildpack with id 'tanzu-buildpacks/java-azure''.
And the kpack-controller error logs:
{"level":"error","ts":"2023-02-27T02:03:21.507046469Z","logger":"controller","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"9343a26-dirty","knative.dev/kind":"builders.kpack.io","knative.dev/traceid":"6bf87daa-0ae4-46d6-a7a5-611947c07f1a","knative.dev/key":"build-service/default.default.1","duration":2.632799627,"error":"could not find buildpack with id 'tanzu-buildpacks/java-azure'","errorVerbose":"could not find buildpack with id 'tanzu-buildpacks/java-azure'\ngithub.com/pivotal/kpack/pkg/cnb.(*StoreBuildpackRepository).findBuildpack\n\tgithub.com/pivotal/kpack/pkg/cnb/store_buildpack_repository.go:71\ngithub.com/pivotal/kpack/pkg/cnb.(*StoreBuildpackRepository).FindByIdAndVersion\n\tgithub.com/pivotal/kpack/pkg/cnb/store_buildpack_repository.go:23\ngithub.com/pivotal/kpack/pkg/cnb.(*RemoteBuilderCreator).CreateBuilder\n\tgithub.com/pivotal/kpack/pkg/cnb/create_builder.go:59\ngithub.com/pivotal/kpack/pkg/reconciler/builder.(*Reconciler).reconcileBuilder\n\tgithub.com/pivotal/kpack/pkg/reconciler/builder/builder.go:145\ngithub.com/pivotal/kpack/pkg/reconciler/builder.(*Reconciler).Reconcile\n\tgithub.com/pivotal/kpack/pkg/reconciler/builder/builder.go:97\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:491\nruntime.goexit\n\truntime/asm_amd64.s:1571","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20220816153547-f78a00694307/controller/controller.go:491"}
I have a founding that Tanzu Java Azure Buildpack takes much longer than other buildpacks, such as .NET Core Buildpack.
Java Azure Builder takes 6min49
.Net Core Builder takes 2min20
It looks like the long time to reconcile builders might be due to the buildpacks in the clusterstore being in a different registry than where the builder is being written to. This results in the kpack controller having to stream all those images to the registry during builder reconciliation. A mitigation for this is to relocate the buildpack images to the builder registry.
@tomkennedy513 For our used case, it's difficult to relocate the buildpack images to the builder registry. And we used two registries all the time but it didn't take so much time to wait for the builder ready. Is that possible to shorten the time?
Why not delete your limit?
limits/cpu: 100m
will make you wait a long time
In our K8s cluster, we notice that create a builder and wait for it to be ready will cost around 12 mins on average. Not sure which step cost most of the time.
Env
Kpack-controller configuration
Builder configuration
The create timestamp shows creationTimestamp: "2023-02-22T02:56:57Z"
The lastTransitionTime shows lastTransitionTime: "2023-02-22T03:08:20Z"
The kpack log shows:
see these two lines {"level":"info","ts":"2023-02-22T02:56:56.72539479Z","logger":"controller","caller":"controller/controller.go:550","msg":"Reconcile succeeded","commit":"9343a26-dirty","knative.dev/kind":"clusterstores.kpack.io","knative.dev/traceid":"aef7fed9-83c9-4be7-b3e8-5ece7ed0d0ce","knative.dev/key":"default","duration":2.210942879}
{"level":"info","ts":"2023-02-22T03:08:20.233470823Z","logger":"controller","caller":"controller/controller.go:550","msg":"Reconcile succeeded","commit":"9343a26-dirty","knative.dev/kind":"builders.kpack.io","knative.dev/traceid":"6080c19b-f298-4fd9-91e8-75938fbd6229","knative.dev/key":"build-service/default.default.1","duration":683.0590809}
I know if the builder image size is large, then it may cost longer time to upload the image, but it won't cost 10 mins.
Which part cost the longest time to reconcile the builder?