Closed turkenh closed 2 years ago
I've updated the issue description, as originally I tried to install provider with controller image 🤦
This is still occurring for some folks (see #3061) - we should reconsider the hard-coded timeout value and expose it as a configurable value.
Let's make the context deadline configurable via a flag and also bump the default up a little - maybe to 5 mins for now? Ideally we'd avoid having configurable deadlines for only only a subset of reconcilers so maybe we can set one value for all of them? That said I could be convinced that package revisions are a special case.
We might also consider bumping the default Crossplane deployment's CPU resources up - we currently request 150m
cores by default (a small portion of one core) which is quite miniscule.
@haarchri brought this up today in the community meeting, as he and others in the community are seeing similar behavior in #2849 and feeling the pain around it.
Thanks @haarchri and @jbw976! This is going to be a priority ASAP.
I just chatted with @avalanche123, who is going to take a shot at this! Per my comment at https://github.com/crossplane/crossplane/issues/2564#issuecomment-1165007894 the quick fix here is likely going to be to make this deadline configurable. That said, I think it would be interesting to check in with @hasheddan to see whether he can think of any low hanging fruit to speed up the reconcile process - perhaps parallelizing/batching CRD installs?
We've had 1-2 reports of this still happening, so I'm reopening to dig into that.
Specifically seeing this now (I'm using upbound/crossplane:v1.9.0-up.2
, which should have https://github.com/crossplane/crossplane/pull/3176):
0s Warning SyncPackage providerrevision/crossplane-provider-jet-azure-b7b0db0f74e0 cannot establish control of object: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
Just confirmed I'm seeing the same with upbound/crossplane:v1.9.0-up.2
too. Both tests using kind v1.24.0 clusters.
And yet crossplane/crossplane:v1.10.0-rc.0.31.g781e909f
works as I would expect. 🤔 I guess perhaps this issue is somehow isolated to UXP. Will close and move discussion there.
@negz we see the warning as well in 1.9.0 after a few loops it is working without handmade interactions
I seem to be running into this again trying to do a fresh install of
Image: xpkg.upbound.io/upbound/provider-aws:v0.35.0
Normal BindClusterRole 21m (x46 over 20h) rbac/providerrevision.pkg.crossplane.io Bound system ClusterRole to provider ServiceAccount(s)
Normal ApplyClusterRoles 17m (x68 over 20h) rbac/providerrevision.pkg.crossplane.io Applied RBAC ClusterRoles
Warning SyncPackage 2m23s (x380 over 19h) packages/providerrevision.pkg.crossplane.io cannot establish control of object: Post "https://10.131.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?dryRun=All": context deadline exceeded
What happened?
Cannot install a provider package containing 750+ CRDs.
Trying to install provider-tf-aws built from this commit, but package is never installed.
I am observing context deadline errors in
providerrevisions.pkg.crossplane.io
events:How can we reproduce it?
kubectl crossplane install provider turkenh/provider-tf-aws:daf1e9f7-2
What environment did it happen in?
Crossplane version: v1.3.1 Kubernetes Cluster: Kind - v1.21.1
Possibly related: https://github.com/crossplane-contrib/terrajet/issues/47