Intermittent deploy failures: Error deploying - error executing `kubectl`

[tobalsgithub]

What happened? From time to time I get the following error when doing devspace deploy.

configurations create_deployments: error deploying configurations: error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -':
configurations Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/manifest.yaml`
mte-metadata-provider create_deployments: error deploying mte-metadata-provider: error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -':
mte-metadata-provider Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/manifest.yaml`
fatal exit status 1

Rerunning devspace deploy seems to work fine, so I don't think it's a problem with the configuration.

This almost always happens when deploying a configuration that has many dependencies. It's acting like the kubectl command is getting an error back from the server that's likely some form of "I'm too busy".

What did you expect to happen instead? Since rerunning the devspace deploy command seems to work, I wonder if there's a way we can automatically retry the commands when they error, perhaps with a backoff.

I don't see any options that I can pass to kubectl to handle this, but I could be missing something.

How can we reproduce the bug? (as minimally and precisely as possible) Reproducing is likely going to be tough. It happens for me with a deployment that has many dependencies, each with lots of different K8 resources.

Local Environment: MacOS Monterey

• DevSpace Version: 6.0.0-beta.3
• Operating System: mac Kubernetes Cluster:
• Cloud Provider: aws
• Kubernetes Version: 1.22

/kind bug

[FabianKramm]

@tobalsgithub thanks for creating this issue! The kubectl error should be visible above the error message. We will add the stderr output of kubectl in the error message itself for the next version. Before we do any changes it would be interesting for me to know what exactly the error was, could you provide the full output after Applying manifests with kubectl...?

[tobalsgithub]

@FabianKramm good call, there was so much logging I didn't see the actual error up above. In this case, it just looks like a connection error. I'll see if this happens again, and if the error is any different.

fqs-enqueue deploy:fqs-enqueue W0628 13:56:41.419974   12195 transport.go:288] Unable to cancel request for *exec.roundTripper
fqs-enqueue deploy:fqs-enqueue Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
fqs-enqueue create_deployments: error deploying fqs-enqueue: error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
fqs-enqueue Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/configmap.yaml`
fqs-monitor create_deployments: error deploying fqs-monitor: error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
fqs-monitor Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/configmap.yaml`
cache-builder run_dependencies: context canceled
translation-sets run_dependencies: context canceled
custom-traffic-processor deploy:custom-traffic-processor 
custom-traffic-processor deploy:custom-traffic-processor [Errno 32] Broken pipe
custom-traffic-processor deploy:custom-traffic-processor Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
custom-traffic-processor deploy:custom-traffic-processor BrokenPipeError: [Errno 32] Broken pipe
external-auth-credential-service deploy:external-auth-credential-service 
external-auth-credential-service deploy:external-auth-credential-service [Errno 32] Broken pipe
external-auth-credential-service deploy:external-auth-credential-service Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
external-auth-credential-service deploy:external-auth-credential-service BrokenPipeError: [Errno 32] Broken pipe
custom-traffic-processor create_deployments: error deploying custom-traffic-processor: error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
custom-traffic-processor Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/migrate.yaml`
external-auth-credential-service create_deployments: error deploying external-auth-credential-service: error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
external-auth-credential-service Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/manifest.yaml`
fatal exit status 1

[FabianKramm]

@tobalsgithub in the newest beta version we have improved the error message to include the actual stderr, so this should be easier to understand now.

[tobalsgithub]

Thanks @FabianKramm . I'm on 6.0.0-beta.4 right now, which I believe is the latest published beta. Just had it happen again. It really does just look like some sort of connection error.

translation-sets deploy:translation-sets configmap/translation-sets-configmap unchanged
fqs-enqueue deploy:fqs-enqueue Unable to connect to the server: dial tcp: lookup B4C5219B8DF1181A9111AA5B4F7DFD46.gr7.us-east-1.eks.amazonaws.com on [fddd:dddd::]:53: read udp [fddd:dddd:1000:0:9247:5f86:98de:c7f2]:56803->[fddd:dddd::]:53: i/o timeout
fqs-enqueue create_deployments: error deploying fqs-enqueue: Unable to connect to the server: dial tcp: lookup B4C5219B8DF1181A9111AA5B4F7DFD46.gr7.us-east-1.eks.amazonaws.com on [fddd:dddd::]:53: read udp [fddd:dddd:1000:0:9247:5f86:98de:c7f2]:56803->[fddd:dddd::]:53: i/o timeout
fqs-enqueue  error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
fqs-enqueue Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/configmap.yaml`
translation-sets create_deployments: error deploying translation-sets:  error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
translation-sets Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/manifest.yaml`
external-auth-credential-service deploy:external-auth-credential-service 
external-auth-credential-service deploy:external-auth-credential-service [Errno 32] Broken pipe
external-auth-credential-service deploy:external-auth-credential-service Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
external-auth-credential-service deploy:external-auth-credential-service BrokenPipeError: [Errno 32] Broken pipe
external-auth-credential-service create_deployments: error deploying external-auth-credential-service: 
external-auth-credential-service [Errno 32] Broken pipe
external-auth-credential-service Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
external-auth-credential-service BrokenPipeError: [Errno 32] Broken pipe
external-auth-credential-service  error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
external-auth-credential-service Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/migrate.yaml`
cache-builder rl-redis-distcache01 deploy:redis 
cache-builder rl-redis-distcache01 deploy:redis [Errno 32] Broken pipe
cache-builder rl-redis-distcache01 deploy:redis Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
cache-builder rl-redis-distcache01 deploy:redis BrokenPipeError: [Errno 32] Broken pipe
cache-builder rl-redis-distcache01 create_deployments: error deploying redis: 
cache-builder rl-redis-distcache01 [Errno 32] Broken pipe
cache-builder rl-redis-distcache01 Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
cache-builder rl-redis-distcache01 BrokenPipeError: [Errno 32] Broken pipe
cache-builder rl-redis-distcache01  error executing 'kubectl --context arn:aws:eks:us-east-1:<redacted>:cluster/redoxlocal-eks-cluster apply --force -f -': 
cache-builder rl-redis-distcache01 Please make sure the command `kubectl apply` does work locally with manifest `./redoxlocal/manifest.yaml`
cache-builder run_dependencies: context canceled
fatal exit status 1