Empty ConfigMap

[Xyrodileas]

Great tool, looking forward to use it !

Description

The following command hang and then cancel port forward: kubectl tap on -p 80 myservice --port-forward

Result :

Establishing port-forward tunnels to Service...
Waiting for Pod containers to become ready..........................................
Pod not running after 90 seconds. Cancelling port-forward, tap still active.

When checking the pod for my service :

kubectl get pods
....
mypod        0/2     ContainerCreating   0          12m
....
kubectl logs mypod
Error from server (BadRequest): a container name must be specified for pod mypod, choose one of: [mypod kubetap]
kubectl logs mypod kubetap
Error from server (BadRequest): container "kubetap" in pod "mypod" is waiting to start: ContainerCreating

Screenshots or other information

Go version: go version go1.13 linux/amd64 Kubernetes client version: 1.16.0 Kubernetes server version: 1.14.9-eks

[Eriner]

When you perform a kubectl describe of the Pod, what does it show? Its seems that it's stuck in a creating state.

[Eriner]

Establish a non-interactive tap so it doesn't automatically untap:

kubectl tap on -p80 myservice

[Xyrodileas]

Seems a configmap cannot be mounted :

kubectl describe pod mypod-6fdfb85784-2rtbs kubetap
[....]
Events:
  Type     Reason       Age                  From                                                      Message
  ----     ------       ----                 ----                                                      -------
  Warning  FailedMount  16m (x82 over 166m)  kubelet, ip-10-153-231-216.ca-central-1.compute.internal  MountVolume.SetUp failed for volume "kubetap-target-mypod" : configmap "kubetap-target-mypod" not found
  Warning  FailedMount  98s (x73 over 164m)  kubelet, ip-10-153-231-216.ca-central-1.compute.internal  Unable to mount volumes for pod "mypod-6fdfb85784-2rtbs_default(4dab51fa-9b66-11ea-afc6-02470cf3a3ac)": timeout expired waiting for volumes to attach or mount for pod "default"/"mypod-6fdfb85784-2rtbs". list of unmounted volumes=[kubetap-target-mypod]. list of unattached volumes=[kubetap-target-mypod mitmproxy-data]

[Eriner]

was the configmap created? Does your user have the ability to create configmaps?

[Xyrodileas]

Seems to be created, but empty :

kubectl describe configmap kubetap-target-mypod-service
Name:         kubetap-target-mypod-service
Namespace:    default
Labels:       <none>
Annotations:  kubetap.io/proxy-config: target-mypod-service

Data
====
Events:  <none>

The user does have the ability to deploy other ConfigMap.

[Eriner]

That seems very odd, because looking at the code an error should be raised (terminating the program) if the configmap fails to write. I'd say this was a fluke, but it probably isn't. I'm testing changes now which add some sanity checks to the process, and ensure that the data in the CM we create matches the data after creation.

Out of curiosity, can you replicate that behavior with a new tap (in a different namespace)?

[Xyrodileas]

I only have the default namespace in my setup, but after trying with another pod, same behavior sadly.

Let me know if you want me to test again once you added those changes ! :)

[Eriner]

Okay, I'll be pushing to a branch in a few minutes. I request that you rebuild on that branch with make and try again. I suspect I know what the result will be, but have to kick the tires to be sure. I still am unsure of the underlying problem.

[Eriner]

I changed my mind, pushed to master. Please pull and let me know if anything has changed :)

[Xyrodileas]

I re-installed with the one-liner as make was giving me an error. cd && GO111MODULE=on go get github.com/soluble-ai/kubetap/cmd/kubectl-tap@latest I hope that did trigger an update of kubectl on master ?

The test gave me the same behavior as previously, no extra output. The configmap is still empty.

[Eriner]

cd into the source directory in your ${GOPATH} and run make. If it gives you an error, we should fix that :)

[rajetta]

I am getting the same error - empty configmap. I installed by brew and kubectl tap version doesn't show the version but brew download is from https://codeload.github.com/soluble-ai/kubetap/tar.gz/v0.1.1.

[Eriner]

@rajetta Does the underlying Pod have access to ConfigMaps?