search

kubernetes / kubectl

Issue tracker and mirror of kubectl code
Apache License 2.0
2.85k stars 921 forks source link

debug: ability to replicate volume mounts #1071

Closed rcoup closed 8 months ago

rcoup commented 3 years ago

What would you like to be added:

When debugging by adding a container to a pod, having the ability to match volume mounts with the target container.

Specifically:

Why is this needed:

From a new debug container added to a running pod, I can't access data other containers can see on volume mounts. Obvious candidates: emptyDir, NFS mounts, configmap/secret mounts, EBS volumes, etc

Ideas:

  1. Replicate some/all volume mounts from one other container
  2. Expose all the volumes available to the pod so they can be mounted by name: --volume-mount=name:/path There are a lot of options exposed (subPaths, read-only, etc) though to try and cram into a CLI option.

IMO (1) is probably easiest and solves most use cases. Maybe something like:

eddiezane commented 3 years ago

/cc @verb

verb commented 3 years ago

This fits within the theme of providing a bit more configurability for kubectl debug. I'll add it to the list in kubernetes/enhancements#1441 to inform the design.

@rcoup for your use case does accessing the target container's root filesystem via /proc/$PID/root work? (assuming linux containers)

rcoup commented 3 years ago

for your use case does accessing the target container's root filesystem via /proc/$PID/root work?

I get ls: cannot access '/proc/7/root/': Permission denied, so I guess I need https://github.com/kubernetes/kubernetes/issues/97103 resolved first? (or add caps to / make my original pod privileged?)

verb commented 3 years ago

That's interesting. no special capabilities are required when shareProcessNamespace is true, but otherwise SYS_PTRACE is required. I didn't realize that.

Ok, so no, this trick doesn't work for your use case.

r-vanooyen commented 3 years ago

It would be nice, if this feature is usable without the --copy-to argument.

soltysh commented 3 years ago

/assign @verb /triage accepted

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/kubectl/issues/1071#issuecomment-1013731915): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
morremeyer commented 2 years ago

/reopen

This is still a viable feature.

k8s-ci-robot commented 2 years ago

@morremeyer: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes/kubectl/issues/1071#issuecomment-1046231738): >/reopen > >This is still a viable feature. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
morremeyer commented 2 years ago

Can a collaborator please reopen this? This is still a feature that would make sense to be implemented.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

verb commented 2 years ago

/remove-lifecycle stale

tonyaw commented 2 years ago

@verb, may I ask the schedule of this feature? I didn't find it in KEP(https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/277-ephemeral-containers)

Thanks in advance!

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

simcax commented 2 years ago

/remove-lifecycle stale

TheDukeDK commented 2 years ago

A good example of a use case for this is the dotnet monitor. It expects to communicate with the dotnet service over a socket mounted in as a volume. See: https://github.com/dotnet/dotnet-monitor/blob/main/documentation/kubernetes.md

Being able to use ephemeral containers to debug existing pods in a non intrusive way would be a great thing. ;-)

NissesSenap commented 1 year ago

Maybe is a separate issue but I would think it would be interesting to extend this functionality even earlier. Hopefully most of us don't allow writing to root disk inside our containers but all apps don't need /tmp disks or similar to run.

It would be really nice if a debug container also can attach ephemeral disk to a container or maybe even better any volume you want. This way you could save some data from your debugging as well, for example profiling.

jbg commented 1 year ago

For anyone who (like me) finds this issue while searching for a solution, I wanted to share that you can do this by making a direct patch to the ephemeralcontainers subresource, rather than using kubectl debug. For example, I needed to mount the same ca-certificates that were mounted in the target container, in the debug container:

$ kubectl proxy
$ curl http://localhost:8001/api/v1/namespaces/your-namespace/pods/your-pod-name/ephemeralcontainers \
  -X PATCH \
  -H 'Content-Type: application/strategic-merge-patch+json' \
  -d '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debugger",
                "command": ["sh"],
                "image": "your-debug-image",
                "targetContainerName": "your-target-container-name",
                "stdin": true,
                "tty": true,
                "volumeMounts": [{
                    "mountPath": "/etc/ca-certificates",
                    "name": "ca-certificates",
                    "readOnly": true
                }]
            }
        ]
    }
}'
$ kubectl -n your-namespace attach your-pod-name -c debugger -ti
nathanmcgarvey-modopayments commented 1 year ago

For anyone who (like me) finds this issue while searching for a solution, I wanted to share that you can do this by making a direct patch to the ephemeralcontainers subresource, rather than using kubectl debug. For example, I needed to mount the same ca-certificates that were mounted in the target container, in the debug container:

$ kubectl proxy
$ curl http://localhost:8001/api/v1/namespaces/your-namespace/pods/your-pod-name/ephemeralcontainers \
  -X PATCH \
  -H 'Content-Type: application/strategic-merge-patch+json' \
  -d '
{
    "spec":
    {
        "ephemeralContainers":
        [
            {
                "name": "debugger",
                "command": ["sh"],
                "image": "your-debug-image",
                "targetContainerName": "your-target-container-name",
                "stdin": true,
                "tty": true,
                "volumeMounts": [{
                    "mountPath": "/etc/ca-certificates",
                    "name": "ca-certificates",
                    "readOnly": true
                }]
            }
        ]
    }
}'
$ kubectl -n your-namespace attach your-pod-name -c debugger -ti

FWIW, I made a shell (zsh, at least) function that copies environment variables, envFrom configuration, and volume mounts and does this manually via a kubectl, jq, and curl commands. Note that the image is hardcoded to alpine:latest, but that's easily changed:

https://gist.github.com/nathanmcgarvey-modopayments/358a84297086de3975f54895a9e7123d

Edit: Note that I only tested this on AWS EKS 1.27. YMMV.

bh-tt commented 9 months ago

Can I request that ephemeralcontainers be added to the allowed subresources of Pods for kubectl patch? This makes the above 3 steps a bit easier as one can simply run kubectl patch --type=merge --patch-file=<file> and then attach, instead of needing to run kubectl proxy as well (which creates a local port, anyone on the same server can send API requests as the person running kubectl proxy).

ardaguclu commented 8 months ago

This will be handled with https://github.com/kubernetes/kubernetes/pull/120346 and I'm closing because we'll track this feature from the KEP; /close

k8s-ci-robot commented 8 months ago

@ardaguclu: Closing this issue.

In response to [this](https://github.com/kubernetes/kubectl/issues/1071#issuecomment-1948003055): >This will be handled with https://github.com/kubernetes/kubernetes/pull/120346 and I'm closing because we'll track this feature from the KEP; >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.