kubectl cp to work on stopped/completed pods

[xellsys]

Status Quo: The current implementation of kubectl cp requires the container to be running by encapsulating the exec command using the tar binary.

Requirement: Allow copying files from (possibly also to) a stopped container.

Additional info: This relates to #58512. It would also more closely align to the docker cp functionality. No need for the tar binary in the container anymore.

---

Background: My current use case is running helm test with a more sophisticated test set (0 and 1 results do not suffice for analysis) and having a simple way of persisting test results at the end (Jenkins or w/e). I know there are other solutions but I want to keep my test pod simple (not actively pushing test results to some endpoint or keeping it alive) and would like to avoid extensive configuration of a persistent storage.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[tsuna]

Would be great to have this capability.

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[tsuna]

/remove-lifecycle rotten

[seans3]

/kind feature /sig cli /area kubectl /priority P3

[tcoopman]

Ran into the same issue, but we didn't have to transfer a lot of data. The easiest hack was to tar all the data and base64 encode is and log it to stdout. Then you can get the data out because you can still get the logs for a stopped container.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[tsuna]

/remove-lifecycle stale

[zedtux]

I'm also looking to this feature !

[rafaelaazevedo]

this would be great for people who run tests on a kubernetes pod and need to get the reports after

[wjcheng1]

yes would love to have this ability!

[yue9944882]

ref https://github.com/kubernetes/kubernetes/issues/58512

[stmaute]

I also would like to have this feature. Are there any plans to implement this???

[cliffburdick]

This would be really useful as large files base64 encoded is not manageable.

[seans3]

I'm not currently sure how feasible this is, but there seems to be a lot of interest--raising the priority.

/priority P2

[FilipB]

Not sure if it's applicable here, but I was advised to use nice workaround for OpenShift 4.1. container_ID is needed. Then you can run on your node where the container was running: sed -n '/path/ s/merged/diff\//p' /var/lib/containers/storage/overlay-containers/$container_ID/userdata/config.json which will give you location of directory which contains added files to your container e.g.: /var/lib/containers/storage/overlay/406944605cabd99dc6e886ab9c3998b89fb25866e135c0906f573da47ffe23e1/diff/

That worked for me to get created files from exited container.

[rocketraman]

Another use case, slightly different than obtaining test results mentioned a couple of times above: containers that dump core or heap will obviously terminate after doing so. At this point, doing kubectl cp on a stopped container is necessary to obtain the core/heap. One example is using the -XX:+HeapDumpOnOutOfMemoryError JVM flag.

[jglick]

A related use case: copying files to or from the persistent volume associated with a scaled-down stateful set, without having to scale it up or attach the volume to a temporary no-op pod.

[samsieber]

So I too am working with k8s, triggering jobs from jenkins and wanting files back. The method I've settled on is a sidecar that opens a fifo called export-files-fifo (via the mkfifo command) at the beginning and in the preStop livecycle hook tars and pipes all the files I'm interested over the fifo. And the jenkins job triggering the job immediately run something along the lines of kubectl exec my-job-pod -- sh 'cat | export-files-fifo' > exported_files.tar

Edit: I had prototyped this with normal pods but it doesn't work well with jobs without a lot of work-arounds because jobs don't do side-car containers well. My work around is to put the real work as an init container and then to have the "side-car" be the normal container that just send the data to the named pipe.

[atheiman]

@samsieber could you provide a simple yaml example of your idea?

I have my own workaround I just wrote for this problem, but its not great: https://gist.github.com/atheiman/a327a6cb5027376192ea396e14830157. It runs the job in an initContainer then starts up an alpine container which runs for a minute and gives the orchestrator time to save the job files with kubectl cp.

[samsieber]

@atheiman Here it is: https://gist.github.com/samsieber/349fde899d508b4e6be119e762fb600c - the job stays active until the './grab-files.sh' is called, at which point the job shuts down automatically.

[j-mew-s]

I'm debugging an error in file modification carried out by a Dockerfile. It is modifying a config file, and the error prevents the container from starting (CrashLoopBackOff). I cannot inspect it purely by checking out the image and running with a different command since it uses environment variables set at docker run time.

This feature would be helpful for me to quickly understand the state of the config file and identify the error. Instead, I'm going to log the file's contents temporarily in the docker file, then revert once I understand the issue (which will involve 2 build/test cycles).

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[MartinX3]

/remove-lifecycle stale

[bolt-juri-gavshin]

Hello! Is there any ETA on fixing this? It is really inconvenient to have so complex workarounds just to get files from "Completed" pods of a job. Have to create pods directly with init containers and do some magic just do such a simple operation... Thanks a lot in advance!

[brianpursley]

Just to understand the issue a little better (correct me if I'm wrong)...

You have something like this...

kubectl run job1 --image=busybox --restart=Never -- /bin/sh -c "echo test > /output.txt"

And you want to be able to do this after the job has completed...

kubectl cp job1:/output.txt output.txt

But it fails with this message:

error: cannot exec into a container in a completed pod; current phase is Succeeded

The reason you don't want to define a volume is that you just want to get the result of your job this one time, it is not something that is needed long-term, is that correct?

[bolt-juri-gavshin]

@brianpursley , you are 100% correct. At the moment what I do is create a pod with tail -f /dev/null as command and then use kubectl exec + kubectl cp + kubectl delete. One of use cases is execution of tests in Jenkins, after execution I need to retrieve junit.xml file. I have ok-ish workaround now (with just 1 additional step), but being able to copy files from Completed pod would require 1 step less...

[seans3]

/remove-priority P3

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[zedtux]

/remove-lifecycle stale

[gionapaolini]

@brianpursley , you are 100% correct. At the moment what I do is create a pod with tail -f /dev/null as command and then use kubectl exec + kubectl cp + kubectl delete. One of use cases is execution of tests in Jenkins, after execution I need to retrieve junit.xml file. I have ok-ish workaround now (with just 1 additional step), but being able to copy files from Completed pod would require 1 step less...

Can you elaborate a bit more on your workaround? That's how I understood:

1. Build an image for your test project and use tail -f /dev/null as entrypoint
2. Run a pod with that image
3. Execute a shell into it and run your tests
4. Copy the reports
5. Delete the pod

Have I understood correctly?

[lindhe]

I would be interested in trying to resolve this issue. But I realize that this is an intricate problem which may require some expertise from more experienced Kubernetes developers. Are there any suggestions on reasonable designs to approach this, or do we know of something that makes this problem intrinsically hard to solve?

[AshwinNS]

@brianpursley , you are 100% correct. At the moment what I do is create a pod with tail -f /dev/null as command and then use kubectl exec + kubectl cp + kubectl delete. One of use cases is execution of tests in Jenkins, after execution I need to retrieve junit.xml file. I have ok-ish workaround now (with just 1 additional step), but being able to copy files from Completed pod would require 1 step less...

Can you elaborate a bit more on your workaround? That's how I understood:

1. Build an image for your test project and use tail -f /dev/null as entrypoint
2. Run a pod with that image
3. Execute a shell into it and run your tests
4. Copy the reports
5. Delete the pod

Have I understood correctly?

sounds good, you can even override the entrypoint to sleep the container for certain amount of time, that would be more elegant.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[lindhe]

/remove-lifecycle stale

[cassador]

Any update or workaround on this? I saw the mentioned workaround but i don't think they could be used in our configuraiton. We are running it on EKS cluster with SSH disabled so i can't use that one with where the pods data are being stored by default on runtime,

[SRodi]

/remove-lifecycle stale

[pisto]

Please also make this work on pods in CrashLoopBackoff state. It's painful to debug these containers when you suspect the filesystem in the pod is incorrect, but there's no way to cp files, or even just list the container filesystem when a container is not running, and workarounds may be difficutl to apply.

[rptaylor]

I agree this would be useful. People need a way to extract files produced in pods after they finish.

My use case is a job that requires a lot of resources and runs for a couple days. I don't want to keep it running longer than necessary with e.g. 'sleep' because that would waste resources, but I also really don't want to lose the output if I am too busy to check on it and retrieve the output until several days afterwards. The output is important but reasonably small, only ~< 1 MB of text files. I could write the output to a persistent volume but that only helps slightly because it is not very conveniently accessible - I would have to start up another pod just for data extraction, to mount the volume and then 'kubectl cp'.

kubectl cp from a finished pod would be one way, but what if you could do something like cp mydata /dev/kubebucket as the last step of execution in your pod before it finishes, where /dev/kubebucket is a device exposed by kubelet to your pod that you can write to, for reasonably small and short-term storage of pod outputs, which is retained for a period of time after pod completion. Then suppose you could kubectl mount <pod or bucket identifier> <optional -c container> mountdir to get a read-only FUSE mount of whatever you put in the "bucket" , so you can get your files out and then fusermount -u mountdir (or kubectl unmount mountdir). Perhaps you could also kubectl get bucket <bucketID> , kubectl delete bucket <bucketID> etc. There could be configurable max size limits and automatic expiration time periods to make sure the buckets don't accumulate too much over time.

Just an idea, no idea how feasible to implement it would be but it could be an alternative way to extract outputs from completed containers by providing some short term storage facility that is outside of the pods.

[riesbl01]

Also agree it would be useful. The use case for why my team is wanting to implement this is that they have QA people who need to run test suites against the services in the cluster. To do that they deploy a kubernetes job in a pipeline, then call kubectl wait to wait until the pod is complete meaning the test suite has finished, and then we want to get the document generated by the test suite off the pod and uploaded for review. That isnt possible because the wait command makes the pipeline wait until the job is finished, and once the job is finished you cant kubectl cp files anymore.

[nagakonduri]

Looking forward for this feature, please share if this is in consideration.

[rafaelaazevedo]

I have been using this workaround, not ideal, but it is working.

How to get your test reports from a pod – Kubernetes

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[SRodi]

/remove-lifecycle stale

[eoinmcafee00]

/remove-lifecycle stale

[tclarke]

I'd like to see this too. My use case is for post mortem debugging of production pods. (so an extrypoint of "sleep 10000" isn't an option)

[Warxcell]

+1. Useful to run unit tests in pod and get coverage reports, which are generated after tests finishes.

[graydon]

This (and in general "access to the pod filesystem using an API that is not just executing-tar-on-stdin") would be very desirable to me. I am sort of astonished that this is not part of the standard k8s API. Files in the filesystem -- not just open filehandles -- are actually important parts of the lifecycle and system interface of most programs.

[brianpursley]

I still think using a persistent volume is the best way to preserve and access the output produced by a completed pod

But I know there are a lot of people who end up here, wanting a way to run some task to completion and then use kubectl cp to access the output that was produced inside one of the container's filesystem.

For anyone who doesn't want to create a persistent volume, for whatever reason, here is an idea I came up with that might work for you.

It makes use of:

• A worker container that is actually an init container. This is what runs automated tests or whatever other task you want to run in the pod.
• A keepalive container that does nothing other than keep the pod alive so you can copy the output
• A results ephemeral volume (emptyDir) to share the output of the worker with the keepalive container

The pod will not become "ready" until the init container completes its work. Because of this, you can use kubectl wait to wait for the worker to finish and the pod to become ready.

Once the pod is ready, the keepalive container keeps the pod alive, so you have an opportunity (60 seconds in my example) to use kubectl cp to get the files stored in the results volume.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  volumes:
  - name: results
    emptyDir: {}
  initContainers:
  - image: busybox
    name: worker
    command: ["/bin/sh", "-c", "echo 'hello from the worker' > /results/output.txt;"]
    volumeMounts:
    - name: results
      mountPath: /results
  containers:
  - image: busybox
    name: keepalive
    command: ["/bin/sh", "-c", "sleep 60"]
    volumeMounts:
    - name: results
      mountPath: /results
EOF

kubectl wait --timeout=1h --for=condition=Ready=true pod/test
kubectl cp test:/results/output.txt /tmp/output.txt
kubectl delete pod test
cat /tmp/output.txt

[Warxcell]

I still think using a persistent volume is the best way to preserve and access the output produced by a completed pod

But I know there are a lot of people who end up here, wanting a way to run some task to completion and then use kubectl cp to access the output that was produced inside one of the container's filesystem.

For anyone who doesn't want to create a persistent volume, for whatever reason, here is an idea I came up with that might work for you.

It makes use of:

• A worker container that is actually an init container. This is what runs automated tests or whatever other task you want to run in the pod.
• A keepalive container that does nothing other than keep the pod alive so you can copy the output
• A results ephemeral volume (emptyDir) to share the output of the worker with the keepalive container

The pod will not become "ready" until the init container completes its work. Because of this, you can use kubectl wait to wait for the worker to finish and the pod to become ready.

Once the pod is ready, the keepalive container keeps the pod alive, so you have an opportunity (60 seconds in my example) to use kubectl cp to get the files stored in the results volume.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  volumes:
  - name: results
    emptyDir: {}
  initContainers:
  - image: busybox
    name: worker
    command: ["/bin/sh", "-c", "echo 'hello from the worker' > /results/output.txt;"]
    volumeMounts:
    - name: results
      mountPath: /results
  containers:
  - image: busybox
    name: keepalive
    command: ["/bin/sh", "-c", "sleep 60"]
    volumeMounts:
    - name: results
      mountPath: /results
EOF

kubectl wait --timeout=1h --for=condition=Ready=true pod/test
kubectl cp test:/results/output.txt /tmp/output.txt
kubectl delete pod test
cat /tmp/output.txt

The reason not to use PV is the added complexity. For my particular case I don't need these files persistant, so it adds additiona l complexity of managing these PVs and cleanup. Problem with this approach is that kubectl wait --timeout=1h --for=condition=Ready=true pod/test doesn't return exit code of init container.