replicatedhq / troubleshoot

Preflight Checks and Support Bundles Framework for Kubernetes Applications
https://troubleshoot.sh
Apache License 2.0
543 stars 92 forks source link

Changing file from pod is not included successfully in the support bundle #1514

Closed mihai-waydev closed 2 weeks ago

mihai-waydev commented 5 months ago

Bug Description

When trying to get a file from a pod that is constantly changing, it fails randomly

Expected Behavior

The file should be in the support bundle with it contents at the moment it was retrieved.

Steps To Reproduce

Generate a support bundle that should copy a file from the pod that is constantly changing.

Additional Context

We can’t get that file and instead we get this one file called “laravel.log-errors.json” with the following contents:

{ “/var/www/html/storage/logs/laravel.log/error”: “failed to stream command output: command terminated with exit code 1”, “/var/www/html/storage/logs/laravel.log/stderr”: “tar: laravel.log: file changed as we read it\n” }

Spec:

apiVersion: troubleshoot.replicated.com/v1beta1 kind: Collector metadata: name: collector-sample spec: collectors:

Troubleshoot version: 0.83.0 OS: Ubuntu 20 (I think it happens on various other OS as well like RHEL 9.3)

xavpaice commented 2 weeks ago

My apologies it's taken a while to respond here.

Unfortunately with Linux in order to snapshot the file prior to copy, we would need to pause the process that is writing to it, or have the file located on a filesystem that supports snapshots. Pausing the process is disruptive to the running application and not desirable. The filesystem level snapshot could be achieved but it's so environment specific that I think we need to rely on a different collector for that, rather than modify the copy or logs collectors.