Add count-based checkpoint retention policies

[Parthiba-Hazra]

Introduced global, container, pod, and namespace-level policies for checkpoint retention, all based on count limits. Introduced a flag applyPoliciesImmediately to enable immediate application of policies

The resource specific policies like container, pod and namespace policy will override the global policy.

[rst0git]

@Parthiba-Hazra To summarize the comments from the meeting earlier today:

• It would be good to add a test for CI to validate the functionality in this pull request.
• The individual container count limit takes precedence over Pod, namespace, and global policy.
• It might be useful to add a markdown file (e.g., in a docs/ folder) that describes how the checkpoint retention policy works and how it can be configured.

[adrianreber]

Nice to see CI complaining about the shell script formatting. So CI works :+1: Good

[Parthiba-Hazra]

@rst0git I think now it's working properly :)

but in local test this was failing -

Show log

``` ➜ checkpoint-restore-operator git:(ci-test) ✗ act -W .github/workflows/tests.yaml push INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock' [tests.yaml/build-and-deploy] 🚀 Start image=catthehacker/ubuntu:act-latest [tests.yaml/build-and-deploy] 🐳 docker pull image=catthehacker/ubuntu:act-latest platform= username= forcePull=true [tests.yaml/build-and-deploy] 🐳 docker create image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [tests.yaml/build-and-deploy] 🐳 docker run image=catthehacker/ubuntu:act-latest platform= entrypoint=["tail" "-f" "/dev/null"] cmd=[] network="host" [tests.yaml/build-and-deploy] ☁ git clone 'https://github.com/actions/setup-go' # ref=v5 [tests.yaml/build-and-deploy] ⭐ Run Main Checkout code [tests.yaml/build-and-deploy] 🐳 docker cp src=/home/ubuntu/os/checkpoint-restore-operator/. dst=/home/ubuntu/os/checkpoint-restore-operator [tests.yaml/build-and-deploy] ✅ Success - Main Checkout code [tests.yaml/build-and-deploy] ⭐ Run Main Set up Go [tests.yaml/build-and-deploy] 🐳 docker cp src=/home/ubuntu/.cache/act/actions-setup-go@v5/ dst=/var/run/act/actions/actions-setup-go@v5/ [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[node /var/run/act/actions/actions-setup-go@v5/dist/setup/index.js] user= workdir= | Setup go version spec 1.21 [tests.yaml/build-and-deploy] 💬 ::debug::isExplicit: [tests.yaml/build-and-deploy] 💬 ::debug::explicit? false [tests.yaml/build-and-deploy] 💬 ::debug::isExplicit: 1.21.12 [tests.yaml/build-and-deploy] 💬 ::debug::explicit? true [tests.yaml/build-and-deploy] 💬 ::debug::evaluating 1 versions [tests.yaml/build-and-deploy] 💬 ::debug::matched: 1.21.12 [tests.yaml/build-and-deploy] 💬 ::debug::checking cache: /opt/hostedtoolcache/go/1.21.12/arm64 [tests.yaml/build-and-deploy] 💬 ::debug::Found tool in cache go 1.21.12 arm64 | Found in cache @ /opt/hostedtoolcache/go/1.21.12/arm64 | Added go to the path | Successfully set up Go version 1.21 [tests.yaml/build-and-deploy] 💬 ::debug::which go :/opt/hostedtoolcache/go/1.21.12/arm64/bin/go: [tests.yaml/build-and-deploy] 💬 ::debug::go env GOPATH :/root/go: [tests.yaml/build-and-deploy] 💬 ::debug::creating /root/go [tests.yaml/build-and-deploy] 💬 ::debug::creating /root/go/bin [tests.yaml/build-and-deploy] 💬 ::debug::add bin true | [command]/opt/hostedtoolcache/go/1.21.12/arm64/bin/go env GOMODCACHE | [command]/opt/hostedtoolcache/go/1.21.12/arm64/bin/go env GOCACHE | /root/go/pkg/mod | /root/.cache/go-build [tests.yaml/build-and-deploy] 💬 ::debug::followSymbolicLinks 'true' [tests.yaml/build-and-deploy] 💬 ::debug::followSymbolicLinks 'true' [tests.yaml/build-and-deploy] 💬 ::debug::implicitDescendants 'true' [tests.yaml/build-and-deploy] 💬 ::debug::matchDirectories 'true' [tests.yaml/build-and-deploy] 💬 ::debug::omitBrokenSymbolicLinks 'true' [tests.yaml/build-and-deploy] 💬 ::debug::Search path '/home/ubuntu/os/checkpoint-restore-operator/go.sum' [tests.yaml/build-and-deploy] 💬 ::debug::/home/ubuntu/os/checkpoint-restore-operator/go.sum [tests.yaml/build-and-deploy] 💬 ::debug::Found 1 files to hash. [tests.yaml/build-and-deploy] 💬 ::debug::primary key is setup-go-Linux-ubuntu20-go-1.21.12-84b075fa9a22072cff932a3c7017f7f85dba0189d1ed11f8bf84aeec1d606a97 [tests.yaml/build-and-deploy] 💬 ::debug::Resolved Keys: [tests.yaml/build-and-deploy] 💬 ::debug::["setup-go-Linux-ubuntu20-go-1.21.12-84b075fa9a22072cff932a3c7017f7f85dba0189d1ed11f8bf84aeec1d606a97"] [tests.yaml/build-and-deploy] 💬 ::debug::Checking zstd --quiet --version [tests.yaml/build-and-deploy] 💬 ::debug::1.4.8 [tests.yaml/build-and-deploy] 💬 ::debug::zstd version: 1.4.8 [tests.yaml/build-and-deploy] 💬 ::debug::Resource Url: http://192.168.64.5:41577/_apis/artifactcache/cache?keys=setup-go-Linux-ubuntu20-go-1.21.12-84b075fa9a22072cff932a3c7017f7f85dba0189d1ed11f8bf84aeec1d606a97&version=80c03ad9bc0f7552c128ae36020fe5294ef09c552e168fbe45ea237491c60cae [tests.yaml/build-and-deploy] ⚙ *** [tests.yaml/build-and-deploy] 💬 ::debug::Cache Result: [tests.yaml/build-and-deploy] 💬 ::debug::{"archiveLocation":"***","cacheKey":"setup-go-linux-ubuntu20-go-1.21.12-84b075fa9a22072cff932a3c7017f7f85dba0189d1ed11f8bf84aeec1d606a97","result":"hit"} [tests.yaml/build-and-deploy] 💬 ::debug::Archive Path: /tmp/ceb5ea98-917d-41b8-8a72-44ffe796d9f3/cache.tzst [tests.yaml/build-and-deploy] 💬 ::debug::Use Azure SDK: false [tests.yaml/build-and-deploy] 💬 ::debug::Download concurrency: 8 [tests.yaml/build-and-deploy] 💬 ::debug::Request timeout (ms): 30000 [tests.yaml/build-and-deploy] 💬 ::debug::Cache segment download timeout mins env var: undefined [tests.yaml/build-and-deploy] 💬 ::debug::Segment download timeout (ms): 600000 [tests.yaml/build-and-deploy] 💬 ::debug::Lookup only: false | Cache Size: ~142 MB (148825117 B) | [command]/usr/bin/tar -xf /tmp/ceb5ea98-917d-41b8-8a72-44ffe796d9f3/cache.tzst -P -C /home/ubuntu/os/checkpoint-restore-operator --use-compress-program unzstd | Cache restored successfully | Cache restored from key: setup-go-linux-ubuntu20-go-1.21.12-84b075fa9a22072cff932a3c7017f7f85dba0189d1ed11f8bf84aeec1d606a97 [tests.yaml/build-and-deploy] ❓ add-matcher /run/act/actions/actions-setup-go@v5/matchers.json | go version go1.21.12 linux/arm64 | [tests.yaml/build-and-deploy] ❓ ::group::go env | GO111MODULE='' | GOARCH='arm64' | GOBIN='' | GOCACHE='/root/.cache/go-build' | GOENV='/root/.config/go/env' | GOEXE='' | GOEXPERIMENT='' | GOFLAGS='' | GOHOSTARCH='arm64' | GOHOSTOS='linux' | GOINSECURE='' | GOMODCACHE='/root/go/pkg/mod' | GONOPROXY='' | GONOSUMDB='' | GOOS='linux' | GOPATH='/root/go' | GOPRIVATE='' | GOPROXY='https://proxy.golang.org,direct' | GOROOT='/opt/hostedtoolcache/go/1.21.12/arm64' | GOSUMDB='sum.golang.org' | GOTMPDIR='' | GOTOOLCHAIN='auto' | GOTOOLDIR='/opt/hostedtoolcache/go/1.21.12/arm64/pkg/tool/linux_arm64' | GOVCS='' | GOVERSION='go1.21.12' | GCCGO='gccgo' | AR='ar' | CC='gcc' | CXX='g++' | CGO_ENABLED='1' | GOMOD='/home/ubuntu/os/checkpoint-restore-operator/go.mod' | GOWORK='' | CGO_CFLAGS='-O2 -g' | CGO_CPPFLAGS='' | CGO_CXXFLAGS='-O2 -g' | CGO_FFLAGS='-O2 -g' | CGO_LDFLAGS='-O2 -g' | PKG_CONFIG='pkg-config' | GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2246315361=/tmp/go-build -gno-record-gcc-switches' | [tests.yaml/build-and-deploy] ❓ ::endgroup:: [tests.yaml/build-and-deploy] ✅ Success - Main Set up Go [tests.yaml/build-and-deploy] ⚙ ::set-output:: cache-hit=true [tests.yaml/build-and-deploy] ⚙ ::set-output:: go-version=1.21.12 [tests.yaml/build-and-deploy] ⚙ ::add-path:: /opt/hostedtoolcache/go/1.21.12/arm64/bin [tests.yaml/build-and-deploy] ⚙ ::add-path:: /root/go/bin [tests.yaml/build-and-deploy] ⭐ Run Main Install dependencies [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/2] user= workdir= | % Total % Received % Xferd Average Speed Time Time Time Current | Dload Upload Total Spent Left Speed 100 47.6M 100 47.6M 0 0 5364k 0 0:00:09 0:00:09 --:--:-- 6887k [tests.yaml/build-and-deploy] ✅ Success - Main Install dependencies [tests.yaml/build-and-deploy] ⭐ Run Main Prepare Checkpoints Directory in Host [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/3] user= workdir= | /var/lib/kubelet/checkpoints [tests.yaml/build-and-deploy] ✅ Success - Main Prepare Checkpoints Directory in Host [tests.yaml/build-and-deploy] ⭐ Run Main Generate Checkpoint Tar Files [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/4] user= workdir= | Checkpoint tar file created at /var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-2024-07-24T18:09:34.tar | Checkpoint tar file created at /var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-2024-07-24T18:09:35.tar | Checkpoint tar file created at /var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-2024-07-24T18:09:36.tar | Checkpoint tar file created at /var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-2024-07-24T18:09:37.tar | Checkpoint tar file created at /var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-2024-07-24T18:09:38.tar [tests.yaml/build-and-deploy] ✅ Success - Main Generate Checkpoint Tar Files [tests.yaml/build-and-deploy] ⭐ Run Main Generate Kind Config [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/5] user= workdir= [tests.yaml/build-and-deploy] ✅ Success - Main Generate Kind Config [tests.yaml/build-and-deploy] ⭐ Run Main Delete Existing Kind Cluster [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/6] user= workdir= | Deleting cluster "kind" ... | Deleted nodes: ["kind-control-plane"] [tests.yaml/build-and-deploy] ✅ Success - Main Delete Existing Kind Cluster [tests.yaml/build-and-deploy] ⭐ Run Main Create Kubernetes cluster [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/7] user= workdir= | Creating cluster "kind" ... ✓ Ensuring node image (kindest/node:v1.30.0) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 | Set kubectl context to "kind-kind" | You can now use your cluster with: | | kubectl cluster-info --context kind-kind | | Thanks for using kind! 😊 [tests.yaml/build-and-deploy] ✅ Success - Main Create Kubernetes cluster [tests.yaml/build-and-deploy] ⭐ Run Main Load Docker image into kind cluster [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/8] user= workdir= | Untagged: checkpoint-restore-operator:ci | Deleted: sha256:01baf479c6c940deca0fd057b8669e80d0376d5af8b53eac2cf3988a4961a48e | mkdir -p /home/ubuntu/os/checkpoint-restore-operator/bin | test -s /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen && /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen --version | grep -q v0.15.0 || \ | GOBIN=/home/ubuntu/os/checkpoint-restore-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.15.0 | go: downloading sigs.k8s.io/controller-tools v0.15.0 | go: sigs.k8s.io/controller-tools@v0.15.0 requires go >= 1.22.0; switching to go1.22.5 | go: downloading go1.22.5 (linux/arm64) | go: downloading github.com/spf13/cobra v1.8.0 | go: downloading github.com/gobuffalo/flect v1.0.2 | go: downloading k8s.io/apiextensions-apiserver v0.30.0 | go: downloading k8s.io/apimachinery v0.30.0 | go: downloading golang.org/x/tools v0.20.0 | go: downloading github.com/fatih/color v1.16.0 | go: downloading k8s.io/api v0.30.0 | go: downloading sigs.k8s.io/yaml v1.4.0 | go: downloading k8s.io/klog/v2 v2.120.1 | go: downloading github.com/mattn/go-colorable v0.1.13 | go: downloading github.com/mattn/go-isatty v0.0.20 | go: downloading golang.org/x/sync v0.7.0 | go: downloading github.com/go-logr/logr v1.4.1 | go: downloading golang.org/x/sys v0.19.0 | go: downloading golang.org/x/mod v0.17.0 | go: downloading golang.org/x/net v0.24.0 | /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases | /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..." | go fmt ./... | go vet ./... | test -s /home/ubuntu/os/checkpoint-restore-operator/bin/setup-envtest || GOBIN=/home/ubuntu/os/checkpoint-restore-operator/bin go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest | go: downloading sigs.k8s.io/controller-runtime v0.18.4 | go: downloading sigs.k8s.io/controller-runtime/tools/setup-envtest v0.0.0-20240717190548-1ed345090869 | go: sigs.k8s.io/controller-runtime/tools/setup-envtest@v0.0.0-20240717190548-1ed345090869 requires go >= 1.22.0; switching to go1.22.5 | go: downloading github.com/go-logr/logr v1.4.2 | go: downloading github.com/go-logr/zapr v1.3.0 | go: downloading github.com/spf13/afero v1.6.0 | go: downloading go.uber.org/zap v1.26.0 | go: downloading go.uber.org/multierr v1.10.0 | go: downloading golang.org/x/text v0.16.0 | KUBEBUILDER_ASSETS="/home/ubuntu/os/checkpoint-restore-operator/bin/k8s/1.27.1-linux-arm64" go test ./... -coverprofile cover.out | ? github.com/checkpoint-restore/checkpoint-restore-operator/api/v1 [no test files] | ? github.com/checkpoint-restore/checkpoint-restore-operator/cmd [no test files] | ok github.com/checkpoint-restore/checkpoint-restore-operator/internal/controller 0.016s coverage: 0.0% of statements | docker build -t checkpoint-restore-operator:ci . [+] Building 0.2s (17/17) FINISHED docker:default | => [internal] load .dockerignore 0.1s | => => transferring context: 160B 0.0s | => [internal] load build definition from Dockerfile 0.1s | => => transferring dockerfile: 1.32kB 0.0s | => [internal] load metadata for gcr.io/distroless/static:nonroot 0.0s | => [internal] load metadata for docker.io/library/golang:1.21 0.0s | => [builder 1/9] FROM docker.io/library/golang:1.21 0.0s | => [stage-1 1/3] FROM gcr.io/distroless/static:nonroot 0.0s | => [internal] load build context 0.0s | => => transferring context: 63.13kB 0.0s | => CACHED [builder 2/9] WORKDIR /workspace 0.0s | => CACHED [builder 3/9] COPY go.mod go.mod 0.0s | => CACHED [builder 4/9] COPY go.sum go.sum 0.0s | => CACHED [builder 5/9] RUN go mod download 0.0s | => CACHED [builder 6/9] COPY cmd/main.go cmd/main.go 0.0s | => CACHED [builder 7/9] COPY api/ api/ 0.0s | => CACHED [builder 8/9] COPY internal/controller/ internal/controller/ 0.0s | => CACHED [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go bui 0.0s | => CACHED [stage-1 2/3] COPY --from=builder /workspace/manager . 0.0s | => exporting to image 0.0s | => => exporting layers 0.0s | => => writing image sha256:9543ec8a74c4e3fe38c64738eab5e633141e0921e1082 0.0s | => => naming to docker.io/library/checkpoint-restore-operator:ci 0.0s | checkpoint-restore-operator ci 9543ec8a74c4 2 days ago 53.4MB | remontada007/checkpoint-restore-operator t10 981b592e461f 2 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t9 da82b2344091 2 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t8 fb6dcef76f76 2 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t7 1f44068f3f6a 2 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t6 5d8f7941dfbb 4 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t5 56844c050f15 4 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t4 16dd8fbdbd7c 4 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t3 927736bc2bc4 4 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t2 51e5ee868846 4 weeks ago 53.4MB | remontada007/checkpoint-restore-operator t1 badb1044695a 4 weeks ago 53.3MB | Image: "checkpoint-restore-operator:ci" with ID "sha256:9543ec8a74c4e3fe38c64738eab5e633141e0921e1082539504d96d73e5d791d" not yet present on node "kind-control-plane", loading... [tests.yaml/build-and-deploy] ✅ Success - Main Load Docker image into kind cluster [tests.yaml/build-and-deploy] ⭐ Run Main Replace Placeholder in Manager [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/9] user= workdir= [tests.yaml/build-and-deploy] ✅ Success - Main Replace Placeholder in Manager [tests.yaml/build-and-deploy] ⭐ Run Main Deploy to Kubernetes [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/10] user= workdir= | test -s /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen && /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen --version | grep -q v0.15.0 || \ | GOBIN=/home/ubuntu/os/checkpoint-restore-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.15.0 | /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases | test -s /home/ubuntu/os/checkpoint-restore-operator/bin/kustomize || GOBIN=/home/ubuntu/os/checkpoint-restore-operator/bin GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@v5.4.2 | go: downloading sigs.k8s.io/kustomize/kustomize/v5 v5.4.2 | go: downloading sigs.k8s.io/kustomize/api v0.17.2 | go: downloading sigs.k8s.io/kustomize/cmd/config v0.14.1 | go: downloading sigs.k8s.io/kustomize/kyaml v0.17.1 | go: downloading github.com/go-errors/errors v1.4.2 | go: downloading github.com/blang/semver/v4 v4.0.0 | go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc | go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 | go: downloading github.com/xlab/treeprint v1.2.0 | go: downloading gopkg.in/evanphx/json-patch.v4 v4.12.0 | go: downloading google.golang.org/protobuf v1.33.0 | go: downloading github.com/go-openapi/swag v0.22.4 | go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 | go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 | /home/ubuntu/os/checkpoint-restore-operator/bin/kustomize build config/crd | kubectl apply -f - | customresourcedefinition.apiextensions.k8s.io/checkpointrestoreoperators.criu.org created | test -s /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen && /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen --version | grep -q v0.15.0 || \ | GOBIN=/home/ubuntu/os/checkpoint-restore-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.15.0 | /home/ubuntu/os/checkpoint-restore-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases | cd config/manager && /home/ubuntu/os/checkpoint-restore-operator/bin/kustomize edit set image controller=checkpoint-restore-operator:ci | /home/ubuntu/os/checkpoint-restore-operator/bin/kustomize build config/default | kubectl apply -f - | # Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. | namespace/checkpoint-restore-operator-system created | customresourcedefinition.apiextensions.k8s.io/checkpointrestoreoperators.criu.org unchanged | serviceaccount/checkpoint-restore-operator-controller-manager created | role.rbac.authorization.k8s.io/checkpoint-restore-operator-leader-election-role created | clusterrole.rbac.authorization.k8s.io/checkpoint-restore-operator-manager-role created | clusterrole.rbac.authorization.k8s.io/checkpoint-restore-operator-metrics-reader created | clusterrole.rbac.authorization.k8s.io/checkpoint-restore-operator-proxy-role created | rolebinding.rbac.authorization.k8s.io/checkpoint-restore-operator-leader-election-rolebinding created | clusterrolebinding.rbac.authorization.k8s.io/checkpoint-restore-operator-manager-rolebinding created | clusterrolebinding.rbac.authorization.k8s.io/checkpoint-restore-operator-proxy-rolebinding created | service/checkpoint-restore-operator-controller-manager-metrics-service created | deployment.apps/checkpoint-restore-operator-controller-manager created | checkpoint-podname_namespace-containername-2024-07-24T18:09:34.tar | checkpoint-podname_namespace-containername-2024-07-24T18:09:35.tar | checkpoint-podname_namespace-containername-2024-07-24T18:09:36.tar | checkpoint-podname_namespace-containername-2024-07-24T18:09:37.tar | checkpoint-podname_namespace-containername-2024-07-24T18:09:38.tar [tests.yaml/build-and-deploy] ✅ Success - Main Deploy to Kubernetes [tests.yaml/build-and-deploy] ⭐ Run Main Wait for deployments to be ready [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/11] user= workdir= | Checking if the checkpoint-restore-operator-controller-manager deployment is ready | Waiting for checkpoint-restore-operator-controller-manager deployment to report a ready status (Attempt 2/20) | Waiting for checkpoint-restore-operator-controller-manager deployment to report a ready status (Attempt 3/20) | Waiting for checkpoint-restore-operator-controller-manager deployment to report a ready status (Attempt 4/20) | Waiting for checkpoint-restore-operator-controller-manager deployment to report a ready status (Attempt 5/20) | Available Progressing | The checkpoint-restore-operator-controller-manager deployment is ready [tests.yaml/build-and-deploy] ✅ Success - Main Wait for deployments to be ready [tests.yaml/build-and-deploy] ⭐ Run Main Check resources [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/12] user= workdir= | NAME READY STATUS RESTARTS AGE | pod/checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j 2/2 Running 0 21s | | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE | service/checkpoint-restore-operator-controller-manager-metrics-service ClusterIP 10.96.188.90 8443/TCP 21s | | NAME READY UP-TO-DATE AVAILABLE AGE | deployment.apps/checkpoint-restore-operator-controller-manager 1/1 1 1 21s | | NAME DESIRED CURRENT READY AGE | replicaset.apps/checkpoint-restore-operator-controller-manager-5dbc4bb588 1 1 1 21s [tests.yaml/build-and-deploy] ✅ Success - Main Check resources [tests.yaml/build-and-deploy] ⭐ Run Main Wait for Checkpoint Tar Files to Reduce [tests.yaml/build-and-deploy] 🐳 docker exec cmd=[bash --noprofile --norc -e -o pipefail /var/run/act/workflow/13] user= workdir= | checkpointrestoreoperator.criu.org/checkpointrestoreoperator-sample created | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Checkpoint tar files count is 5 (waiting for 2) | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | Timeout reached: Checkpoint tar files count is still 5 (should be 2) | Fetching checkpoint files for debugging: | total 60 | -rw-r--r-- 1 root root 10240 Jul 24 18:09 checkpoint-podname_namespace-containername-2024-07-24T18:09:34.tar | -rw-r--r-- 1 root root 10240 Jul 24 18:09 checkpoint-podname_namespace-containername-2024-07-24T18:09:35.tar | -rw-r--r-- 1 root root 10240 Jul 24 18:09 checkpoint-podname_namespace-containername-2024-07-24T18:09:36.tar | -rw-r--r-- 1 root root 10240 Jul 24 18:09 checkpoint-podname_namespace-containername-2024-07-24T18:09:37.tar | -rw-r--r-- 1 root root 10240 Jul 24 18:09 checkpoint-podname_namespace-containername-2024-07-24T18:09:38.tar | Fetching logs of operator pods: | Logs for pod checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j: | 2024-07-24T18:13:10Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"} | 2024-07-24T18:13:10Z INFO setup starting manager | 2024-07-24T18:13:10Z INFO Existing file {"name": "ORIGINAL_NAME"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:38+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:39+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-08T20:36:41+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-2024-07-11T21:58:21+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:44+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:45+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:46+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-06-25T00:48:47+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-2024-07-04T20:37:52+05:30.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:06.tar"} | 2024-07-24T18:13:10Z INFO Existing file {"name": "checkpoint-podname_namespace-containername-2024-07-24T23:22:07.tar"} | 2024-07-24T18:13:10Z INFO Watching {"directory": "/var/lib/kubelet/checkpoints"} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerContainer {"maxCheckpointsPerContainer": 10} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerPod {"maxCheckpointsPerPod": 20} | 2024-07-24T18:13:10Z INFO MaxCheckpointsPerNamespace {"maxCheckpointsPerNamespace": 30} | 2024-07-24T18:13:10Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"} | 2024-07-24T18:13:10Z INFO starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"} | I0724 18:13:10.248098 1 leaderelection.go:250] attempting to acquire leader lease checkpoint-restore-operator-system/f1499914.criu.org... | 2024-07-24T18:13:10Z DEBUG events checkpoint-restore-operator-controller-manager-5dbc4bb588-hwf5j_5bea18c9-092c-4754-a91e-10f8443cec97 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"checkpoint-restore-operator-system","name":"f1499914.criu.org","uid":"36dc2c61-ae47-4671-ad93-1e41bfa67bef","apiVersion":"coordination.k8s.io/v1","resourceVersion":"705"}, "reason": "LeaderElection"} | I0724 18:13:10.268995 1 leaderelection.go:260] successfully acquired lease checkpoint-restore-operator-system/f1499914.criu.org | 2024-07-24T18:13:10Z INFO Starting EventSource {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "source": "kind source: *v1.CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting Controller {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator"} | 2024-07-24T18:13:10Z INFO Starting workers {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "worker count": 1} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerContainer {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerContainer": 5} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerPod {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerPod": 10} | 2024-07-24T18:13:20Z INFO Changed MaxCheckpointsPerNamespace {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "maxCheckpointsPerNamespace": 10} | 2024-07-24T18:13:20Z INFO Applying policies immediately {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-q2sv5"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-q2sv5_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "containername"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-podname_namespace-containername-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global container-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "container": "nginx"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-nginx-[0-9][0-9][0-9][0-9]-[0-2][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-6][0-9]*.tar"} | 2024-07-24T18:13:20Z INFO Applying global pod-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pod": "nginx-deployment-85bfd45d84-hq752"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-nginx-deployment-85bfd45d84-hq752_default-*-*.tar"} | 2024-07-24T18:13:20Z INFO Applying global namespace-level policy {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "namespace": "default"} | 2024-07-24T18:13:20Z INFO Looking for checkpoint archives {"controller": "checkpointrestoreoperator", "controllerGroup": "criu.org", "controllerKind": "CheckpointRestoreOperator", "CheckpointRestoreOperator": {"name":"checkpointrestoreoperator-sample","namespace":"default"}, "namespace": "default", "name": "checkpointrestoreoperator-sample", "reconcileID": "a66b28e3-3a92-44cf-bff3-2b7be29faed8", "pattern": "/var/lib/kubelet/checkpoints/checkpoint-*_default-*-*.tar"} [tests.yaml/build-and-deploy] ❌ Failure - Main Wait for Checkpoint Tar Files to Reduce [tests.yaml/build-and-deploy] exitcode '1': failure [tests.yaml/build-and-deploy] 🏁 Job failed Error: Job 'build-and-deploy' failed ```

[adrianreber]

I think it would be helpful if the script test/wait_for_checkpoint_reduction.sh would be a bit more verbose. Maybe how many checkpoint archives it found initially and what it is waiting for. Currently the output is very terse.

[adrianreber]

I am also a fan of shellcheck and shfmt but we can add that in a later step.