Guide

[mind-mind]

I'm very interested in your thesis and I want to use it, but there is no guide, so it hard to try to make it work. Can you please give me a guide? Or what I should read to make this code work on migration.

Also, I already install K8s using CRI-O but have no idea what to do next. I think if you use CRIU, it has to use Podman container right since docker cannot use with CRIU?

Thank you so much.

[schrej]

Hey, glad to hear that! I've written a somewhat rough guide for it here. Not sure if it's still 100% accurate, so if you have any issues, please let me know and I'll try to help you. You can also reach me on k8s slack (also @schrej). You should probably build the binaries yourself, I'm pretty sure the ones available for download are outdated.

Edit: @vutuong has written a guide as well. I haven't tested it, but it looks good after a brief look: https://github.com/kubernetes/kubernetes/issues/3949#issuecomment-749936235 Again, I recommend building your own binaries from my fork at https://github.com/schrej/kubernetes

The thesis is available on ResearchGate (in case someone else finds this issue).

[mind-mind]

Okay. Since one part of my thesis has to use pod migrating K8s, I'II try to use this and ask you a question. Thank you so much.

[mind-mind]

Hi again, I follow this source

https://github.com/SSU-DCN/podmigration-operator/blob/main/init-cluster-containerd-CRIU.md

I got the problem while trying to install contained. Any idea?

[mind-mind]

I also follow this step

But I got

I'm curious that when directory /var/lib/kubelet/migration/ was create?

Thank you for your answer in advance.

[vutuong]

Hi @mind-mind, I guess that you are using go-version 1.16.3 (or higher). I found that there is a problem with golang new version (https://blog.golang.org/go116-module-changes). I fixed it with this command:

$ cd containerd-cri/
$ go env -w GO111MODULE=auto

After that, you can continue with:

$ go get github.com/containerd/cri/cmd/containerd
$ make
...

The contained binaries should be found at _output/ directory. I also wrote in the document that there are two ways to get the containerd binaries. If you still can not build the containerd birary from @schrej Jakob source code, you can ask @schrej to download the binaries directly from his site:

$ wget https://k8s-pod-migration.obs.eu-de.otc.t-systems.com/v2/containerd

[vutuong]

I also follow this step

But I got

I'm curious that when directory /var/lib/kubelet/migration/ was create?

Thank you for your answer in advance.

Yes, sorry. This is my mistake. You should create the /var/lib/kubelet/migration/ directory at every node in the cluster first before doing step 9 with:

$ sudo mkdir /var/lib/kubelet/migration/
$ sudo chmod 777 /var/lib/kubelet/migration

[mind-mind]

Thank you @vutuong

I already did the setup and in this source: https://github.com/SSU-DCN/podmigration-operator/blob/main/init-cluster-containerd-CRIU.md

and follow this video https://www.youtube.com/watch?v=M4Ik7aUKhas

I use kubectl migrate

But I got this. Seems like no command migrate. What should I do?

[mind-mind]

I already create a migration directory in every node as you recommend.

[vutuong]

Thank you @vutuong

I already did the setup and in this source: https://github.com/SSU-DCN/podmigration-operator/blob/main/init-cluster-containerd-CRIU.md

and follow this video https://www.youtube.com/watch?v=M4Ik7aUKhas

I use kubectl migrate

But I got this. Seems like no command migrate. What should I do?

Hi, since as you can see in my guide (https://github.com/SSU-DCN/podmigration-operator), you should follow the How to run part ( except run the GUI) to make something like the demo video:

• Firstly, download the source code in the repo:

  $ git clone https://github.com/SSU-DCN/podmigration-operator.git

• In one terminal, run the podmigration controller:

  $ cd podmigration-operator/
  $ make manifests
  $ sudo snap install kustomize
  $ sudo apt-get install gcc
  $ make install
  $ make run

• In other terminal run the API server to listen the request from kubectl migrate command:

  $ cd podmigration-operator/
  $ go run ./api-server/cmd/main.go

• Finally, because there is no kubectl migrate in the default kubernetes, thus; I wrote the kubectl plugin for it. To install kubectl migrate/checkpoint command, follow the guide at https://github.com/SSU-DCN/podmigration-operator/tree/main/kubectl-plugin P/S: I should check your cluster to make sure it's boot well before all. Every node should be in Ready status.

[mind-mind]

Thank you. Yeah, I know that K8s doesn't have command migrate by default and you wrote the kubectl plugin. I have to set up this on every node right?

[vutuong]

Thank you. Yeah, I know that K8s doesn't have command migrate by default and you wrote the kubectl plugin. I have to set up this on every node right?

No, you don't have to. Kubectl is used at the master node. So you just only have to set up these commands in the master node.

[mind-mind]

@vutuong Thank you though, but I got the problem when trying "make run"

[vutuong]

@vutuong Thank you thought, but I got the problem when trying "make run"

<img alt="image" width="839" src="https://user-images.githubusercontent.com/68138683/119264841-ee639600-bc0e-11eb-9578-ad56efd0f6a9.png"

@vutuong Thank you thought, but I got the problem when trying "make run"

Maybe, there is a conflict with the go version. Anyway, the latency-monitor is my own monitor testing directory, which is included in my project but it is not a podmigration component. Therefore, you can remove this directory and make run again.

[mind-mind]

Remove "latency-monitor" made it works now. Thank you so much.

[mind-mind]

Hi,

I got this

It seems to work until pull image. I have no idea.

Maybe should I fix my YAML file?

I tried to use your YAML file on "podmigration-operator/config/samples/migration-example" but it failed to assign to the node.

My status node seems ok

Here is my YAML file:

apiVersion: v1 kind: Pod metadata: name: rnn1 labels: app: custom spec: containers:

• name: rnndnn image: mind01docker/deep:rnn command: ["python3","rnn.py"] volumeMounts:

  • name: dnn mountPath: "/home/ubuntu"

  volumes:

• name: dnn persistentVolumeClaim: claimName: deep restartPolicy: Never

[mind-mind]

Also, I'm curious about something. Did you test on Ubuntu before?

https://github.com/checkpoint-restore/criu/issues/1420

I did try CRIU on podman before, it doesn't work on Ubuntu, kernel problem. I use you migrate API seems to work except for has status StartError.

Is it possible that because of the same reason? Your Demo uses Ubuntu whether or not?

[vutuong]

• Please check my example first to make sure your settings are correct. The reason why it failed to assign to the node is I used the nodeSelector in the yaml file. You should change the worker node name in the yaml file to your_worker_node_name:

  ...
  nodeSelector:
  kubernetes.io/hostname: your_worker_node_name

• Btw, I found that you use the volumes:. Since you migrate the pod to the new node, the mount path is not included in the new node. Therefore, it causes the problem. To solve that, you should configure to share the mount folder between two nodes. Please check my example for Redmine application at https://github.com/SSU-DCN/podmigration-operator/blob/main/config/samples/checkpoint-example/redmine.yaml. In which, I move the mount path to the /var/lib/kubernetes/migration directory, thus; it will be shared between two nodes.

[mind-mind]

@vutuong,

I have a problem with your example YAML file either, so I might set up something wrong.

I think the log seems normal to me. Not sure that what I miss?

[mind-mind]

Here is a log of the pod

[vutuong]

@vutuong,

I have a problem with your example YAML file either, so I might set up something wrong.

I think the log seems normal to me. Not sure that what I miss?

Since I not sure what is your problem since the logs is not enough for me. Check kubelet log with journalctl -fu kubelet

• However, I just pushed the bug-fixed version. Could you please pull the repo again, and restart the controller with make run and restart the api-server with go run ./api-server/cmd/main.go ? Then check with my example. In your case, you should configure to share the volumes path.... If you don't mind please reach me on k8s slack @Tuong Vu or another chat platform I will try to help you.

[mind-mind]

I don't mind at all, how can I reach you on K8s slack or another chat platform?

[mind-mind]

Here is the log that you want. While waiting for your reply, I think I should try to run on Debian10.

I told you about I knew that CRIU cannot use on Ubuntu before. I think if you didn't do anything with CRIU except for integrating, possibly it shouldn't work either right?

Expectedly, you already test on Ubuntu and it works.

Ref: checkpoint-restore/criu#1420

[vutuong]

I don't mind as all, how can I reach you on K8s slack or another chat platform?

Please send me a mess in the k8s slack : https://app.slack.com/client/T09NY5SBT/C09NXKJKA ID: @Tuong Vu I used the ubuntu-server and it's working fine as the demo video.

[mind-mind]

I don't know how to add you by ID, could you please https://join.slack.com/t/thammasatgroup/shared_invite/zt-r1n08pkw-8kT_hvLKzsylLbfLLxHOWw go to this link instead.

[mind-mind]

It already works. Thanks for @vutuong and @schrej

By the way, I wrote this https://gitlab.com/mindhappyokk/pod-migrating reference from you guys to set up pod migrating. It is easy to install. Hope you don't mind.

[tonyliu666]

I also follow this step

But I got

I'm curious that when directory /var/lib/kubelet/migration/ was create? Thank you for your answer in advance.

Yes, sorry. This is my mistake. You should create the /var/lib/kubelet/migration/ directory at every node in the cluster first before doing step 9 with:

$ sudo mkdir /var/lib/kubelet/migration/
$ sudo chmod 777 /var/lib/kubelet/migration

Sorry will it bother you if I ask you some questions. How to connect to every node in the cluster ? For example,use ssh to get into the node.As I did is like following picture: But I don't know where I did something wrong. Could you give me some advices,thanks~

[schrej]

That's a ssh authentication issue, and not really related to this project.

How to access nodes using ssh heavily depends on how you installed them. If you didn't provide a public key that you have the private key to, it's probably not going to be possible like that. You can try this though: https://github.com/kvaps/kubectl-node-shell

[120L020314]

@mind-mind i am very thankful for your guide! because the pod migration is also very importent for my thesis. and could you tell me your cluster's configuration detail? for example:your version of the ubuntu 、k8s and docker/podman ,thank you very much.