huanghaiqing1
commented
1 month ago
Open huanghaiqing1 opened 1 month ago
huanghaiqing1
commented
1 month ago
caseydavenport
commented
1 month ago @huanghaiqing1 are you using custom built images? Or are these the official Calico images we host?
Warning Failed 10m (x3 over 10m) kubelet (combined from similar events): Error: failed to generate container "37eafe5c181ee2edb9cc0559622ed142c75064ba72ba37ffc395d57cf4c43a43" spec: failed to generate spec: no command specified
This suggests potentially a problem with the container, as the image we provide has a CMD directive to provide a default command.
huanghaiqing1
commented
1 month ago On Jul 23, 2024, at 1:01 AM, Casey Davenport @.> wrote: Hello, here I'm using official Calico images. And based on official KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry , I need to modify yaml: tigera-operator.yaml and custom-resources.yaml. But the instructions is too short or maybe I didn't get its point. Here I modify according to the BK's suggestion, but related pods about calico part don't startup normally. @huanghaiqing1 are you using custom built images? Or are these the official Calico images we host? Warning Failed 10m (x3 over 10m) kubelet (combined from similar events): Error: failed to generate container "37eafe5c181ee2edb9cc0559622ed142c75064ba72ba37ffc395d57cf4c43a43" spec: failed to generate spec: no command specified This suggests potentially a problem with the container, as the image we provide has a CMD directive to provide a default command. — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned. Message ID: @.>
caseydavenport
commented
1 month ago Could you run the following for me to see the output:
docker inspect localhost:5000/calico/typha:v3.28.0Specifically looking to see what the "Cmd" section of the output shows.
caseydavenport
commented
1 month ago Also, to be sure - have you specified the --pod-infra-container-image argument on the kubelet by chance?
huanghaiqing1
commented
1 month ago Hello, below is the output. CMD session shows null. I'm not sure the download calico package for private registry is complete or not. Because based on official KB, it requires. But my unzip calico folder about images only gives 7 tar ball.
docker tag quay.io/tigera/operator:v1.34.0 $REGISTRY/tigera/operator:v1.34.0 docker tag calico/typha:v3.28.0 $REGISTRY/calico/typha:v3.28.0 docker tag calico/ctl:v3.28.0 $REGISTRY/calico/ctl:v3.28.0 docker tag calico/node:v3.28.0 $REGISTRY/calico/node:v3.28.0 docker tag calico/cni:v3.28.0 $REGISTRY/calico/cni:v3.28.0 docker tag calico/apiserver:v3.28.0 $REGISTRY/calico/apiserver:v3.28.0 docker tag calico/kube-controllers:v3.28.0 $REGISTRY/calico/kube-controllers:v3.28.0 docker tag calico/dikastes:v3.28.0 $REGISTRY/calico/dikastes:v3.28.0 docker tag calico/pod2daemon-flexvol:v3.28.0 $REGISTRY/calico/pod2daemon-flexvol:v3.28.0 docker tag calico/csi:v3.28.0 $REGISTRY/calico/csi:v3.28.0 docker tag calico/node-driver-registrar:v3.28.0 $REGISTRY/calico/node-driver-registrar:v3.28.0 ***@***.*** images]# docker images | grep -i localhost localhost:5000/calico/pod2daemon v3.28.0 651b8c0ee75e 14 minutes ago 13.4MB localhost:5000/calico/flannel-migration-controller v3.28.0 d0c308187ddb 15 minutes ago 128MB localhost:5000/calico/dikastes v3.28.0 3f03f0b0cf90 19 minutes ago 41.9MB localhost:5000/calico/kube-controllers v3.28.0 2bd71868d777 22 minutes ago 79.2MB localhost:5000/calico/cni v3.28.0 b144a54fe61f 23 minutes ago 209MB localhost:5000/calico/node v3.28.0 5bc4d7581211 26 minutes ago 355MB localhost:5000/calico/typha v3.28.0 337aa4a7808a 29 minutes ago 71.2MB ***@***.*** images]# docker images | grep -i "^calico" calico/pod2daemon v3.28.0 651b8c0ee75e 14 minutes ago 13.4MB calico/flannel-migration-controller v3.28.0 d0c308187ddb 16 minutes ago 128MB calico/dikastes v3.28.0 3f03f0b0cf90 20 minutes ago 41.9MB calico/kube-controllers v3.28.0 2bd71868d777 22 minutes ago 79.2MB calico/cni v3.28.0 b144a54fe61f 23 minutes ago 209MB calico/node v3.28.0 5bc4d7581211 26 minutes ago 355MB calico/typha v3.28.0 337aa4a7808a 29 minutes ago 71.2MB ***@***.*** images]# docker inspect localhost:5000/calico/typha:v3.28.0 [ { "Id": "sha256:337aa4a7808a28ab0c30f9348c9157600e1ed0882b2e780fcbd7f1b4e63626b5", "RepoTags": [ "calico/typha:v3.28.0", "localhost:5000/calico/typha:v3.28.0" ], "RepoDigests": [], "Parent": "", "Comment": "Imported from -", "Created": "2024-07-24T00:55:07.84429967Z", "DockerVersion": "26.1.1", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": null, "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": null }, "Architecture": "amd64", "Os": "linux", "Size": 71173819, "GraphDriver": { "Data": { "MergedDir": "/var/lib/docker/overlay2/9b965a65bcffe20919974634876e91d53ed0f939cf5bf4f3933c9674dbbe6106/merged", "UpperDir": "/var/lib/docker/overlay2/9b965a65bcffe20919974634876e91d53ed0f939cf5bf4f3933c9674dbbe6106/diff", "WorkDir": "/var/lib/docker/overlay2/9b965a65bcffe20919974634876e91d53ed0f939cf5bf4f3933c9674dbbe6106/work" }, "Name": "overlay2" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:5f6c59d25589c88a42e4bafba0dd87729c844bf77be55c5e22847dcd0e65e240" ] }, "Metadata": { "LastTagTime": "2024-07-24T09:24:41.47737321+08:00" } } ]Jul 24, 202412:59 AM，Casey Davenport @.> 写道： Also, to be sure - have you specified the --pod-infra-container-image argument on the kubelet by chance? — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned. Message ID: @.>
huanghaiqing1
commented
1 month ago I didn't include " --pod-infra-container-image" in my kubectl realetd service and I also didn't include it manually. Is there a way to provide an complete calico download tar-ball, which can be used for setup private registray and deploy from local. I'm in China and technically I can't get these images from calico directly, if not use VPN. Jul 24, 202412:59 AM，Casey Davenport @.> 写道： Also, to be sure - have you specified the --pod-infra-container-image argument on the kubelet by chance? — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned. Message ID: @.>
caseydavenport
commented
1 month ago @huanghaiqing1 I downloaded and loaded the images from release-v3.28.0.tar from here: https://github.com/projectcalico/calico/releases/tag/v3.28.0
This is the output I get for the calico/typha image:
[
{
"Id": "sha256:a9372c0f51b54c589e5a16013ed3049b2a052dd6903d72603849fab2c4216fbc",
"RepoTags": [
"calico/typha:v3.28.0"
],
"RepoDigests": [],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2024-05-11T00:15:52.055728329Z",
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "999",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/usr/bin/calico-typha"
],
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"description": "Calico Typha is a fan-out datastore proxy",
"maintainer": "maintainers@tigera.io",
"name": "Calico Typha",
"release": "1",
"summary": "Calico Typha is a fan-out datastore proxy",
"vendor": "Project Calico",
"version": "v3.28.0"
}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 71143933,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/e4a8269bcc993b26c93df9883fa4f2386e26673d1cf6b73c5dc3d69a6f2c286a/diff",
"MergedDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/merged",
"UpperDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/diff",
"WorkDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:29ebc113185d6c09c2f84abf7d4fbbb49e2d6e4a169c0a5f9d14d1358d70827e",
"sha256:31719a1450dd2c929a14d1bd057b3b663cb7b99286198effa4e9e10b62007641"
]
},
"Metadata": {
"LastTagTime": "0001-01-01T00:00:00Z"
}
}
]Note that the "Cmd" section is not null. I think the images you have loaded into your registry have been modified in some way to remove the embedded command.
huanghaiqing1
commented
1 month ago Hello, I compared the release-v3.28.0.tar with my current downloaded one. It’s same size. And I use "docker import calico-typha.tar calico/typha:v3.28.0” to import it and other tar.ball as docker images and then push them to my private registry. But inspect calico/typha, the cmd becomes “null”. What’s the problem? If you think this is the cause for invalid applying tigera-operator.yaml and custom-resources.yaml based on KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry. How should I bypass?


2024年7月25日 01:13，Casey Davenport @.***> 写道：
@huanghaiqing1 https://github.com/huanghaiqing1 I downloaded and loaded the images from release-v3.28.0.tar from here: https://github.com/projectcalico/calico/releases/tag/v3.28.0
This is the output I get for the calico/typha image:
[ { "Id": "sha256:a9372c0f51b54c589e5a16013ed3049b2a052dd6903d72603849fab2c4216fbc", "RepoTags": [ "calico/typha:v3.28.0" ], "RepoDigests": [], "Parent": "", "Comment": "buildkit.dockerfile.v0", "Created": "2024-05-11T00:15:52.055728329Z", "DockerVersion": "", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "999", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "/usr/bin/calico-typha" ], "ArgsEscaped": true, "Image": "", "Volumes": null, "WorkingDir": "/", "Entrypoint": null, "OnBuild": null, "Labels": { "description": "Calico Typha is a fan-out datastore proxy", "maintainer": @.***", "name": "Calico Typha", "release": "1", "summary": "Calico Typha is a fan-out datastore proxy", "vendor": "Project Calico", "version": "v3.28.0" } }, "Architecture": "amd64", "Os": "linux", "Size": 71143933, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/e4a8269bcc993b26c93df9883fa4f2386e26673d1cf6b73c5dc3d69a6f2c286a/diff", "MergedDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/merged", "UpperDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/diff", "WorkDir": "/var/lib/docker/overlay2/7c4bf0a7dee90c680d744ff2c0ec6476f661281ae6a28e5b1a9bd9ceac7ddd55/work" }, "Name": "overlay2" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:29ebc113185d6c09c2f84abf7d4fbbb49e2d6e4a169c0a5f9d14d1358d70827e", "sha256:31719a1450dd2c929a14d1bd057b3b663cb7b99286198effa4e9e10b62007641" ] }, "Metadata": { "LastTagTime": "0001-01-01T00:00:00Z" } } ] Note that the "Cmd" section is not null. I think the images you have loaded into your registry have been modified in some way to remove the embedded command.
— Reply to this email directly, view it on GitHub https://github.com/projectcalico/calico/issues/9017#issuecomment-2248521013, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHFAPJE222UBI4N4OZFSBOLZN7ODLAVCNFSM6AAAAABK76WBSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYGUZDCMBRGM. You are receiving this because you were mentioned.
caseydavenport
commented
1 month ago @huanghaiqing1 I think the problem is the docker import command.
docker import simply loads the filesystem contained within the tar file, but does not retain any metadata from the original image.
I would recommend using docker load instead, which will load the image as well as the metadata (including the Cmd).
For example:
docker load < calico-typha.tarVery thanks for your remind. Here I use docker load to push imanges to local from downlaod folder release-v3.28.0/images. And I also create tag and push to my local private registry. And I can use "docker inspect" to see related local dockers have conten in "cmd". But when I follow KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry and use customized yamls: tigera-operator.yaml and custom-resources.yaml. But there are still different warnings during calico pods setup in k8s.
my steps:
1. kubeadm init --pod-network-cidr=192.1.0.0/16 --image-repository registry.aliyuncs.com/google_containers 2. adjust tigera-operator.yaml and custom-resources.yaml then create calico pods from local registry. You can also refer to attached modifed yaml files.
image: quay.io/tigera/operator:v1.34.0
cidr: 192.168.0.0/1617,19d16 < variant: Calico < imagePath: calico < registry: localhost:5000 29a27
1. kubectl create -f ./tigera-operator.yaml kubectl create -f ./custom-resources.yaml
2. kubectl get pod -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-b7fd956f5-xpqgg 0/1 Pending 0 15s calico-node-c975x 0/1 Init:CreateContainerError 0 16s calico-typha-55458bc957-crc4k 0/1 CreateContainerError 0 18s csi-node-driver-2gbzt 0/2 ContainerCreating 0 16s
4. I output pod descrption and local docker inspect info to output.log for your reference. Maybe you can give some additional advice. Thanks.
change in tiger-operator.yaml:
Configure use of your image registry | Calico Documentationhttps://docs.tigera.io/calico/latest/operations/image-options/alternate-registry Configure Calico to pull images from a public or private registry. docs.tigera.io
From: Casey Davenport @.> Sent: Friday, July 26, 2024 11:06 PM To: projectcalico/calico @.> Cc: Huang Haiqing @.>; Mention @.> Subject: Re: [projectcalico/calico] can't setup calico related pods in k8s (Issue #9017)
@huanghaiqing1https://github.com/huanghaiqing1 I think the problem is the docker import command.
docker import simply loads the filesystem contained within the tar file, but does not retain any metadata from the original image.
I would recommend using docker load instead, which will load the image as well as the metadata (including the Cmd).
For example:
docker load < calico-typha.tar
— Reply to this email directly, view it on GitHubhttps://github.com/projectcalico/calico/issues/9017#issuecomment-2252961278, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHFAPJH33OJRH5NQ4PKJ2MLZOJQX5AVCNFSM6AAAAABK76WBSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJSHE3DCMRXHA. You are receiving this because you were mentioned.
caseydavenport
commented
1 month ago @huanghaiqing1 I don't see any diagnostics on your last comment - you likely need to embed them as links into the GitHub post rather than via email.
huanghaiqing1
commented
1 month ago Here I submit related attachments here for your reference.
I use docker load to push imanges to local from downlaod folder release-v3.28.0/images. And I also create tag and push to my local private registry. And I can use "docker inspect" to see related local dockers have conten in "cmd". But when I follow KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry and use customized yamls: tigera-operator.yaml and custom-resources.yaml. But there are still different warnings during calico pods setup in k8s. What I'm curious is even I switch to docker load mode, I still see "spec: failed to generate spec: no command specified" when use "kubectl describe pod calico-typha-67f78bf575-2d6jz -n calico-system". But when I use "docker inspect calico/typha:v3.28.0", I can see "cmd="/usr/bin/calico-typha" in image defination.
my steps:
1.kubeadm init --pod-network-cidr=192.1.0.0/16 --image-repository registry.aliyuncs.com/google_containers 2.adjust tigera-operator.yaml and custom-resources.yaml, then create calico pods from local registry. You can also refer to attached modifed yaml files from here.
image: quay.io/tigera/operator:v1.34.0
cidr: 192.168.0.0/1617,19d16 < variant: Calico < imagePath: calico < registry: localhost:5000 29a27
3.kubectl create -f ./tigera-operator.yaml kubectl create -f ./custom-resources.yaml
kubectl get pod -n calico-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-b7fd956f5-xpqgg 0/1 Pending 0 15s calico-node-c975x 0/1 Init:CreateContainerError 0 16s calico-typha-55458bc957-crc4k 0/1 CreateContainerError 0 18s csi-node-driver-2gbzt 0/2 ContainerCreating 0 16s
huanghaiqing1
commented
1 month ago Hello, I have updated in github's comment. You can check there.
From: Casey Davenport @.> Sent: Wednesday, July 31, 2024 5:01 AM To: projectcalico/calico @.> Cc: Huang Haiqing @.>; Mention @.> Subject: Re: [projectcalico/calico] can't setup calico related pods in k8s (Issue #9017)
@huanghaiqing1https://github.com/huanghaiqing1 I don't see any diagnostics on your last comment - you likely need to embed them as links into the GitHub post rather than via email.
— Reply to this email directly, view it on GitHubhttps://github.com/projectcalico/calico/issues/9017#issuecomment-2259202265, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHFAPJFJIW4XWRIDV3T4HNLZO75LDAVCNFSM6AAAAABK76WBSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGIYDEMRWGU. You are receiving this because you were mentioned.Message ID: @.***>
caseydavenport
commented
1 month ago when use "kubectl describe pod calico-typha-67f78bf575-2d6jz -n calico-system". But when I use "docker inspect calico/typha:v3.28.0", I can see "cmd="/usr/bin/calico-typha" in image defination.
What about docker inspect localhost:5000/calico/typha:v3.28.0?
I think your custom-resources.yaml and tigera-operator.yaml both look correct to me. I suspect that this is a problem with the way the images are loaded into the registry, and it not pushing the manifest contents correct.
huanghaiqing1
commented
4 weeks ago [root@k8sma ~]# docker inspect localhost:5000/calico/typha:v3.28.0
[
{
"Id": "sha256:a9372c0f51b54c589e5a16013ed3049b2a052dd6903d72603849fab2c4216fbc",
"RepoTags": [
"calico/typha:v3.28.0",
"localhost:5000/calico/typha:v3.28.0"
],
"RepoDigests": [
"localhost:5000/calico/typha@sha256:dc37e0ef67d141bea4bccee6f6488007b5ce7f560768a654246fa65ccf157b63"
],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2024-05-11T00:15:52.055728329Z",
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "999",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/usr/bin/calico-typha"
],
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"description": "Calico Typha is a fan-out datastore proxy",
"maintainer": "maintainers@tigera.io",
"name": "Calico Typha",
"release": "1",
"summary": "Calico Typha is a fan-out datastore proxy",
"vendor": "Project Calico",
"version": "v3.28.0"
}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 71143933,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/cc4b4480639c02341b04e33d0e4b840838fcf66973876738235bfd75e036d628/diff",
"MergedDir": "/var/lib/docker/overlay2/b005e6ed593572d788e0ae6b18d2c7d6b6ea7a5c24986e5dd96eac2063d37cfa/merged",
"UpperDir": "/var/lib/docker/overlay2/b005e6ed593572d788e0ae6b18d2c7d6b6ea7a5c24986e5dd96eac2063d37cfa/diff",
"WorkDir": "/var/lib/docker/overlay2/b005e6ed593572d788e0ae6b18d2c7d6b6ea7a5c24986e5dd96eac2063d37cfa/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:29ebc113185d6c09c2f84abf7d4fbbb49e2d6e4a169c0a5f9d14d1358d70827e",
"sha256:31719a1450dd2c929a14d1bd057b3b663cb7b99286198effa4e9e10b62007641"
]
},
"Metadata": {
"LastTagTime": "2024-07-27T07:50:01.289194125+08:00"
}
}
]
[root@k8sma ~]#
coutinhop
commented
1 week ago @huanghaiqing1 do you change the image contents/Dockerfile in any way before pushing to your custom registry?
huanghaiqing1
commented
6 days ago Hello, not clear about your detail intention. I modifed tigera-operator.yaml and custom-resources.yaml. based on KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry. Then try create and failed to setup calico pods inside k8s cluster. Do you have a example tigera-operator.yaml and custom-resources.yaml for my reference?
Here we are using calico as network plugin for k8s based on KB: https://docs.tigera.io/calico/latest/operations/image-options/alternate-registry by using local registry as calico dockers images. We adjust based on that KB in tigera-operator.yaml and custom-resources.yaml. But calico related pods can't be in normal status after apply custom-resources.yaml. Below are the detail error info, any suggestion?