joedborg
commented
4 years ago Sadly this is a bug in containerd https://github.com/containerd/cri/issues/1433
We’ll have to wait until 1.4 comes out to bundle into microk8s.
Closed muhammadnasr closed 4 years ago
joedborg
commented
4 years ago Sadly this is a bug in containerd https://github.com/containerd/cri/issues/1433
We’ll have to wait until 1.4 comes out to bundle into microk8s.
muhammadnasr
commented
4 years ago Thank for the prompt reply.
Is there any workarounds? For ex: using self-signed crt or skip crt verification
muhammadnasr
commented
4 years ago just tried skip tls verify: [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://registry-1.docker.io", ] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."http://172.22.0.92:5050"] endpoint = ["http://172.22.0.92:5050"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.22.0.92:5050"] endpoint = ["http://172.22.0.92:5050"] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.configs."172.22.0.92"] [plugins."io.containerd.grpc.v1.cri".registry.configs."172.22.0.92".tls] insecure_skip_verify = true
but it give falcom@node1:~$ microk8s.ctr image pull 172.22.0.92:5050/my-busybox:latest ctr: failed to resolve reference "172.22.0.92:5050/my-busybox:latest": failed to do request: Head "https://172.22.0.92:5050/v2/my-busybox/manifests/latest": x509: certificate signed by unknown authority
balchua
commented
4 years ago What i normally do is to add that certificate to your system's trusted certificate.
joedborg
commented
4 years ago Ah thanks @balchua. Would you happen to have 5 mins to list the steps? I can then add that to our docs until containerd 1.4 is released.
balchua
commented
4 years ago Sure. I will have to recall the exact steps coz its not something i do all the time hahaha. I will post the steps here.
joedborg
commented
4 years ago That's awesome, thanks so much.
balchua
commented
4 years ago Here's roughly how i add a self signed cert in ubuntu.
/usr/local/share/ca-certificates/.crt format into the directory you just created.sudo update-ca-certificates
Any other self signed cert signed by this CA is now host trusted. Make sure you do this to all the nodes in the kubernetes cluster.
muhammadnasr
commented
4 years ago Thank you for your support.
Are you sure these will work with microk8s? it uses DinD and containerd so we will have to do that to all new pods which is technically not practical. Correct me if I am wrong.
On Thu, Jul 30, 2020 at 1:02 AM balchua notifications@github.com wrote:
Here's roughly how i add a self signed cert in ubuntu.
- Go to /usr/local/share/ca-certificates/
- Create a new directory (ex. "my-ca") to identify the self signed CA.
- Copy the CA's certificate in .crt format into the directory you just created.
- Run sudo update-ca-certificates Any other self signed cert signed by this CA is now host trusted.
Make sure you do this to all the nodes in the kubernetes cluster.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ubuntu/microk8s/issues/1444#issuecomment-665953025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABIMCCPLXYFKBVXQR3LEGTR6CL75ANCNFSM4PK6JLTA .
balchua
commented
4 years ago MicroK8s is not using DinD. All cluster components are in fact managed by systemd, including containerd.
decewei
commented
4 years ago Sadly this is a bug in containerd containerd/cri#1433
We’ll have to wait until 1.4 comes out to bundle into microk8s.
Which version of microk8s has this issue? We used revision 1515. It doesn't have this issue. Now it automatically update to 1550, and it stops working. We updated to latest revision 1551 which is using microk8s.ctr 1.3.4, the issue still preserves. Wasn't sure why it wasn't an issue in 1515. It's kind of a big roadblock for us now.
joedborg
commented
4 years ago @decewei which channel / track are you following?
decewei
commented
4 years ago @decewei which channel / track are you following?
Was 1.18/stable. For revision 1551, we switched to the stable/ channel.
joedborg
commented
4 years ago @decewei I would suggest reverting to 1.18/stable until this is fixed upstream. From what I understand, it's been a issue in multiple (and not necessarily concurrent) versions of containerd as there are multiple bugs and PRs to address it.
decewei
commented
4 years ago @decewei I would suggest reverting to 1.18/stable until this is fixed upstream. From what I understand, it's been a issue in multiple (and not necessarily concurrent) versions of containerd as there are multiple bugs and PRs to address it.
Thanks for the prompt reply. The thing is 1.18/stable which the team updated over the weekend was also broken... we didn't have this last week....
ktsakalozos
commented
4 years ago The 1.18 branch [1] has been frozen for some time now. From revision 1515 to 1550 nothing went into that branch. Containerd is pinned to specific commit.
Where can I read more on your setup (setup gitlab autodevops with microk8s)? Is it something I can do easily in order to reproduce the error? I have not used gitlab in the past.
decewei
commented
4 years ago The 1.18 branch [1] has been frozen for some time now. From revision 1515 to 1550 nothing went into that branch. Containerd is pinned to specific commit.
Where can I read more on your setup (setup gitlab autodevops with microk8s)? Is it something I can do easily in order to reproduce the error? I have not used gitlab in the past.
We didn't use gitlab autodevops with microk8s. Our setup is only a snap install microk8s --channel=1.18/stable.
We removed the last release install, that's no where to find. But it is definitely revision 1515.
The steps to reproduce this issue in revision 1550 are: (I reproduced this on my home vm machine too):
snap install microk8s --channel=1.18/stable (install the revision 1550)microk8s.ctr image pull registry_ip:port/yourimageWithout --plain-http, this generates ctr: failed to resolve reference "ip/yourimage": failed to do request: Head "https://ip/v2/yourimage/manifests/v1": x509 certificate singed by unknown authority.. It seems for insecure registry, it does NOT check the endpoint in the template, bu automatically add https. This wasn't the behaviour in revision 1515, we could pull from the registry with no problem using http.
decewei
commented
4 years ago Some thoughts, I've looked through some code in containerd/containerd and containerd/cri.
It seems like the error message we get is coming from containerd. E.g. Here in containerd/containerd pull.go, Pull calls fetch which causes the failed to resolve reference error indicating above. There is no endpoint checking from containerd-template.toml.
Whereas in code of cri here checks the endpoint when it gets resolver.
I guess the logic should be cri check endpoint resolves -> containerd do its logic. Seems like cri fails to resolve the host and protocol before it passes to containerd.
muhammadnasr
commented
4 years ago I have the same problem with the gitlab private repository that is connected to microK8s cluster for autodevoos I tried configuration using insecure and using secured self signed certificate but both does not work.
I am think of trying k3s but first will try the mention solution of installing certificate on my ubuntu nodes
في جمعة، 31 يوليو، 2020 في 6:03 م، كتب decewei notifications@github.com:
Some thoughts, I've looked through some code in containerd/containerd and container/cri.
It seems like the error message we get is coming from containerd. E.g. Here in containerd/containerd pull.go https://github.com/containerd/containerd/blob/d184a0a3430dc4a17a47cce37fb36126ac0c699a/pull.go, Pull calls fetch which causes the failed to reference error indicating above. There is no endpoint checking.
Whereas in code of cri here https://github.com/containerd/cri/blob/release/1.2/pkg/server/image_pull.go checks the endpoint when it gets resolver.
It seems like microk8s.ctr is using the former logic rather than the latter logic. I'm not sure where the invoke of these functions happening. Could this be the problem?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ubuntu/microk8s/issues/1444#issuecomment-667167048, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABIMCCVMDDTSIZMNIS7AVDR6LMMRANCNFSM4PK6JLTA .
muhammadnasr
commented
4 years ago As suggested by @balchua
في جمعة، 31 يوليو، 2020 في 6:25 م، كتب Muhammad Hassan Nasr < muhammad.hassan.nasr@gmail.com>:
I have the same problem with the gitlab private repository that is connected to microK8s cluster for autodevoos I tried configuration using insecure and using secured self signed certificate but both does not work.
I am think of trying k3s but first will try the mention solution of installing certificate on my ubuntu nodes
في جمعة، 31 يوليو، 2020 في 6:03 م، كتب decewei notifications@github.com:
Some thoughts, I've looked through some code in containerd/containerd and container/cri.
It seems like the error message we get is coming from containerd. E.g. Here in containerd/containerd pull.go https://github.com/containerd/containerd/blob/d184a0a3430dc4a17a47cce37fb36126ac0c699a/pull.go, Pull calls fetch which causes the failed to reference error indicating above. There is no endpoint checking.
Whereas in code of cri here https://github.com/containerd/cri/blob/release/1.2/pkg/server/image_pull.go checks the endpoint when it gets resolver.
It seems like microk8s.ctr is using the former logic rather than the latter logic. I'm not sure where the invoke of these functions happening. Could this be the problem?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ubuntu/microk8s/issues/1444#issuecomment-667167048, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABIMCCVMDDTSIZMNIS7AVDR6LMMRANCNFSM4PK6JLTA .
decewei
commented
4 years ago As suggested by @balchua في جمعة، 31 يوليو، 2020 في 6:25 م، كتب Muhammad Hassan Nasr < muhammad.hassan.nasr@gmail.com>: … I have the same problem with the gitlab private repository that is connected to microK8s cluster for autodevoos I tried configuration using insecure and using secured self signed certificate but both does not work. I am think of trying k3s but first will try the mention solution of installing certificate on my ubuntu nodes في جمعة، 31 يوليو، 2020 في 6:03 م، كتب decewei @.***>: > Some thoughts, I've looked through some code in containerd/containerd and > container/cri. > > It seems like the error message we get is coming from containerd. E.g. Here > in containerd/containerd pull.go > https://github.com/containerd/containerd/blob/d184a0a3430dc4a17a47cce37fb36126ac0c699a/pull.go, > Pull calls fetch which causes the failed to reference error indicating > above. There is no endpoint checking. > > Whereas in code of cri here > https://github.com/containerd/cri/blob/release/1.2/pkg/server/image_pull.go > checks the endpoint when it gets resolver. > > It seems like microk8s.ctr is using the former logic rather than the > latter logic. I'm not sure where the invoke of these functions happening. > Could this be the problem? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#1444 (comment)>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/AABIMCCVMDDTSIZMNIS7AVDR6LMMRANCNFSM4PK6JLTA > . >
Thanks. We've tried adding certificates on RHEL7. But we are still getting x509 issue when it falls back to https. Which channel and revision of microk8s are you using? What about the ctr version?
ktsakalozos
commented
4 years ago The steps to reproduce this issue in revision 1550 are: (I reproduced this on my home vm machine too):
@decewei I must be missing something. I am following the instructions on https://microk8s.io/docs/registry-built-in with no problems on an AWS machine. Here is a full run:
ubuntu@ip-172-31-40-63:~/microk8s$ snap install microk8s --channel=1.18/stable --classic
microk8s (1.18/stable) v1.18.6 from Canonical✓ installed
ubuntu@ip-172-31-40-63:~/microk8s$ microk8s enable registry:size=40Gi
Enabling the private registry
Enabling default storage class
deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
Applying registry manifest
namespace/container-registry created
persistentvolumeclaim/registry-claim created
deployment.apps/registry created
service/registry created
The registry is enabled
ubuntu@ip-172-31-40-63:~/microk8s$ cd ../tmp/
ubuntu@ip-172-31-40-63:~/tmp$ cat > Dockerfile
FROM nginx
ubuntu@ip-172-31-40-63:~/tmp$ sudo docker build . -t localhost:32000/mynginx:registry
Sending build context to Docker daemon 12.29kB
Step 1/1 : FROM nginx
latest: Pulling from library/nginx
6ec8c9369e08: Pull complete
d3cb09a117e5: Pull complete
7ef2f1459687: Pull complete
e4d1bf8c9482: Pull complete
795301d236d7: Pull complete
Digest: sha256:0e188877aa60537d1a1c6484b8c3929cfe09988145327ee47e8e91ddf6f76f5c
Status: Downloaded newer image for nginx:latest
---> 8cf1bfb43ff5
Successfully built 8cf1bfb43ff5
Successfully tagged localhost:32000/mynginx:registry
ubuntu@ip-172-31-40-63:~/tmp$ docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/json: dial unix /var/run/docker.sock: connect: permission denied
ubuntu@ip-172-31-40-63:~/tmp$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
busybox latest 018c9d7b792b 3 days ago 1.22MB
localhost:32000/my-busybox latest 018c9d7b792b 3 days ago 1.22MB
nginx latest 8cf1bfb43ff5 9 days ago 132MB
localhost:32000/mynginx registry 8cf1bfb43ff5 9 days ago 132MB
busybox <none> c7c37e472d31 4 weeks ago 1.22MB
localhost:32000/my-busybox <none> c7c37e472d31 4 weeks ago 1.22MB
ubuntu@ip-172-31-40-63:~/tmp$ sudo docker push localhost:32000/mynginx
The push refers to repository [localhost:32000/mynginx]
98b4c818e603: Pushed
1698c1b7e3e6: Pushed
227442bb48dc: Pushed
d899691659b0: Pushed
95ef25a32043: Pushed
registry: digest: sha256:deb724a427ea79face617392a5a471fdcb4cdb57f971ee6b7e492b90fecb199f size: 1362
ubuntu@ip-172-31-40-63:~/tmp$
cat > d.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: localhost:32000/mynginx:registry
ports:
- containerPort: 80
ubuntu@ip-172-31-40-63:~/tmp$ microk8s.kubectl apply -f ./d.yaml
deployment.apps/nginx-deployment created
ubuntu@ip-172-31-40-63:~/tmp$ microk8s.kubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
container-registry pod/registry-7cf58dcdcc-l8p52 1/1 Running 0 9m43s
default pod/nginx-deployment-7b77749f6-s52vl 1/1 Running 0 12s
kube-system pod/hostpath-provisioner-75fdc8fccd-rns59 1/1 Running 0 9m43s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
container-registry service/registry NodePort 10.152.183.11 <none> 5000:32000/TCP 9m47s
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 9m50s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
container-registry deployment.apps/registry 1/1 1 1 9m48s
default deployment.apps/nginx-deployment 1/1 1 1 12s
kube-system deployment.apps/hostpath-provisioner 1/1 1 1 9m48s
NAMESPACE NAME DESIRED CURRENT READY AGE
container-registry replicaset.apps/registry-7cf58dcdcc 1 1 1 9m44s
default replicaset.apps/nginx-deployment-7b77749f6 1 1 1 12s
kube-system replicaset.apps/hostpath-provisioner-75fdc8fccd 1 1 1 9m44s
ubuntu@ip-172-31-40-63:~/tmp$ microk8s.ctr image pull localhost:32000/mynginx:registry
localhost:32000/mynginx:registry: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:deb724a427ea79face617392a5a471fdcb4cdb57f971ee6b7e492b90fecb199f: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:795301d236d7c3cd7c21f28faa8e9dc7f6381c980a0241cae40614986ee070b5: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d3cb09a117e500ee7466b6d21351c35321c9443442d21404267bc9e338bf86b6: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7ef2f145968791b3e117e32ead3685173095d01e8dd887225f857d7fea64cfc8: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e4d1bf8c948256cc69d7121bf623603039d70e616a90ac92eb690aed97918e58: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:8cf1bfb43ff5d9b05af9b6b63983440f137c6a08320fa7592197c1474ef30241: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6ec8c9369e08152361a01729f2c8a1e7aae898426c6e67267f41894bf9524827: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:deb724a427ea79face617392a5a471fdcb4cdb57f971ee6b7e492b90fecb199f...
doneWhat am I doing differently?
ktsakalozos
commented
4 years ago Might be worth taking a look at https://github.com/ubuntu/microk8s/issues/196#issuecomment-667160883
decewei
commented
4 years ago Might be worth taking a look at #196 (comment)
Let me perform this on a clean install. I'll report the result back. The thing is we have multiple machines having the same problem. There must be something missing here.
In the meantime, for people who steps on this issue, and need quick solution. For RHEL, uses the following command to install the CA you used to sign your registry certficate on your microk8s machine:
echo "Installing registry CA... "
mv mk8s-registry.ca /etc/pki/ca-trust/source/anchors
# Check if it is moved to the folder:
ls /etc/pki/ca-trust/source/anchors
# Update host machien certificate
sudo update-ca-trust extract
# Check if your certificate is in /etc/ssl/certs/ca-bundle.crt
cat /etc/ssl/certs/ca-bundle.crt | grep SOME_SUB_STRING_OF_YOUR_CA@joedborg Thank you for your efforts. Is this pushed to snap or not yet? I am guessing it is included in those snap channel: 1.16/stable: v1.16.14 2020-08-23 (1606) and latest/candidate: v1.18.8 2020-08-19 (1626)
ktsakalozos
commented
4 years ago Hi @muhammadnasr
The containerd upgrade to v1.3.7 should be on latest/candidate and will be on the v1.19 release (1.19/* channels) but will not be backported to 1.16 channels (so it is not on 1.16/stable).
Thanks for the awesome project.
I am trying to setup gitlab autodevops over microk8s, autobuild seem stuck at build due to some HTTP/HTTPs problem
Logging to GitLab Container Registry with CI credentials... Error response from daemon: Get https://172.22.0.92:5050/v2/: http: server gave HTTP response to HTTPS client
I tried to edit /var/snap/microk8s/current/args/containerd-template.toml and restart microk8s but seems it does not effect