Open phalox opened 2 years ago
We have an nginx-ingress in use, but we want it to handle ssl for other services, so we cannot just bypass ssl as someone recommended here: https://github.com/skupperproject/skupper/issues/633
Enabling passthrough on the controller does not mean that all other uses will then do passthrough. Passthrough needs to be explicitly requested for a given ingress through an annotation.
By default, when you create the token it creates a token of type 'claim'. This is exchanged at the time of linking for an actual TLS certificate. However by default the claim expires after 15 mins. The 'no such claim' error suggests that the record of the claim may have been deleted by the time it was used. Is that possible? if you create a new token and immediately try to use it, does that work?
Great, that 15min timeout was indeed the issue. Can I somehow help improve the docs as this was not clear to me. Also, I assume that the token can only be used once?
So, I now see the pods exposed in the 2 k8s clusters. On the cluster that generated the token (and allows for incoming connections - the AMS3 cluster), I see:
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroachdb-ams3-0 1/1 Running 1 (36h ago) 2d19h
cockroachdb-ams3-1 1/1 Running 1 (6h5m ago) 2d19h
cockroachdb-ams3-2 1/1 Running 1 (10h ago) 2d19h
cockroachdb-ny3-0 1/1 Running 0 4m17s
cockroachdb-ny3-1 1/1 Running 0 4m14s
cockroachdb-ny3-2 1/1 Running 0 4m8s
skupper-router-7c8b58d8fc-8pjzv 2/2 Running 0 6m8s
skupper-service-controller-64bbf9bf77-8r87k 1/1 Running 0 6m8s
skupper status
Skupper is enabled for namespace "cockroach" in interior mode. It is connected to 1 other site. It has 1 exposed service.
The site console url is: https://.......:31602
The credentials for internal console-auth mode are held in secret: 'skupper-console-users'
On the cluster connecting to AMS (called NY3) I see:
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroachdb-ny3-0 1/1 Running 0 23m
cockroachdb-ny3-1 1/1 Running 0 24m
cockroachdb-ny3-2 1/1 Running 0 24m
cockroachdb-proxy-0 1/1 Running 0 103m
cockroachdb-proxy-1 1/1 Running 0 103m
cockroachdb-proxy-2 1/1 Running 0 103m
skupper-router-7487cc7b65-8xpwj 2/2 Running 0 43h
skupper-service-controller-689cd95f8b-6f4l5 1/1 Running 0 43h
skupper status
Skupper is enabled for namespace "cockroach" in interior mode. It is connected to 1 other site. It has 1 exposed service.
But it seems I still have connection issues. For one, the typical hostnames that would be created by a statefulset, don't exist for the proxy pods...
e.g. ping cockroachdb-ny3-1.cockroachdb from the NY3 cluster works ping cockroachdb-ny3-1.cockroachdb from the AMS3 cluster doesn't work (the hostname doesn't resolve)
Any ideas what's up? I'm not sure it will fix all my issues to group 2 clusters into a cockroachdb cluster, but at least I'll have a hostname that's usable. I'm currently following this guide: https://www.wrong.dev/post/cockroachdb-multi-cloud-using-skupper
And a side question: Why did the NY3 pods get renamed to -proxy, while the AMS3 pods kept the name from the other cluster?
Thanks for your help! I was happy to see the synchronization taking effect. At one point I might still experment with nginx ingress as well.
Great, that 15min timeout was indeed the issue. Can I somehow help improve the docs as this was not clear to me. Also, I assume that the token can only be used once?
By default, yes, the token can only be used once and within 15 mins. You can alter that through options on the skupper token create
command however.
Improvements to the docs are always very much welcome!
the typical hostnames that would be created by a statefulset, don't exist for the proxy pods...
e.g. ping cockroachdb-ny3-1.cockroachdb from the NY3 cluster works ping cockroachdb-ny3-1.cockroachdb from the AMS3 cluster doesn't work (the hostname doesn't resolve)
Any ideas what's up?
What do you see for kubectl get svc
?
Fyi we also have an example for this here: https://github.com/grs/skupper-example-cockroachdb
AMS3 (main skupper):
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cockroachdb ClusterIP None <none> 26257/TCP,8080/TCP 2d20h
cockroachdb-public ClusterIP 10.245.241.196 <none> 26257/TCP,8080/TCP 2d20h
skupper NodePort 10.245.115.108 <none> 8080:31602/TCP,8081:30696/TCP 46h
skupper-router NodePort 10.245.151.63 <none> 55671:32183/TCP,45671:32538/TCP 46h
skupper-router-local ClusterIP 10.245.123.22 <none> 5671/TCP 46h
NY3:
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cockroachdb ClusterIP None <none> 26257/TCP,8080/TCP 2d20h
cockroachdb-public ClusterIP 10.245.133.161 <none> 26257/TCP,8080/TCP 2d20h
skupper ClusterIP 10.245.97.248 <none> 8080/TCP,8081/TCP 44h
skupper-router ClusterIP 10.245.16.228 <none> 55671/TCP,45671/TCP 44h
skupper-router-local ClusterIP 10.245.185.131 <none> 5671/TCP 44h
I'll check out the example. I didn't set up skupper in the 2nd direction. I assume that wasn't a requirement?
You only need to link skupper one way. I suspect the issue may be a collision over the headless service? If you prefix/postfix the headless service to match the statefulset it is used with then that would avoid that.
Great, that seems to indeed have fixed the dns resolution error. I can now ping from NY3 to AMS3 pods. Why do the AMS3 pods receive a name with proxy inside?
For some reason, cockroach db is still not connecting to the proxied nodes.
The pods (on NY3) start with this:
cockroach start --logtostderr --certs-dir /cockroach/cockroach-certs --advertise-host $(hostname -f) --http-addr 0.0.0.0 --join cockroachdb-ny3-0.cockroachdb-internal-ny3,cockroachdb-ny3-1.cockroachdb-internal-ny3,cockroachdb-ny3-2.cockroachdb-internal-ny3,cockroachdb-ams3-0.cockroachdb-internal-ams3,cockroachdb-ams3-1.cockroachdb-internal-ams3,cockroachdb-ams3-2.cockroachdb-internal-ams3 --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB
But I don't see any mention of a connection to ams3 in there. I'm also using cockroach in secure mode, but all ssl certs should have been shared and configured for the right host names. I'll go debug some more...
The -proxy pods are server side proxies for requests from the other cluster. Did you expose the -ams3 statefulset?
Ohhhh, I think I should not initialize the cockroach clusters twice (before joining them). I've removed the 2nd initialized one, started everything up again and now have 6 cockroach nodes talking to one another!
And indeed, I should have exposed the other statefulset as well. I've done so now.
ams:
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroachdb-ams3-0 1/1 Running 0 37m
cockroachdb-ams3-1 1/1 Running 0 37m
cockroachdb-ams3-2 1/1 Running 0 37m
cockroachdb-internal-ams3-proxy-0 1/1 Running 0 36m
cockroachdb-internal-ams3-proxy-1 1/1 Running 0 36m
cockroachdb-internal-ams3-proxy-2 1/1 Running 0 36m
cockroachdb-ny3-0 1/1 Running 0 5s
cockroachdb-ny3-1 1/1 Running 0 3s
cockroachdb-ny3-2 0/1 ContainerCreating 0 1s
skupper-router-7c8b58d8fc-8pjzv 2/2 Running 0 3h45m
skupper-service-controller-64bbf9bf77-8r87k 1/1 Running 0 3h45m
ny:
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroachdb-ams3-0 1/1 Running 0 36m
cockroachdb-ams3-1 1/1 Running 0 36m
cockroachdb-ams3-2 1/1 Running 0 36m
cockroachdb-internal-ny3-proxy-0 1/1 Running 0 16s
cockroachdb-internal-ny3-proxy-1 1/1 Running 0 14s
cockroachdb-internal-ny3-proxy-2 1/1 Running 0 13s
cockroachdb-ny3-0 1/1 Running 0 6m3s
cockroachdb-ny3-1 1/1 Running 0 6m3s
cockroachdb-ny3-2 1/1 Running 0 6m3s
skupper-router-7487cc7b65-8xpwj 2/2 Running 0 47h
skupper-service-controller-689cd95f8b-6f4l5 1/1 Running 0 47h
Thanks a bunch for your help! Now I should have a look if I should be running cockroach in secure mode, or if I can make it work non-secure as well. From their docs I understand that operation in non-secure doesn't really work...
Hi Skupper team,
As we're running on digitalocean, the regular skupper init doesn't work (loadbalancers need to be set up separately). But in any case, we don't want to use the digitalocean loadbalancer. So we've tried a couple of other things:
On the main cluster: skupper init --ingress nodeport --ingress-host PUBLIC-DNS-ENTRY (we set up DNS entry with external-dns in cloudflare) On the 2nd cluster: skupper init --ingress none
Doing a curl returns this:
Which I believe means communication can happen? When I generate the token, I do see that another port (31696, going to 8081 on skupper service). That one doesn't seem to take HTTP requests.
I then set up the link from the 2nd cluster
But the link does not seem to get created:
What are we missing here? How can I debug this? I can curl from the 2nd cluster to the main cluster on the skupper port (the 8080 one).