LetsEncrypt Issues Due to Jetstream2 internal, non-routed host

[julienchastang]

We were recently unable to obtain LetEncrypt certs following instructions here.

We kept obtaining connect: no route to host errors when examining kubectl describe challenge <challenge-xyz>

This may be due to a limitation on JS2 concerning non-routed hosts.

@robertej09 has a solution in place already involving k8s tolerations that will be forthcoming soon.

[zonca]

copying here the text from the Jetstream website, in case they shuffle it around:

I can’t ping or reach a public/floating IP from an internal, non-routed host This as a change in the network between Jetstream 1 and 2. You cannot presently ping a floating IP address from internal, non-routable IPs. You can only to talk from non-routed nodes to bastion hosts and floating IPs only work from other routable hosts. To communicate from an internal only Jetstream2 instance to an instance on Jetstream2 with a floating IP, you’ll need to use the internal networking address or name.

[zonca]

ok, thanks @robertej09 @julienchastang, I merged the PR and updated the tutorial.

[zonca]

Actually I had an issue in my setup and had to do the test again.

Doing the test again, I do not see this problem.

I have the cert-manager pods running in a node and not on master:

k get pods -n cert-manager -o wide
NAME                                       READY   STATUS    RESTARTS   AGE     IP             NODE                 NOMINATED NODE   READINESS GATES
cert-manager-86b4798576-cxkz4              1/1     Running   0          3m50s   10.233.65.30   kubejs2-k8s-node-1   <none>           <none>
cert-manager-cainjector-85656d565c-vhvh8   1/1     Running   0          3m50s   10.233.65.29   kubejs2-k8s-node-1   <none>           <none>
cert-manager-webhook-8665f985d6-zv98h      1/1     Running   0          3m50s   10.233.65.31   kubejs2-k8s-node-1   <none>           <none>

Status of the certificate:

Status:
  Conditions:
    Last Transition Time:  2022-08-10T21:42:19Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2022-11-08T20:42:18Z
  Not Before:              2022-08-10T20:42:19Z
  Renewal Time:            2022-10-09T20:42:18Z
  Revision:                1
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Issuing    5m18s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  5m18s  cert-manager  Stored new private key in temporary Secret resource "certmanager-tls-jupyterhub-7ct4x"
  Normal  Requested  5m18s  cert-manager  Created new CertificateRequest resource "certmanager-tls-jupyterhub-9j684"
  Normal  Issuing    4m15s  cert-manager  The certificate has been successfully issued

[zonca]

I read again @julienchastang 's email address, is it possible this is due to all my VMs having a floating IP address?

[zonca]

If this is the case, better to leave this patch in the tutorial, it is anyway harmless and quick and removes a pain point for users that use no floating IPs in their instances.

[ana-v-espinoza]

Andrea, yes I think that would be why you don't see the issue. I suppose it would be worth specifying that this patch is only necessary if your worker nodes do not have a floating IP on JetStream2. As per the JetStream2 FAQ: "You cannot presently ping a floating IP address from internal, non-routable IPs. You can only to talk from non-routed nodes to bastion hosts and floating IPs only work from other routable hosts."

In our workflow we do not assign floating IPs to our worker nodes, and thus this patch is necessary.

[zonca]

Ok very good! Thanks