IssuerNotFound

[julienchastang]

kubectl get certificaterequest  --all-namespaces

yields

...
Events:
  Type    Reason           Age                From          Message
  ----    ------           ----               ----          -------
  Normal  cert-manager.io  59m                cert-manager  Certificate request has been approved by cert-manager.io
  Normal  IssuerNotFound   59m (x5 over 59m)  cert-manager  Referenced "Issuer" not found: issuer.cert-manager.io "letsencrypt" not found

[zonca]

Try to check if the issuer is there and if it is in the right name space

[zonca]

Also, I think it would be better to use a cluster issuer

[julienchastang]

Thank you for the super-quick response

$  kubectl get issuer -n jhub
No resources found in jhub namespace.
$  kubectl get issuer --all-namespaces
No resources found

[zonca]

It is possible that I updated one of the tutorials to use a cluster issuer instead of a standard issue

[zonca]

So try to check if you have a cluster insurance running and if that is the case you can configure ingress to use that instead of the standard is sure

[julienchastang]

Thank you! I had missed this line in your instructions:

    cert-manager.io/cluster-issuer: "letsencrypt"

(not cert-manager.io/issuer: "letsencrypt" as in the past)

[zonca]

Sorry speech to text interprets "issuer" differently every time

[julienchastang]

Follow up question: Do you have any idea how to get around: Warning Failed 35m cert-manager The certificate request has failed to complete and will be retried: Failed to wait for order resource "certmanager-tls-jupyterhub-wbk9c-3112338203" to become ready: order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:rejectedIdentifier: NewOrder request did not include a SAN short enough to fit in CN

The JS2 auto-DNS names are now very long, e.g., scsu2022fall-k8s-master-1.tg-ees220002.projects.jetstream-cloud.org

[zonca]

I haven't tried those yet. Do you have the option of testing with a normal domain?

[julienchastang]

Unfortunately, this may not be possible: https://community.letsencrypt.org/t/the-server-will-not-issue-certificates-for-the-identifier-neworder-request-did-not-include-a-san-short-enough-to-fit-in-cn/156353

Do you have the option of testing with a normal domain?

Not easily on JS2 as I understand the situation.

[zonca]

I work around is mentioned here but I'm not sure how to implement it into kubernetes

https://support.cpanel.net/hc/en-us/articles/4405807056023-Let-s-Encrypt-NewOrder-request-did-not-include-a-SAN-short-enough-to-fit-in-CN-

[julienchastang]

Actually, if the limit really is 64, choosing shorter domain names should be possible. This is 67:

scsu2022fall-k8s-master-1.tg-ees220002.projects.jetstream-cloud.org

[julienchastang]

The "mandatory" part is 53 characters: k8s-master-1.tg-ees220002.projects.jetstream-cloud.org

[ana-v-espinoza]

The "mandatory" part is 53 characters: k8s-master-1.tg-ees220002.projects.jetstream-cloud.org

I think we can save ourselves some characters (12) here by removing "k8s-master-n" from this line when building the cluster: https://github.com/zonca/jetstream_kubespray/blob/b5513e089145e3cfaecd7665259610c67e74e8f7/contrib/terraform/openstack/modules/compute/main.tf#L212

@zonca Any idea whether this would break anything? We generally deploy our JHubs with only a single k8s master node.

[zonca]

I think there is no problem. However I recommend just to remove k8s-master- And leave the number alone. We save 11 characters and we can support multiple masternodes.

If you implement this, please make a pull request on the repository

[julienchastang]

Also, I am trying with a smaller domain name. Please stand by while I figure that out.

[julienchastang]

OK, it works.

[julienchastang]

In summary and for posterity:

1. cluster-issuer not issuer
2. Ensure DNS names are < 64 characters.

[zonca]

In case you have modified the kube spray recipe. Can you provide the pull request with that fix?

[zonca]

@julienchastang,

In case you have modified the kube spray recipe. Can you please send me the modification you did?

On Thu, Jul 21, 2022 at 3:52 PM Julien Chastang @.***> wrote:

In summary and for posterity:

1. cluster-issuer not issuer
2. Ensure DNS names are < 64 characters.

— Reply to this email directly, view it on GitHub https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/issues/51#issuecomment-1192008027, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC5Q4UGVLUV2YGAZTTYN5DVVHIBPANCNFSM54IV5T6Q . You are receiving this because you were mentioned.Message ID: @.*** com>

[julienchastang]

Beyond what I mentioned earlier. There are a couple of additional tweaks, mainly to get letsencrypt working. I am setting up a hub today so I will detail these shortly.

[julienchastang]

To have automatic DNS for Jetstream2, you'll want to :

openstack network set --dns-domain <your project allocation id>.projects.jetstream-cloud.org.  <network id you just created for this jhub>-network
openstack server remove floating ip <hub id>-k8s-master-1 <ip of master node>
openstack server add floating ip <hub id>-k8s-master-1 <ip of master node>

Not sure if the last two commands are strictly necessary, but it does not hurt.

After this, make sure the host is accessible from some machine not on JS2:

dig <hub id>-k8s-master-1.<project id>.projects.jetstream-cloud.org

This will ensure letsencrypt has an avenue to work.

ping @robertej09, @jlf599

[zonca]

ok, got everything working, I will be updating the tutorial at: https://zonca.dev/2022/03/kubernetes-jetstream2-kubespray.html thanks for your contributions