Open deeduda opened 4 years ago
@deeduda you did not enable https: enabled: false :)))
Sorry I disable it again to get jupyterhub to work but it was enabled when I got these errors. Besides traefik won't attempt to use the certificate if https is disabled.
eladmin@AZLAPNSGEA01:~$ sudo tljh-config show
users:
admin:
- eladmin
https:
tls:
cert: /root/ssl/jupyter_platform_manulife_io_cert.cer
key: /root/ssl/jupyter.platform.manulife.io.key
enabled: true
auth:
FirstUseAuthenticator:
create_users: true
@deeduda
Are you sure the cert is readable by traefik and the format is corrected ? Putting SSL certs to the home directory of root is a little odd btw. What does:
openssl x509 -in /root/ssl/jupyter_platform_manulife_io_cert.cer -text -noout
gives you?
$ sudo openssl x509 -in /root/ssl/jupyter_platform_manulife_io_cert.cer -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
9b:89:24:8e:45:8d:98:38:3a:92:18:31:3a:ba:2e:59
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Organization Validation Secure Server CA
Validity
Not Before: Oct 29 00:00:00 2019 GMT
Not After : Oct 28 23:59:59 2021 GMT
Subject: C = CA, postalCode = M4W 1E5, ST = Ontario, L = Toronto, street = 200 Bloor Street East, O = Manulife Financial, OU = Global Infrastructure, OU = Enterprise SSL Pro, CN = jupyter.platform.manulife.io
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:96:93:f5:70:f3:9f:05:c1:86:b8:75:30:68:c1:
30:46:b2:f0:6e:a9:90:f2:2d:c6:f7:99:88:7e:ea:
f5:ee:73:0a:19:af:d4:70:8f:fa:4e:5d:a4:23:b8:
76:78:8a:2f:22:48:60:0b:0d:f2:d9:ca:4d:be:2d:
95:6e:ef:97:52:d4:56:cf:83:51:97:9f:df:b6:6c:
2e:fa:5b:87:3e:14:31:7d:59:3d:37:f0:e0:3a:37:
cd:17:95:cb:1d:67:24:62:26:9c:5b:bd:81:f2:5c:
84:5f:d4:7d:b8:fb:65:57:96:1c:b8:97:cd:a0:ba:
aa:70:99:a8:a3:5d:fc:a4:bf:20:15:83:90:6b:72:
3a:8b:99:ef:18:a2:12:71:81:a5:e2:fa:f1:65:20:
e6:7e:51:c2:0a:e9:ab:f1:7f:08:3f:86:3f:13:e1:
e8:6f:58:c4:2c:c2:8e:a7:be:56:1a:7f:84:97:85:
bf:31:6d:db:fc:dc:e5:08:2b:20:0c:71:b8:b3:ff:
cb:af:0b:e0:69:2c:53:0c:c6:5d:11:5c:88:dc:85:
ec:bf:e7:ef:d5:2b:ae:3d:74:f4:4c:0a:cc:7d:5a:
f6:42:a0:d4:0f:0b:8a:40:ed:3e:15:61:cd:b8:1f:
bf:23:85:99:43:55:4d:05:a2:88:d5:a3:57:f2:f6:
26:c1
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:17:D9:D6:25:27:67:F9:31:C2:49:43:D9:30:36:44:8C:6C:A9:4F:EB
X509v3 Subject Key Identifier:
20:5C:09:9B:EC:1B:26:22:8B:2D:DE:EE:FE:7B:36:38:88:E3:E6:D6
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Certificate Policies:
Policy: 1.3.6.1.4.1.6449.1.2.1.3.4
CPS: https://sectigo.com/CPS
Policy: 2.23.140.1.2.2
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.sectigo.com/SectigoRSAOrganizationValidationSecureServerCA.crl
Authority Information Access:
CA Issuers - URI:http://crt.sectigo.com/SectigoRSAOrganizationValidationSecureServerCA.crt
OCSP - URI:http://ocsp.sectigo.com
X509v3 Subject Alternative Name:
DNS:jupyter.platform.manulife.io, DNS:www.jupyter.platform.manulife.io
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 7D:3E:F2:F8:8F:FF:88:55:68:24:C2:C0:CA:9E:52:89:
79:2B:C5:0E:78:09:7F:2E:6A:97:68:99:7E:22:F0:D7
Timestamp : Oct 29 00:36:25.413 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:C6:5C:70:31:A3:8E:AF:05:DE:38:85:
F0:B8:C4:AE:1D:8B:C8:06:05:F3:E4:A6:0A:30:75:B3:
78:94:61:51:42:02:20:68:C6:93:D8:1F:42:A2:F1:D2:
8F:09:3B:A6:48:9E:6E:61:E5:35:94:75:98:58:7F:EE:
AA:AA:C4:E4:89:CE:13
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 44:94:65:2E:B0:EE:CE:AF:C4:40:07:D8:A8:FE:28:C0:
DA:E6:82:BE:D8:CB:31:B5:3F:D3:33:96:B5:B6:81:A8
Timestamp : Oct 29 00:36:25.391 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:A7:D2:3B:38:14:40:5F:B3:02:05:3F:
76:82:31:61:57:A4:6C:22:EA:5E:8C:20:CB:94:5F:38:
32:48:56:5F:AB:02:21:00:D3:CE:78:34:55:DD:EF:39:
B4:E0:36:47:26:E0:11:A1:BE:43:C8:DA:3B:1E:89:C2:
31:45:4A:07:C3:1B:97:48
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 55:81:D4:C2:16:90:36:01:4A:EA:0B:9B:57:3C:53:F0:
C0:E4:38:78:70:25:08:17:2F:A3:AA:1D:07:13:D3:0C
Timestamp : Oct 29 00:36:25.352 2019 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:44:02:20:44:A0:57:AA:C8:4E:56:2D:DE:12:00:8F:
15:2C:73:04:AD:B3:31:75:BA:77:4B:94:CF:80:EF:68:
E4:A8:64:0E:02:20:76:7A:26:A3:7A:FE:E0:D4:15:4F:
C4:10:AA:B6:58:E8:D4:AE:BB:54:21:20:16:5D:05:69:
14:31:05:DA:4F:06
Signature Algorithm: sha256WithRSAEncryption
64:08:b8:d0:ef:60:13:e9:cf:87:b6:30:d9:dc:0c:1a:cd:2b:
cb:02:c5:15:10:1a:04:60:5f:0e:67:3a:4a:08:60:da:c9:fd:
b2:3e:19:92:30:2c:0b:87:74:fc:c7:1e:c6:99:6b:9d:1d:54:
26:9c:b4:5a:7f:8f:d0:1d:2c:f9:8d:ac:d0:1e:0b:97:f4:62:
04:ad:1b:b1:74:6d:d6:1f:1d:78:ab:7a:0d:be:46:30:b0:6e:
66:f9:d4:06:c3:22:9b:cb:f0:09:17:3d:5f:81:6c:2b:4f:5c:
d0:27:b3:db:de:1e:72:75:ac:63:63:8f:1d:12:ef:fb:e3:94:
52:e8:4b:57:e5:e5:43:27:0c:b2:a5:89:d6:4a:3f:83:2f:d0:
ab:f3:c1:d6:9c:25:e3:a6:6d:b5:26:34:aa:39:13:9b:83:59:
2e:a2:8e:b6:bc:20:d7:a6:89:6b:d8:93:70:0f:d3:dc:73:dd:
44:9e:0d:ae:c5:5d:7f:36:aa:35:7c:cb:2e:af:92:d3:ae:6f:
92:8c:2f:46:d2:ab:e7:23:0b:9c:fc:88:3a:2d:a8:51:ef:72:
1c:19:81:7f:74:90:43:6d:aa:57:3e:6e:e6:4b:12:50:d8:4e:
6a:97:66:9e:81:20:78:9e:49:2a:f2:9a:66:ae:1b:ba:da:f9:
c5:4a:6a:84
I tried to move it to /tmp and other directory and the same thing happened.
The odd thing is I can trace the go lang to this particular function and it seems to be looking for a default certificate in the store.
look for ""failed to load X509 key pair"
Hey @deeduda! Sorry for the late reply.
Try putting the cert and key in /opt/tljh/state
. This is the working directory of traefik and should have access to it.
Hope it helps.
I ran into this as well. Putting the cert & key into /opt/tljh/state
worked for me. Tnx
Update by Erik 2021-10-25
I think the resolution to this issue was addressed by https://github.com/jupyterhub/the-littlest-jupyterhub/issues/462#issuecomment-573086796 suggesting a location where traefik could access the cert/key configured under
https.tls.cert
andhttps.tls.key
.Perhaps we could run a check to verify our traefik user will have access to the configured location, and if not, emit a error suggesting putting them in the /opt/tljh/state location?
Original issue
I have not been able to run TLJH in https mode due to traefik spilling out these errors.
The config is very simple and I am able to use it in http mode:
Both the cert and key are signed by a commercial provider and checked by openssl verify
I believed tljh is running 1.17 traefik.