Closed abhillman closed 3 years ago
I meet the same error
root@ubuntu:~# sudo tljh-config reload
--
active
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
I get the same:
# tljh-config reload hub
active
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
/opt/tljh/hub/lib/python3.6/site-packages/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
Hub reload with new configuration complete
# lsof -i -P -n | grep LISTEN
rpcbind 480 root 8u IPv4 16447 0t0 TCP *:111 (LISTEN)
rpcbind 480 root 11u IPv6 16450 0t0 TCP *:111 (LISTEN)
systemd-r 755 systemd-resolve 13u IPv4 18668 0t0 TCP 127.0.0.53:53 (LISTEN)
sshd 1101 root 3u IPv4 21228 0t0 TCP *:22 (LISTEN)
sshd 1101 root 4u IPv6 21230 0t0 TCP *:22 (LISTEN)
traefik 5291 root 3u IPv6 60372 0t0 TCP *:443 (LISTEN)
traefik 5291 root 5u IPv4 60374 0t0 TCP 127.0.0.1:8099 (LISTEN)
traefik 5291 root 6u IPv6 60375 0t0 TCP *:80 (LISTEN)
python3 6140 root 8u IPv4 63896 0t0 TCP 127.0.0.1:15001 (LISTEN)
$ sudo journalctl -u jupyterhub -f
-- Logs begin at Tue 2020-07-14 14:44:18 UTC. --
Jul 14 19:10:04 jupyter python3[6140]: [I 2020-07-14 19:10:04.437 JupyterHub proxy:320] Checking routes
Jul 14 19:10:04 jupyter python3[6140]: [I 2020-07-14 19:10:04.671 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 23.42ms
Jul 14 19:11:04 jupyter python3[6140]: [I 2020-07-14 19:11:04.671 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 22.39ms
Jul 14 19:12:04 jupyter python3[6140]: [I 2020-07-14 19:12:04.674 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 23.90ms
Jul 14 19:13:04 jupyter python3[6140]: [I 2020-07-14 19:13:04.679 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 30.59ms
Jul 14 19:14:04 jupyter python3[6140]: [I 2020-07-14 19:14:04.671 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 21.32ms
Jul 14 19:15:04 jupyter python3[6140]: [I 2020-07-14 19:15:04.409 JupyterHub proxy:320] Checking routes
Jul 14 19:15:04 jupyter python3[6140]: [I 2020-07-14 19:15:04.671 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 23.01ms
Jul 14 19:16:04 jupyter python3[6140]: [I 2020-07-14 19:16:04.671 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 22.31ms
Jul 14 19:17:04 jupyter python3[6140]: [I 2020-07-14 19:17:04.679 JupyterHub log:174] 200 GET /hub/api/users (cull-idle@127.0.0.1) 28.87ms
$ sudo journalctl -u traefik
-- Logs begin at Tue 2020-07-14 14:44:18 UTC, end at Tue 2020-07-14 19:19:01 UTC. --
Jul 14 15:28:55 jupyter systemd[1]: Started traefik.service.
Jul 14 15:28:55 jupyter systemd[1]: Stopping traefik.service...
Jul 14 15:28:55 jupyter systemd[1]: Stopped traefik.service.
Jul 14 15:28:55 jupyter systemd[1]: Started traefik.service.
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Traefik version v1.7.18 built on 2019-09-26_01:56:30PM"
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v1.7/basics/#collected-data\n"
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc000915b60} with
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS:<nil> Redirect:<nil> Auth:0xc0007fc6c0 WhitelistSourceRange:[] WhiteList:0xc000914a00 Compress:false ProxyProtocol:<nil> Forward
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Starting server on :80"
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Starting server on 127.0.0.1:8099"
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Starting provider *file.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Directory\":\"rules
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 15:28:55 jupyter traefik[4675]: time="2020-07-14T15:28:55Z" level=info msg="Server configuration reloaded on :80"
Jul 14 15:28:57 jupyter traefik[4675]: time="2020-07-14T15:28:57Z" level=info msg="Skipping same configuration for provider file"
Jul 14 15:28:57 jupyter traefik[4675]: time="2020-07-14T15:28:57Z" level=info msg="Server configuration reloaded on :80"
Jul 14 15:28:57 jupyter traefik[4675]: time="2020-07-14T15:28:57Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 15:28:57 jupyter traefik[4675]: time="2020-07-14T15:28:57Z" level=info msg="Skipping same configuration for provider file"
Jul 14 15:28:57 jupyter traefik[4675]: time="2020-07-14T15:28:57Z" level=info msg="Skipping same configuration for provider file"
Jul 14 15:38:56 jupyter traefik[4675]: time="2020-07-14T15:38:56Z" level=warning msg="A new release has been found: 2.2.5. Please consider updating."
Jul 14 18:05:42 jupyter traefik[4675]: time="2020-07-14T18:05:42Z" level=info msg="Skipping same configuration for provider file"
Jul 14 18:05:42 jupyter traefik[4675]: time="2020-07-14T18:05:42Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:05:42 jupyter traefik[4675]: time="2020-07-14T18:05:42Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:05:44 jupyter traefik[4675]: time="2020-07-14T18:05:44Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:05:44 jupyter traefik[4675]: time="2020-07-14T18:05:44Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:11:14 jupyter traefik[4675]: time="2020-07-14T18:11:14Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:11:14 jupyter traefik[4675]: time="2020-07-14T18:11:14Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:11:14 jupyter systemd[1]: Stopping traefik.service...
Jul 14 18:11:14 jupyter traefik[4675]: time="2020-07-14T18:11:14Z" level=info msg="I have to go..."
Jul 14 18:11:14 jupyter traefik[4675]: time="2020-07-14T18:11:14Z" level=info msg="Stopping server gracefully"
Jul 14 18:11:14 jupyter traefik[4675]: time="2020-07-14T18:11:14Z" level=info msg="Server stopped"
Jul 14 18:11:14 jupyter traefik[4675]: time="2020-07-14T18:11:14Z" level=info msg="Shutting down"
Jul 14 18:11:14 jupyter systemd[1]: Stopped traefik.service.
Jul 14 18:11:14 jupyter systemd[1]: Started traefik.service.
Jul 14 18:11:14 jupyter traefik[5291]: time="2020-07-14T18:11:14Z" level=info msg="Using TOML configuration file /opt/tljh/state/traefik.toml"
Jul 14 18:11:14 jupyter traefik[5291]: time="2020-07-14T18:11:14Z" level=info msg="Traefik version v1.7.18 built on 2019-09-26_01:56:30PM"
Jul 14 18:11:14 jupyter traefik[5291]: time="2020-07-14T18:11:14Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v1.7/basics/#collected-data\n"
Jul 14 18:11:14 jupyter traefik[5291]: time="2020-07-14T18:11:14Z" level=info msg="Preparing server https &{Address::443 TLS:0xc0005606c0 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0002070
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Preparing server auth_api &{Address:127.0.0.1:8099 TLS:<nil> Redirect:<nil> Auth:0xc0005b8600 WhitelistSourceRange:[] WhiteList:0xc000206020 Compress:false ProxyProtocol:<nil> Forward
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:0xc00008b540 Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0002070e0
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Starting server on :443"
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Starting server on 127.0.0.1:8099"
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Starting server on :80"
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Starting provider *file.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Directory\":\"rules
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Starting provider *acme.Provider {\"Email\":\"marcin.tomasz.gil@gmail.com\",\"ACMELogging\":false,\"CAServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"Storage\":\"acme.js
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="Testing certificate renew..."
Jul 14 18:11:15 jupyter traefik[5291]: time="2020-07-14T18:11:15Z" level=info msg="The key type is empty. Use default key type 4096."
Jul 14 18:11:16 jupyter traefik[5291]: time="2020-07-14T18:11:16Z" level=info msg="Server configuration reloaded on :443"
Jul 14 18:11:16 jupyter traefik[5291]: time="2020-07-14T18:11:16Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:11:16 jupyter traefik[5291]: time="2020-07-14T18:11:16Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:11:16 jupyter traefik[5291]: time="2020-07-14T18:11:16Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:11:16 jupyter traefik[5291]: time="2020-07-14T18:11:16Z" level=info msg="Server configuration reloaded on :443"
Jul 14 18:11:16 jupyter traefik[5291]: time="2020-07-14T18:11:16Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:11:18 jupyter traefik[5291]: time="2020-07-14T18:11:18Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
ul 14 18:11:18 jupyter traefik[5291]: time="2020-07-14T18:11:18Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:11:18 jupyter traefik[5291]: time="2020-07-14T18:11:18Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:11:18 jupyter traefik[5291]: time="2020-07-14T18:11:18Z" level=info msg="Server configuration reloaded on :443"
Jul 14 18:11:21 jupyter traefik[5291]: time="2020-07-14T18:11:21Z" level=info msg=Register...
Jul 14 18:11:26 jupyter traefik[5291]: time="2020-07-14T18:11:26Z" level=error msg="Unable to obtain ACME certificate for domains \"vaxar.online\" : unable to generate a certificate for the domains [vaxar.online]: acme: Error -> One or more domains had a problem:\n[
Jul 14 18:16:16 jupyter traefik[5291]: time="2020-07-14T18:16:16Z" level=info msg="Skipping same configuration for provider file"
Jul 14 18:16:16 jupyter traefik[5291]: time="2020-07-14T18:16:16Z" level=info msg="Skipping same configuration for provider file"
Jul 14 18:16:16 jupyter traefik[5291]: time="2020-07-14T18:16:16Z" level=info msg="Server configuration reloaded on :443"
Jul 14 18:16:16 jupyter traefik[5291]: time="2020-07-14T18:16:16Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:16:16 jupyter traefik[5291]: time="2020-07-14T18:16:16Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:16:18 jupyter traefik[5291]: time="2020-07-14T18:16:18Z" level=info msg="Server configuration reloaded on :80"
Jul 14 18:16:18 jupyter traefik[5291]: time="2020-07-14T18:16:18Z" level=info msg="Server configuration reloaded on :443"
Jul 14 18:16:18 jupyter traefik[5291]: time="2020-07-14T18:16:18Z" level=info msg="Server configuration reloaded on 127.0.0.1:8099"
Jul 14 18:21:15 jupyter traefik[5291]: time="2020-07-14T18:21:15Z" level=warning msg="A new release has been found: 2.2.5. Please consider updating."
Jul 14 18:55:03 jupyter traefik[5291]: {"BackendAddr":"127.0.0.1:15001","BackendName":"backend__2F","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:15001","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"12
Jul 14 18:55:04 jupyter traefik[5291]: {"BackendAddr":"127.0.0.1:15001","BackendName":"backend__2F","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:15001","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"12
Same here
sudo tljh-config reload
active
/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
Hub reload with new configuration complete
I think this is resolved by #357!
This is still happening right now for me.
I have just installed TLJH from https://tljh.jupyter.org/bootstrap.py changed some bits of configurations and then I can't apply them since sudo tljh-config reload
gets stuck into this loop. I am doing exactly the same fresh install which I was doing a week ago, when it worked just fine (but then, various pip and conda installs might have changed since)
Let me know if I should open a separate issue with more details instead of this "footnote" on closed issue.
This issue persists with an existing SSL as well. Constant InsecureRequestWarning loop.
In case others land here and feel stuck, there is still something like a race condition or other issue that cause a loop of these warnings, which eventually clears. That is "normal" (should not, but we can live with it).
In some other circumstances, the finite loop becomes infinite. It is very annoying to not see the reason, but double checking everything should allow one to find the solution. In my case, it was an incorrect value set in the field auth.NativeAuthenticator.secret_key
This issue still exists. I don't get stuck in a loop, but I keep getting warnings when reloading the hub (not the proxy):
$ sudo tljh-config reload hub
active
/opt/tljh/hub/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
/opt/tljh/hub/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
Hub reload with new configuration complete
@consideRatio : #357 does not fix this, as it seems to only disable these warnings for upgrades specifically. I assume a similar fix could be done for tljh-config reload. Or maybe some sort of hostname configuration needs to be done, so the SSL requests to 127.0.0.1 get redirected to the domain?
I'm still getting this same error. Are there any solutions?
Sometimes the errors don't go away when I mess with configuration, perhaps implying that other errors are clobbered.
Related, but closed issues, appear to be:
pip freeze output:
Running on Ubuntu LTS on Google Cloud per instructions here https://the-littlest-jupyterhub.readthedocs.io/.