httpx.ConnectError: All connection attempts failed - Wrong http scheme being selected?

[shrike71]

Got round to setting up an https (self-signed) test environment for local testing before putting it out onto our test AWS envs. Installation seemed to go well up to and until running the https://<mysite>/admin/ bit to complete the walkthough of the fief workspace, which returned an 'Internal Server Error':

for this docker-compose.yml setup:

version: "3.9"

services:
  fief:
    container_name: fief
    hostname: fief
    image: ghcr.io/fief-dev/fief:latest
    ports:
      - "8001:8001"
    environment:
      - PORT=8001  # Run Uvicorn on 8001 to match the external port
      - LOG_LEVEL=DEBUG  # Run Uvicorn on 8001 to match the external port
      - DATABASE_TYPE=POSTGRESQL
      - DATABASE_URL=postgresql://postgres:postgres@postgres-eval_db_1:5432/fief
      - SECRET=*************
      - FIEF_CLIENT_ID=***********
      - FIEF_CLIENT_SECRET=**********
      - ENCRYPTION_KEY=********
      - ROOT_DOMAIN=fief.ingress.local
      - FIEF_DOMAIN=fief.ingress.local
      - FIEF_BASE_URL=https://fief.ingress.local
      - CSRF_COOKIE_SECURE=False
      - LOGIN_SESSION_COOKIE_SECURE=False
      - SESSION_COOKIE_SECURE=False
      - FIEF_ADMIN_SESSION_COOKIE_SECURE=False
    volumes:
      - fief-server-data:/data
    restart: unless-stopped
    extra_hosts:
        - "ingress.local:127.0.0.1"  # Map ingress.local hostname to localhost inside container
        - "fief.ingress.local:127.0.0.1"  # Map ingress.local hostname to localhost inside container
    networks:
      - eval

volumes:
  fief-server-data:
    external: true

networks:
  eval:
    external: true

I ended up putting uvicorn into debug mode and i put some print statements into

apps/admin/routers/auth.py at about line 22 to spit out the URI:

Here's the relevant part of the stack trace:

[2022-05-12 06:19:45,672] [PID 10] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '1.12.3' is booting up.
[2022-05-12 06:19:45,670] [PID 15] [MainThread] [dramatiq.WorkerProcess(0)] [INFO] Worker process is ready for action.
[2022-05-12 06:19:45,679] [PID 22] [MainThread] [dramatiq.ForkProcess(0)] [INFO] Fork process 'dramatiq.middleware.prometheus:_run_exposition_server' is ready for action.
INFO:     Started server process [17]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     172.18.0.7:45702 - "GET /auth/userinfo HTTP/1.1" 401 Unauthorized
INFO:     172.18.0.7:45704 - "GET /workspaces/ HTTP/1.1" 401 Unauthorized
redir_uri: http://fief.ingress.local/admin/api/auth/callback
INFO:     172.18.0.7:45706 - "GET /auth/login?redirect_uri=https%3A%2F%2Ffief.ingress.local%2Fadmin%2F HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 372, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/debug.py", line 96, in __call__
    raise exc from None
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/debug.py", line 93, in __call__
    await self.app(scope, receive, inner_send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 261, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 115, in _run_asgi3
    return await self._run_app(scope, lambda: self.app(scope, receive, send))
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 162, in _run_app
    raise exc from None
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/asgi.py", line 159, in _run_app
    return await callback()
  File "/usr/local/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 408, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 261, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/fief/apps/admin/routers/auth.py", line 23, in login
    url = await fief.auth_url(
  File "/usr/local/lib/python3.10/site-packages/fief_client/client.py", line 374, in auth_url
    openid_configuration = await self._get_openid_configuration()
  File "/usr/local/lib/python3.10/site-packages/fief_client/client.py", line 463, in _get_openid_configuration
    response = await client.send(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1593, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1621, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1658, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1695, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

The debug line above is labeled'redir_uri:'. What's bothering me is that it is an 'http' scheme instead of 'https'. Might this be the issue? FWIW, i also took out any 'http' only entries in the DB using PGAdmin ... but it does not make any difference.

Ingress is controlled by NGINX proxy manager, with a self-signed cert, https-only access and HTTP/2 support set up:

And FWIW, there's full access from the proxy to each container, and I can also 'curl' in between each container

[frankie567]

I think there is two things there:

• The redirection URL is generated following the scheme received in the original request. When serving behind a proxy, Uvicorn may refuse to forward SSL if the IP is not allowed and fallback to HTTP. To solve this, try to set the environment variable FORWARDED_ALLOW_IPS to *.
• Concerning the error you get, it looks like HTTPX is not able to establish an HTTPS connection on https://fief.ingress.local. You confirm that if you make a curl request to https://fief.ingress.local from inside the container, it works?

[shrike71]

Hi frankie567,

Ok... think i got to the bottom of it.

Based on your response above, i realized that although I had tried curling into the container from the other dependent containers in the network, I HADN'T tried a curl on the local container. After implementing the changes you suggested above (FORWARDED_ALLOW_IPS did not have any effect on uvicorn, see below), i realized that the entry in the /etc/hosts file setting fief.ingress.local to 127.0.0.1 was the problem. It was not getting out of the container. So I set this to my host VM IP of 192.168.56.150 and i then got this in the logs:

File "/usr/local/lib/python3.10/site-packages/fief/apps/admin/routers/auth.py", line 22, in login
    url = await fief.auth_url(
  File "/usr/local/lib/python3.10/site-packages/fief_client/client.py", line 374, in auth_url
    openid_configuration = await self._get_openid_configuration()
  File "/usr/local/lib/python3.10/site-packages/fief_client/client.py", line 462, in _get_openid_configuration
    response = await client.send(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1593, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1621, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1658, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1695, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
    stream = await self._connect(request)
  File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 150, in _connect
    stream = await stream.start_tls(**kwargs)
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 72, in start_tls
    raise exc
  File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 63, in start_tls
    ssl_stream = await anyio.streams.tls.TLSStream.wrap(
  File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 100, in wrap
    await wrapper._call_sslobject_method(ssl_object.do_handshake)
  File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 108, in _call_sslobject_method
    result = func(*args)
  File "/usr/local/lib/python3.10/ssl.py", line 974, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:997)

Based on the output and some googling around, i wondered whether FORWARDED_ALLOW_IPS was having any effect on the running uvicorn instance, so I shipped a copy of my certificate and key files into the container as /tmp/selfsign.crt and /tmp/selfsign.key. Then I changed cli.py to run uvicorn in this config:

uvicorn.run("fief.app:app", host=host, port=port, debug=True, ssl_keyfile="/tmp/selfsign.key", ssl_certfile="/tmp/selfsign.crt")

However, the running service did not like this at all. Although the uvicorn instance was running, i seemed to get nothing but 'connection is reset' errors when trying a local CURL, and a subsequent 504 error in the nginx proxy when trying to connect externally. Resetting this back to

uvicorn.run("fief.app:app", host=host, port=port, debug=True)

at least got the service responding with reasonable errors again. following the stack trace down, i next looked at the fief_client in fief_client/client.py on the container. Changing the contexts from

with httpx.Client(base_url=self.base_url) as client: to with httpx.Client(base_url=self.base_url, verify='/tmp/selfsign.crt') as client:

and

async with httpx.AsyncClient(base_url=self.base_url) to async with httpx.AsyncClient(base_url=self.base_url,verify='/tmp/selfsign.crt')

Seems to have solved it for now.

[frankie567]

So, from what I understand, the problems comes from the fact that you are using a self-signed certificate, right? Could you try something like this and tell me if it works:

async with httpx.AsyncClient(base_url=self.base_url, verify=False)

If so, I'm wondering if an option/variable to disable SSL check here would be useful for cases like yours.

EDIT: The other solution I'm thinking about is to always call localhost there instead of relying on the external URL. But I remember I had some tricky problems with this during development. I'll try to explore this more.

[shrike71]

Exactly right.

'verify=False'

works as well... i.e. from a development/staging perspective it would save you having to package your certs with the container

[frankie567]

Hello @shrike71!

I've made some improvements about this in version 0.12.18.

Normally, it should avoid you all the tricks regarding routing with extra hosts and SSL. Could you try on your side and tell me if it improves the situation? 😄

[shrike71]

Proceeding with a test on this. Might you also want to consider a 'pass-through' setting for 'verify=False' or 'verify=' for the fief-client pip package?

[frankie567]

Yes, I'll consider it!

[frankie567]

Hello @shrike71 👋

Did you have the chance to try 0.12.18 with the improvements on internal routing? 😃

Cheers!

[badarsebard]

Is there still consideration to accept verify= as a pass-through setting? Would be very useful for development, especially when working with a proxy.

[frankie567]

@badarsebard That's not something I kept in mind, but we could reconsider it, for sure! The problem is that you have self-signed certificates, right?

[badarsebard]

Correct. Specifically I've been running fief inside of a kubernetes (minikube) cluster behind an nginx proxy. Setting this through an environment variable would be ideal.

[frankie567]

Ok! I'll put this in the backlog!

[frankie567]

Hey there 👋 It's now possible to set custom verify and cert parameters on the client with version 0.14.6!

[ocontant]

I come late a little bit, but thanks for that feature! Greatly appreciated!