aiohttp.client_exceptions.ClientOSError: [Errno 1] [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2548)

[Zapzatron]

Windows 10 Python 3.10.9

package versions: aiohttp==3.9.3 fastapi==0.110.0 uvicorn[standard]==0.28.0 ngrok==1.2.0

client.py:

import aiohttp
import asyncio
from source import config

path = f"https://{config.DOMAIN}/some_long_func"

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.request("post", path, json={"some": "message"}) as response:
            print(await response.json())

asyncio.run(main())

api.py:

import ngrok
import uvicorn
import asyncio
from source import config
from fastapi import FastAPI, Request

app = FastAPI()

@app.post('/some_long_func')
async def some_long_func(request: Request):
    print(await request.json())
    await asyncio.sleep(60 * 5)
    return await request.json()

ngrok_tunnel = ngrok.forward(config.API_PORT, authtoken=config.NGROK_AUTHTOKEN, domain=config.DOMAIN)

print(f"Public URL: {ngrok_tunnel.url()}")

if __name__ == "__main__":
    uvicorn.run(app, port=config.API_PORT)

Error:

Traceback (most recent call last):
  File "F:\Games\Programming_Projects\Python\Zapzatron_Club\Retrieval_QA\local\test\ngrok_ssl_error\client.py", line 14, in <module>
    asyncio.run(main())
  File "F:\Techno\Languages\Python3109\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "F:\Techno\Languages\Python3109\lib\asyncio\base_events.py", line 649, in run_until_complete
    return future.result()
  File "F:\Games\Programming_Projects\Python\Zapzatron_Club\Retrieval_QA\local\test\ngrok_ssl_error\client.py", line 10, in main
    async with session.request("post", path, json={"some": "message"}) as response:
  File "F:\Games\Programming_Projects\Python\Zapzatron_Club\Retrieval_QA\.venv\lib\site-packages\aiohttp\client.py", line 1194, in __aenter__
    self._resp = await self._coro
  File "F:\Games\Programming_Projects\Python\Zapzatron_Club\Retrieval_QA\.venv\lib\site-packages\aiohttp\client.py", line 605, in _request
    await resp.start(conn)
  File "F:\Games\Programming_Projects\Python\Zapzatron_Club\Retrieval_QA\.venv\lib\site-packages\aiohttp\client_reqrep.py", line 966, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "F:\Games\Programming_Projects\Python\Zapzatron_Club\Retrieval_QA\.venv\lib\site-packages\aiohttp\streams.py", line 622, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 1] [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2548)

[bobzilladev]

Hi there, thanks for writing in! It appears that is a TLS enabled backend, in order for ngrok.forward to connect to that it will need to have the scheme as part of the address so it knows to use encryption, something like:

ngrok.forward(f"https://{config.DOMAIN}:{config.API_PORT}", authtoken=config.NGROK_AUTHTOKEN)

That will use the certificates on the host to validate the backend. If it is a self-signed cert than an environment variable can be used to specify the filename of the trusted certs, e.g. SSL_CERT_FILE=/path/to/ca.crt. A new option is specifying verify_upstream_tls=False to disable verification entirely.

The TLS Backends section of the readme has additional details. Hopefully that is helpful, let us know if anything else comes up!

[Zapzatron]

If I change it to

ngrok_tunnel = ngrok.forward(f "https://{config.DOMAIN}:{config.API_PORT}", authtoken=config.NGROK_AUTHTOKEN)

it generates a random link, but I need it on the domain.

If I change to

ngrok_tunnel = ngrok.forward(config.API_PORT, authtoken=config.NGROK_AUTHTOKEN, domain=f"https://{config.DOMAIN}")

I get the error

ValueError: ('failed to start listener', 'Custom subdomains are a feature on ngrok's paid plans.\nFailed to bind the custom subdomain 'ngrok-free.app' for the account '...'.\nThis account is on the 'Free' plan.\nUpgrade to a paid plan at: https://dashboard.ngrok.com/billing/subscription', 'ERR_NGROK_313')

In config.DOMAIN is the free domain from ngrok.

[bobzilladev]

Ah, in that case the code seems fine, except for the 5 minute sleep which is getting a gateway timeout (a 4 minute sleep works, as you've likely found). Hitting the ngrok endpoint with curl:

curl -v -XPOST https://<subdomain>.ngrok.app/some_long_func -d '{"some": "message"}'

Returns:

< HTTP/2 503 
< content-type: text/html
< ngrok-error-code: ERR_NGROK_3004
...
The server returned an invalid or incomplete HTTP response. (ERR_NGROK_3004)

For something this long running that returns no response until done, it may be better to use something like a websocket, or a TCP tunnel type. Another possibility could be converting the API into an asynchronous style, where the http request starts a background task and returns the http response immediately with an identifier. Then the client can either poll for status periodically or register a webhook callback for when the task is complete.

Hopefully this is getting more to the crux of the issue, thanks for the response!

[bobzilladev]

Hello, going to close out this issue, feel free to reopen if you run into anything more. Thank you for writing in!

[Zapzatron]

Thanks, I forgot to close it. The problem was solved by sending the task status every 20 seconds.