fix: Networking broken on Apple M1

[0xlxy]

After running ctc setup to config my RPC url, and then running llama pools, I got the following error:

ConnectionKey(host='yields.llama.fi', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)

The RPC URL is valid and works fine on other projects. Wonder what's the causes of it and it would be great to have better error message display. Thanks!

[detroitcoder]

This is likely not related to the RPC URL setup because this data is pulled directly from the defi llama API directly. It looks like this error is an http request error. Are you able to access the below URL from your network in a browser? You should get json back that says{"message": "Not Found"} https://yields.llama.fi/

-Michael

[sslivkoff]

are you using an M1 mac? Ive been receiving multiple reports of networking errors in M1 environments

if not, can you provide your os version + python version

[0xlxy]

Yes, I am using M1 Pro.

• OS Version: macOS Monterey 12.3
• Python Version: Python 3.10.6

[sslivkoff]

not sure why, but networking seems to break on apple m1's

I don't have access to an M1 right now so unfortunately cannot debug this very well

[0xlxy]

that's fine - thanks sir

[kasperpawlowski]

@XingyuLi20 Navigate to your /Applications/Python 3.x/ folder and double click the Install Certificates.command

More on this in /Applications/Python 3.x/ReadMe.rtf

[sslivkoff]

@kasperpawlowski is it a cert issue? that would explain a lot

[kasperpawlowski]

Yes, I did some debugging and it appeared to be an issue with SSL cert verification. At least in my case.

[SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')]

[sslivkoff]

@kasperpawlowski and did running /Applications/Python 3.x/Install Certificates.command seem to totally resolve the issue?

if so then ctc can check for this in a straightforward way and give M1 users a helpful error message about how to resolve

[kasperpawlowski]

@sslivkoff that's right, running /Applications/Python 3.x/Install Certificates.command resolved issue completely.

it's hard to say though if only M1 users are affected. for me that was the case, but I don't have any other macbook handy to verify. it seems more like a python/pip issue. from python 3.10 readme:

Certificate verification and OpenSSL

This package includes its own private copy of OpenSSL 1.1.1. The trust certificates in system and user keychains managed by the Keychain Access application and the security command line utility are not used as defaults by the Python ssl module. A sample command script is included in /Applications/Python 3.10 to install a curated bundle of default root certificates from the third-party certifi package (https://pypi.org/project/certifi/). Double-click on Install Certificates to run it.

The bundled pip has its own default certificate store for verifying download connections.

so I think rather than giving the helpful error message only to M1 users, it would be better to catch this exception right here in the async_send_http() and give the message to all the users if it's SSLCertVerificationError https://github.com/fei-protocol/checkthechain/blob/8a06fe91de5e99020c4fbc97671f5b65da59eb8f/src/ctc/rpc/rpc_protocols/rpc_http_async.py#L16

I don't have time to investigate any further now, but the message I was seeing was something related to ConnectionKey. it was not http rpc request failed after... which means that the context manager in async_send_http() was not returning the response properly, thus the request was not retried. looks like aiohttp.ClientSession just throws in that case

[sslivkoff]

@kasperpawlowski thanks for the insights. I will dig into this more when I get a chance

[0xlxy]

Hey @kasperpawlowski thanks for the response. However, I'm having trouble finding /Applications/Python 3.x/Install Certificates.command on my laptop. I was able to navigate to the site-packages folder but don't see any relevant commands. Any instructions for find Install Certificates.command? Appreciate it!

[sslivkoff]

@XingyuLi20 can you post the full stack trace that of the error that you get during setup?

[0xlxy]

Command: llama pools --debug Full Stack Trace:

[ENTERING DEBUGGER]
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1089, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 1119, in _create_connection_transport
    await waiter
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 534, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 975, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/toolcli/command_utils/execution.py", line 56, in run_cli
    execute_parsed_command(parse_spec=parse_spec, args=args)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/toolcli/command_utils/execution.py", line 84, in execute_parsed_command
    execute_command_spec(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/toolcli/command_utils/execution.py", line 130, in execute_command_spec
    asyncio.run(coroutine)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/toolcli/command_utils/execution.py", line 143, in _async_execute_in_context_manager
    await function(**args)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ctc/protocols/llama_utils/cli/llama_pools_command.py", line 121, in async_llama_pools_command
    await llama_yields.async_print_llama_pools_summary(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ctc/protocols/llama_utils/llama_yields.py", line 36, in async_print_llama_pools_summary
    pools = await llama_requests.async_get_llama_pools(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ctc/protocols/llama_utils/llama_requests.py", line 135, in async_get_llama_pools
    data = await async_request_llama_data('pools')
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/ctc/protocols/llama_utils/llama_requests.py", line 53, in async_request_llama_data
    async with session.get(url) as response:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host yields.llama.fi:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')]

> /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/aiohttp/connector.py(988)_wrap_create_connection()
-> raise ClientConnectorCertificateError(req.connection_key, exc) from exc
(Pdb)

[kasperpawlowski]

@XingyuLi20 it might be worth reinstalling python on your machine using the official installer. it seems that you're facing the same issue, but I have no idea where you can find the relevant Install Certificates.command under your setup

https://www.python.org/downloads/