SSL handshake now failing -

[jerryneedell]

I just updated to the latest adafruit_requests and now the same test I have been running on esp32s2s is failing

Adafruit CircuitPython 6.1.0-alpha.0-175-gaca53cf41 on 2020-10-31; Metro ESP32S2 with ESP32S2
>>> import wifi_test
cp mem free 2034288
idf total mem 165384
idf mem free 51348
idf largest block 47012
<Network> Needell Airport -43 1
<Network> WYZE_FBAB62DE71811B38 -64 11
<Network> central2.4 -78 8
<Network> 31A -79 8
<Network> Expelliarmus2 -85 10
None
ip 10.0.0.30
8.8.4.4
ping 0.038
200
This is a test of Adafruit WiFi!
If you can read this, its working :)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "wifi_test.py", line 39, in <module>
  File "adafruit_requests.py", line 555, in get
  File "adafruit_requests.py", line 535, in request
  File "adafruit_requests.py", line 429, in _get_socket
  File "adafruit_requests.py", line 425, in _get_socket
OSError: Failed SSL handshake
>>> 

this is the request that fails

response = requests.get("https://httpbin.org/get")
print(response.status_code)
print(response.json())

if I revert back to the adafruit_requests from the 20201016 bundle , it works.... with the older library

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 6.1.0-alpha.0-175-gaca53cf41 on 2020-10-31; Metro ESP32S2 with ESP32S2
>>>
>>> import wifi_test
cp mem free 2034336
idf total mem 165384
idf mem free 50868
idf largest block 47012
<Network> Needell Airport -65 6
<Network> Needell Airport -43 1
<Network> WYZE_FBAB62DE71811B38 -69 11
<Network> central2.4 -79 8
<Network> 31A -79 8
<Network> Expelliarmus2 -86 10
None
ip 10.0.0.30
8.8.4.4
ping 0.037
200
This is a test of Adafruit WiFi!
If you can read this, its working :)

200
{'url': 'https://httpbin.org/get', 'headers': {'User-Agent': 'Adafruit CircuitPython', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-5f9dc0e2-44ad6a752dab2d383a9e51f8'}, 'args': {}, 'origin': '73.61.89.123'}

200
{'timezone': 'America/New_York', 'utc_datetime': '2020-10-31T19:54:11.744445+00:00', 'raw_offset': -18000, 'client_ip': '73.61.89.123', 'dst_from': '2020-03-08T07:00:00+00:00', 'unixtime': 1604174051, 'utc_offset': '-04:00', 'datetime': '2020-10-31T15:54:11.744445-04:00', 'week_number': 44, 'abbreviation': 'EDT', 'day_of_year': 305, 'day_of_week': 6, 'dst': True, 'dst_offset': 3600, 'dst_until': '2020-11-01T06:00:00+00:00'}

200
circuitpython open issues 345
circuitpython stars 1885
done

[ladyada]

out of curiosity, what if you connect to httpbin first?

[jerryneedell]

also get the same result on a unfeathers2

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 6.1.0-alpha.0-141-g05d663b18 on 2020-10-29; FeatherS2 with ESP32S2
>>> import wifi_test
cp mem free 8181696
idf total mem 165688
idf mem free 53288
idf largest block 52880
<Network> Needell Airport -61 6
<Network> Needell Airport -53 1
<Network> WYZE_FBAB62DE71811B38 -62 11
<Network> central2.4 -76 8
<Network> Expelliarmus2 -82 10
None
ip 10.0.0.42
8.8.4.4
ping 0.034
200
This is a test of Adafruit WiFi!
If you can read this, its working :)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "wifi_test.py", line 39, in <module>
  File "adafruit_requests.py", line 555, in get
  File "adafruit_requests.py", line 535, in request
  File "adafruit_requests.py", line 429, in _get_socket
  File "adafruit_requests.py", line 425, in _get_socket
OSError: Failed SSL handshake
>>> 

but interestingly on a saola_wrover with a recent CP update by @tannewt it passes that request, but fails on a later one... the failing request here is

response = requests.get("https://api.github.com/repos/adafruit/circuitpython")
print(response.status_code)
#print(response.json())
print("circuitpython open issues", response.json()["open_issues_count"])
print("circuitpython stars", response.json()["stargazers_count"])

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 6.0.0-rc.0-19-gd4e5e9320 on 2020-10-30; Saola 1 w/Wrover with ESP32S2
>>> 
>>> import wifi_test
cp mem free 2034704
idf total mem 166336
idf mem free 88800
idf largest block 46576
<Network> Needell Airport -59 6
<Network> WYZE_FBAB62DE71811B38 -61 11
<Network> 31A -87 8
<Network> central2.4 -89 8
<Network> Expelliarmus2 -81 10
connection started 0
None
ip 10.0.0.135
8.8.4.4
ping 0.039
200
This is a test of Adafruit WiFi!
If you can read this, its working :)

200
{'url': 'https://httpbin.org/get', 'headers': {'User-Agent': 'Adafruit CircuitPython', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-5f9e98db-1c5965fb7b10cf447e75a7fb'}, 'args': {}, 'origin': '73.61.89.123'}

200
{'timezone': 'America/New_York', 'utc_datetime': '2020-11-01T11:15:40.557482+00:00', 'raw_offset': -18000, 'client_ip': '73.61.89.123', 'dst_from': None, 'unixtime': 1604229340, 'utc_offset': '-05:00', 'datetime': '2020-11-01T06:15:40.557482-05:00', 'week_number': 44, 'abbreviation': 'EST', 'day_of_year': 306, 'day_of_week': 0, 'dst': False, 'dst_offset': 0, 'dst_until': None}

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "wifi_test.py", line 52, in <module>
  File "adafruit_requests.py", line 555, in get
  File "adafruit_requests.py", line 535, in request
  File "adafruit_requests.py", line 429, in _get_socket
  File "adafruit_requests.py", line 425, in _get_socket
OSError: Failed SSL handshake
>>> 

[jerryneedell]

out of curiosity, what if you connect to httpbin first?

@ladyada I'm not sure what you are suggesting here. What should I change?

[ladyada]

try connecing to httpbin website (currently failing) before any other sites rather than after adafruit wifitest

[jerryneedell]

with that first it fails differently

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 6.1.0-alpha.0-175-gaca53cf41 on 2020-10-31; Metro ESP32S2 with ESP32S2
>>> import wifi_test
cp mem free 2034288
idf total mem 165384
idf mem free 52880
idf largest block 52488
<Network> Needell Airport -63 6
<Network> Needell Airport -53 1
<Network> WYZE_FBAB62DE71811B38 -62 11
<Network> central2.4 -86 8
<Network> central2.4 -87 5
<Network> 31A -88 5
<Network> Expelliarmus2 -86 10
None
ip 10.0.0.30
8.8.4.4
ping 0.047
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "wifi_test.py", line 33, in <module>
  File "adafruit_requests.py", line 555, in get
  File "adafruit_requests.py", line 535, in request
  File "adafruit_requests.py", line 440, in _get_socket
espidf.MemoryError: 
>>> 

code

import ipaddress
import wifi
import socketpool
import time
import adafruit_requests
import ssl
import espidf

import gc

print("cp mem free", gc.mem_free())

print("idf total mem", espidf.heap_caps_get_total_size())
print("idf mem free", espidf.heap_caps_get_free_size())
print("idf largest block", espidf.heap_caps_get_largest_free_block())

for network in wifi.radio.start_scanning_networks():
    print(network, network.ssid, network.rssi, network.channel)

wifi.radio.stop_scanning_networks()

print(wifi.radio.connect("<ssid>", "<password>"))

print("ip", wifi.radio.ipv4_address)

ipv4 = ipaddress.ip_address("8.8.4.4")
print(ipv4)
print("ping", wifi.radio.ping(ipv4))

pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

response = requests.get("https://httpbin.org/get")
print(response.status_code)
print(response.json())

response = requests.get("http://wifitest.adafruit.com/testwifi/index.html")
print(response.status_code)
print(response.text)

print()

response = requests.get("http://worldtimeapi.org/api/ip")
print(response.status_code)
print(response.json())

print()

#response = requests.get("https://api.thingspeak.com/channels/1417/feeds.json?results=1")
response = requests.get("https://api.github.com/repos/adafruit/circuitpython")
print(response.status_code)
#print(response.json())
print("circuitpython open issues", response.json()["open_issues_count"])
print("circuitpython stars", response.json()["stargazers_count"])
print("done")

[jerryneedell]

FYI - it works if I use http instead of https response = requests.get("http://httpbin.org/get")

[jerryneedell]

Seem like all https requests are now failing -- some worked with the older library and some had SSL handshake errors. I had been assuming the SSL handshake issues we due to outdated certificates, but not really sure.

[mytechnotalent]

@jerryneedell I can confirm identical behavior on the Metro ESP32-S2.

[jerryneedell]

hmm -- getting stranger. the https request succeeds with the test version of CP from @tannewt repo "fix_recv"

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 6.0.0-rc.0-19-gd4e5e9320 on 2020-10-30; Saola 1 w/Wrover with ESP32S2
>>> 
>>> import wifi_test
<Network> Needell Airport -59 6
<Network> Needell Airport -39 1
<Network> WYZE_FBAB62DE71811B38 -55 11
<Network> 31A -82 8
<Network> central2.4 -83 8
<Network> 31A -86 5
<Network> Expelliarmus2 -82 10
connection started 0
None
ip 10.0.0.135
8.8.4.4
ping 0.035
200
{'url': 'https://httpbin.org/get', 'headers': {'User-Agent': 'Adafruit CircuitPython', 'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-5f9ff439-570b61511ddf709971841524'}, 'args': {}, 'origin': '73.61.89.123'}
200
This is a test of Adafruit WiFi!
If you can read this, its working :)

200
{'timezone': 'America/New_York', 'utc_datetime': '2020-11-02T11:57:46.749203+00:00', 'raw_offset': -18000, 'client_ip': '73.61.89.123', 'dst_from': None, 'unixtime': 1604318266, 'utc_offset': '-05:00', 'datetime': '2020-11-02T06:57:46.749203-05:00', 'week_number': 45, 'abbreviation': 'EST', 'day_of_year': 307, 'day_of_week': 1, 'dst': False, 'dst_offset': 0, 'dst_until': None}

done
>>> 

but it fails with a bulid from the tip of main

Adafruit CircuitPython 6.1.0-alpha.0-181-gb468bb6b9 on 2020-11-02; Saola 1 w/Wrover with ESP32S2
>>> import wifi_test
<Network> Needell Airport -59 6
<Network> Needell Airport -45 1
<Network> WYZE_FBAB62DE71811B38 -62 11
<Network> central2.4 -91 8
<Network> Expelliarmus2 -82 10
None
ip 10.0.0.135
8.8.4.4
ping 0.04
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "wifi_test.py", line 33, in <module>
  File "adafruit_requests.py", line 555, in get
  File "adafruit_requests.py", line 535, in request
  File "adafruit_requests.py", line 440, in _get_socket
espidf.MemoryError: 
>>> 

so perhaps this is a CP issue rather than a Requests issue

[mytechnotalent]

@jerryneedell I will try to reproduce as well however I am getting the SSL issue as you did originally.

[jerryneedell]

I have been assuming that the SSL error are due to the out-of -date certificates but the espidf.Memory Error is a real issue -- possible related to https://github.com/adafruit/circuitpython/issues/3562

[mytechnotalent]

@jerryneedell I can confirm similar behavior.

Device - Metro ESP32-S2 (built and flashed firmware as of this morning)

Scenario 1:

import ipaddress
import wifi
import adafruit_requests
import time
import ssl
import socketpool
from secrets import WIFI, WIFI_PASSWORD

print(wifi.radio.connect(WIFI, WIFI_PASSWORD))

print('ip', wifi.radio.ipv4_address)

ipv4 = ipaddress.ip_address('8.8.8.8')
print('ping', wifi.radio.ping(ipv4))

pool = socketpool.SocketPool(wifi.radio)

http = adafruit_requests.Session(pool, ssl.create_default_context())

TEXT_URL = "http://wifitest.adafruit.com/testwifi/index.html"
JSON_GET_URL = "https://httpbin.org/get"
JSON_POST_URL = "http://httpbin.org/post"

print("Fetching text from %s" % TEXT_URL)
response = http.get(TEXT_URL)
print("-" * 40)

print("Text Response: ", response.text)
print("-" * 40)
response.close()

print("Fetching JSON data from %s" % JSON_GET_URL)
response = http.get(JSON_GET_URL)
print("-" * 40)

print("JSON Response: ", response.json())
print("-" * 40)
response.close()

data = "31F"
print("POSTing data to {0}: {1}".format(JSON_POST_URL, data))
response = http.post(JSON_POST_URL, data=data)
print("-" * 40)

json_resp = response.json()
# Parse out the 'data' key from json_resp dict.
print("Data received from server:", json_resp["data"])
print("-" * 40)
response.close()

json_data = {"Date": "July 25, 2019"}
print("POSTing data to {0}: {1}".format(JSON_POST_URL, json_data))
response = http.post(JSON_POST_URL, json=json_data)
response = http.post(JSON_POST_URL, json=json_data)
print("-" * 40)

json_resp = response.json()
# Parse out the 'json' key from json_resp dict.
print("JSON Data received from server:", json_resp["json"])
print("-" * 40)
response.close()

Result 1:

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
None
ip 192.168.1.10
ping 0.019
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
Text Response:  This is a test of Adafruit WiFi!
If you can read this, its working :)

----------------------------------------
Fetching JSON data from https://httpbin.org/get
Traceback (most recent call last):
  File "code.py", line 34, in <module>
  File "/lib/adafruit_requests.py", line 555, in get
  File "/lib/adafruit_requests.py", line 535, in request
  File "/lib/adafruit_requests.py", line 429, in _get_socket
  File "/lib/adafruit_requests.py", line 425, in _get_socket
OSError: Failed SSL handshake

Scenario 2:

import ipaddress
import wifi
import adafruit_requests
import time
import ssl
import socketpool
from secrets import WIFI, WIFI_PASSWORD

print(wifi.radio.connect(WIFI, WIFI_PASSWORD))

print('ip', wifi.radio.ipv4_address)

ipv4 = ipaddress.ip_address('8.8.8.8')
print('ping', wifi.radio.ping(ipv4))

pool = socketpool.SocketPool(wifi.radio)

http = adafruit_requests.Session(pool, ssl.create_default_context())

# TEXT_URL = "http://wifitest.adafruit.com/testwifi/index.html"
TEXT_URL = "https://google.com"
JSON_GET_URL = "https://httpbin.org/get"
JSON_POST_URL = "http://httpbin.org/post"

print("Fetching text from %s" % TEXT_URL)
response = http.get(TEXT_URL)
print("-" * 40)

print("Text Response: ", response.text)
print("-" * 40)
response.close()

print("Fetching JSON data from %s" % JSON_GET_URL)
response = http.get(JSON_GET_URL)
print("-" * 40)

print("JSON Response: ", response.json())
print("-" * 40)
response.close()

data = "31F"
print("POSTing data to {0}: {1}".format(JSON_POST_URL, data))
response = http.post(JSON_POST_URL, data=data)
print("-" * 40)

json_resp = response.json()
# Parse out the 'data' key from json_resp dict.
print("Data received from server:", json_resp["data"])
print("-" * 40)
response.close()

json_data = {"Date": "July 25, 2019"}
print("POSTing data to {0}: {1}".format(JSON_POST_URL, json_data))
response = http.post(JSON_POST_URL, json=json_data)
response = http.post(JSON_POST_URL, json=json_data)
print("-" * 40)

json_resp = response.json()
# Parse out the 'json' key from json_resp dict.
print("JSON Data received from server:", json_resp["json"])
print("-" * 40)
response.close()

Result 2:

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
None
ip 192.168.1.10
ping 0.007
Fetching text from https://google.com
Traceback (most recent call last):
  File "code.py", line 27, in <module>
  File "/lib/adafruit_requests.py", line 555, in get
  File "/lib/adafruit_requests.py", line 535, in request
  File "/lib/adafruit_requests.py", line 440, in _get_socket
espidf.MemoryError: 

Press any key to enter the REPL. Use CTRL-D to reload.

[tannewt]

For debugging SSL, it'd be helpful to have the UART output with mbedtls logging on. The esp TLS wrapper obscures the error numbers iirc. I'm going to get my SSL changes in a PR today. The main thing is I tried to reduce the overall wifi memory footprint and the footprint of each SSL connection.

[mytechnotalent]

@tannewt let me know when you do as I am working on a very basic tutorial based on the one @FoamyGuy @anecdata and I put together except this one is for the Metro ESP32-S2 as I would like to test your SSL changes. I am assuming once you have it merged I would grab the most recent version of the firmware as I am assuming SSL is built in?

[tannewt]

My settings changes for SSL are in: https://github.com/adafruit/circuitpython/pull/3639

[mytechnotalent]

Great news @tannewt I pulled in your branch and it works!

ip 192.168.1.10
ping 0.023
Fetching DATA_SOURCE
Fetching ARTICLE
looperhacks
230
GitHub: Widespread Injection Vulnerabilities in Actions

[mytechnotalent]

Larger code demo in https://github.com/adafruit/circuitpython/pull/3639

[ladyada]

i think this issue is resolved - or at least i havent seen it in a while. please comment if you still see it - will close for now!