Added support for EC-Functions and DTLSv1.2 (Updated OpenSSL-Lib to V1.0.2j)

mcfreis commented 7 years ago

Included is a wrapper DtlsSocket (in wrapper.py) for easy usage with CoAPthon in a client/server environment. See also: https://github.com/mcfreis/CoAPthon

Attention: DLL updates for Win32 only! (Till now)

rbit commented 7 years ago

Thank you! A couple of thoughts:

First, it would be nice if the functionality could be separated into two patches, one for v1.2 and another for the configuration options that have been added via the new SSLContext.

Second, how would you propose we go about testing this? How do we know the v1.2 selection is having effect, and passes in all the cases where v1.0 passed? And how do we exercise and verify the new configuration options, for example EC? And if you'd like to add the CoAPthon wrapper, then we should find a way to test that as well.

As I was looking into whether the existing unit tests pass with this patch, I noticed that they seem to have been broken by a Python version released after my most recent updates. I'm investigating this now and should have the unit tests back up and running shortly.

mcfreis commented 7 years ago

Thank you for having a look at my changes. With the separation of the functionality, do you mean just two separat merges for a better understanding? I'm not actually sure where to split it up, because during development I included the configuration options directly within init, _init_server and _init_client of the SSLConnection.

Well about testing V1.2 and V1.0: I created a setup for myself "client - udp-proxy (https://github.com/mcfreis/udpcap) - server" on just one machine. I then analyzed the pcap files with Wireshark. I tested the following combinations (server/client): V1.0/V1.0, V1.2/V1.2, any/V1.0 and any/V1.2. All of them worked and showed the expected version in the Wireshark logs. I also did I few tests with EC.

Concerning the CoAPthon, I promised to include an example in their project. But of course the wrapper can be used for any kind of client/server application and therefor a short example without CoAPthon could also be supplied within this package.

Concerning the DLLs, where did you get the versions from? Maybe we should update them first to the newest V1.0.2 or even V1.1.x. If you would update them, then I would do the testing with my changes. A branch could be usefull.

Regards Björn

rbit commented 7 years ago

Yes, I mean two separate commits. I think it will be quite helpful in the long run to be able to trace to which functionality group a particular changed line belongs. Since there doesn't seem an inherent connection between v1.2 and EC configuration options, this separation ought to be possible. Perhaps it would help to do the patches sequentially, and base one on the other, if doing them entirely independently would result in conflicts. We'll want to ensure that all tests pass with each patch separately.

Great to hear about your validation approach. But we'll also need something that is accessible to PyDTLS users and developers. When someone makes a change down the road, they ought to be able to detect that they've broken the functionality that you contributed by running the unit test suite. Perhaps a unit test that verifies that connecting v1.0 and v1.2 doesn't work would be a good start?

I looks to me that testing all the options of your SSLContext might be a bit involved, since there are several options and many combinations. The approach I took for testing dtls was by adapting the standard library's test_ssl.py (now in unit.py). Perhaps you can do something similar. At minimum I think that there should be at least one unit test for each option.

The wrapper seems useful, and I have nothing against including it. But whatever we include we do need to test somehow. So some unit tests specifically for the wrapper sounds like a good approach.

I built the OpenSSL DLLs for Windows myself. I can build new versions that accommodate v1.2 for your patch. I'm unsure whether the MinGW versions continue to be useful; perhaps I should drop those.

mcfreis commented 7 years ago

I'm splitting the different packages up into smaller patches. But now I'm facing a problem concerning MTU size and auto discovery. I've commited a small patch under: https://github.com/mcfreis/pydtls/commit/b66fe6e84a825372813378d901348501fd75e56a

I just wanted to get OpenSSL 1.0.2j with your version (DTLSv1) including the unittests running, but unfortunately I need to manually adjust the MTU size. Do you have any ideas?

Regards, Björn

rbit commented 7 years ago

I just pushed a commit to master that fixes the unit tests. I'm using the following version of OpenSSL on Ubuntu 16.04: dtls.sslconnection.DTLS_OPENSSL_VERSION: 'OpenSSL 1.0.2g 1 Mar 2016'.

You can now run the unit tests with from the project root directory like this: python -m dtls.test.unit [-v]. Non-verbose output on Ubuntu looks like this for me:

Suite run: demux: platform-native, protocol: 2
.........................
----------------------------------------------------------------------
Ran 25 tests in 1.320s

OK
Suite run: demux: platform-native, protocol: 10
.........................
----------------------------------------------------------------------
Ran 25 tests in 1.322s

OK
Suite run: demux: routing, protocol: 2
.........................
----------------------------------------------------------------------
Ran 25 tests in 1.606s

OK
Suite run: demux: routing, protocol: 10
.........................
----------------------------------------------------------------------
Ran 25 tests in 1.621s

OK

Verbose output is as follows:

Suite run: demux: platform-native, protocol: 2
test_ciphers (__main__.BasicSocketTests) ...  server:  new connection from 127.0.0.1:(54037,)
 server:  new connection from 127.0.0.1:(35619,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae230>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab69ce60>
ok
test_constants (__main__.BasicSocketTests) ... ok
test_dtls_openssl_version (__main__.BasicSocketTests) ... ok
test_refcycle (__main__.BasicSocketTests) ... ok
test_wrapped_unconnected (__main__.BasicSocketTests) ... ok
test_sslwrap_simple (__main__.BasicTests) ... ok
test_connect (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(41346,)
 server:  new connection from 127.0.0.1:(44199,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae2a8>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae410>
ok
test_connect_ex (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(52288,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae230>
ok
test_get_server_certificate (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(53473,)
 server:  new connection from 127.0.0.1:(38715,)
 server:  new connection from 127.0.0.1:(60465,)

Verified certificate is
-----BEGIN CERTIFICATE-----
MIICDTCCAXYCCQCxc2uXBLZhDjANBgkqhkiG9w0BAQUFADBKMQswCQYDVQQGEwJV
UzETMBEGA1UECBMKV2FzaGluZ3RvbjETMBEGA1UEChMKUmF5IENBIEluYzERMA8G
A1UEAxMIUmF5Q0FJbmMwHhcNMTQwMTE4MjEwMjUwWhcNMjQwMTE2MjEwMjUwWjBM
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3RvbjEUMBIGA1UEChMLUmF5
IFNydiBJbmMxEjAQBgNVBAMTCVJheVNydkluYzCBnzANBgkqhkiG9w0BAQEFAAOB
jQAwgYkCgYEA2Mv6DsylMQHjRWjanFZvHfdjCBdDpBucuDVDIqWsfoMQSi3INFYj
9l+29ZuMZSHTNQaFBovMXdk2B0xWQXmPTAKBF3A+ifVy3nAlSDoYJa9+LgH8eVCt
aDwkNpaga4NxOouHoph8N1ZOVYIGrek5bXiBV7d3XZHwHG708IryOu8CAwEAATAN
BgkqhkiG9w0BAQUFAAOBgQBw0XUTYzfiI0Fi9g4GuyWD2hjET3NtrT4Ccu+Jiivy
EvwhzHtVGAPhrV+VCL8sS9uSOZlmfK/ZVraDiFGpJLDMvPP5y5fwq5VGrFuZispG
X6bTBq2AIKzGGXxhwPqD8F7su7bmZDnZFRMRk2Bh16rv0mtzx9yHtqC5YJZ2a3JK
2g==
-----END CERTIFICATE-----

 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae230>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae398>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae488>
ok
test_makefile_close (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(57367,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae2a8>
ok
test_non_blocking_connect_ex (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(45216,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae140>
ok
test_non_blocking_handshake (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(35796,)

Needed 4 calls to do_handshake() to establish session.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae0c8>
ok
test_asyncore_server (__main__.ThreadedTests)
Check the example asyncore integration. ... 
 server:  new connection from 127.0.0.1:(34198,)
 client:  sending 'TEST MESSAGE of mixed case\n'...
 client:  read 'test message of mixed case\n'
 client:  closing connection.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6ae140>
ok
test_echo (__main__.ThreadedTests)
Basic test of an SSL client connecting to a server ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad62e5f0>
 server:  new connection from ('127.0.0.1', 49761)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
 client:  sending 'FOO\n'...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending bytearray(b'FOO\n')...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending <memory at 0x7f75ab6f12b0>...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  closing connection.
 server: client closed connection
ok
test_empty_cert (__main__.ThreadedTests)
Connecting with an empty cert file ... 
SSLError is [(151441516, 'error:0906D06C:PEM routines:PEM_read_bio:no start line'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_getpeercert (__main__.ThreadedTests) ... 
{'notAfter': 'Jan 16 21:02:50 2024 GMT',
 'subject': ((('countryName', u'US'),),
             (('stateOrProvinceName', u'Washington'),),
             (('organizationName', u'Ray Srv Inc'),),
             (('commonName', u'RaySrvInc'),))}
Connection cipher is ('AES256-SHA', 'TLSv1/SSLv3', 256).
ok
test_handshake_timeout (__main__.ThreadedTests) ... ok
test_malformed_cert (__main__.ThreadedTests)
Connecting with a badly formatted certificate (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_malformed_key (__main__.ThreadedTests)
Connecting with a badly formatted key (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_nonexisting_cert (__main__.ThreadedTests)
Connecting with a non-existing cert file ... 
SSLError is [(336602136, 'error:14102418:SSL routines:dtls1_read_bytes:tlsv1 alert unknown ca')]
ok
test_protocol_dtlsv1 (__main__.ThreadedTests)
Connecting to a DTLSv1 server with various client options ... 
 DTLSv1->DTLSv1 CERT_NONE
 DTLSv1->DTLSv1 CERT_OPTIONAL
 DTLSv1->DTLSv1 CERT_REQUIRED
ok
test_recv_send (__main__.ThreadedTests)
Test recv(), send() and friends. ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('127.0.0.1', 42881)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
ok
test_socketserver (__main__.ThreadedTests)
Using a SocketServer to create and manage SSL connections. ... 
 server (('127.0.0.1', 47545):47545 None):
   [28/Feb/2017 05:50:42] "GET /keycert.pem HTTP/1.1" 200 -
 client: read 17 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 44 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 40 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 43 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 25 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 49 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 5 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 1024 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
 client: read 666 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:47545>>'
ok
test_starttls (__main__.ThreadedTests)
Switching from clear text to encrypted and back again. ...  server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('127.0.0.1', 44667)

 client:  sending 'msg 1'...
 server: read 'msg 1' (unencrypted), sending back 'msg 1' (unencrypted)...
 client:  read 'msg 1' from server
 client:  sending 'MSG 2'...
 server: read 'MSG 2' (unencrypted), sending back 'msg 2' (unencrypted)...
 client:  read 'msg 2' from server
 client:  sending 'STARTTLS'...
 server: read STARTTLS from client, sending OK...
 client:  read 'OK\n' from server, starting TLS...
 client:  sending 'MSG 3'...
 server: read 'MSG 3' (encrypted), sending back 'msg 3' (encrypted)...
 client:  read 'msg 3' from server
 client:  sending 'msg 4'...
 server: read 'msg 4' (encrypted), sending back 'msg 4' (encrypted)...
 client:  read 'msg 4' from server
 client:  sending 'ENDTLS'...
 server: read ENDTLS from client, sending OK...
 client:  read 'OK\n' from server, ending TLS...
 client:  sending 'msg 5'...
 server: connection is now unencrypted...
 server: read 'msg 5' (unencrypted), sending back 'msg 5' (unencrypted)...
 client:  read 'msg 5' from server
 client:  sending 'msg 6'...
 server: read 'msg 6' (unencrypted), sending back 'msg 6' (unencrypted)...
 client:  read 'msg 6' from server
 client:  closing connection.
 server: client closed connection
ok
test_unreachable (__main__.ThreadedTests) ... ok

----------------------------------------------------------------------
Ran 25 tests in 1.384s

OK
Suite run: demux: platform-native, protocol: 10
test_ciphers (__main__.BasicSocketTests) ...  server:  new connection from ::1:(44795, 0L, 0L)
 server:  new connection from ::1:(43798, 0L, 0L)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657d70>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657c80>
ok
test_constants (__main__.BasicSocketTests) ... ok
test_dtls_openssl_version (__main__.BasicSocketTests) ... ok
test_refcycle (__main__.BasicSocketTests) ... ok
test_wrapped_unconnected (__main__.BasicSocketTests) ... ok
test_sslwrap_simple (__main__.BasicTests) ... ok
test_connect (__main__.NetworkedTests) ...  server:  new connection from ::1:(39232, 0L, 0L)
 server:  new connection from ::1:(48800, 0L, 0L)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657d70>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657e60>
ok
test_connect_ex (__main__.NetworkedTests) ...  server:  new connection from ::1:(46235, 0L, 0L)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657f50>
ok
test_get_server_certificate (__main__.NetworkedTests) ...  server:  new connection from ::1:(37789, 0L, 0L)
 server:  new connection from ::1:(58211, 0L, 0L)
 server:  new connection from ::1:(50237, 0L, 0L)

Verified certificate is
-----BEGIN CERTIFICATE-----
MIICDTCCAXYCCQCxc2uXBLZhDjANBgkqhkiG9w0BAQUFADBKMQswCQYDVQQGEwJV
UzETMBEGA1UECBMKV2FzaGluZ3RvbjETMBEGA1UEChMKUmF5IENBIEluYzERMA8G
A1UEAxMIUmF5Q0FJbmMwHhcNMTQwMTE4MjEwMjUwWhcNMjQwMTE2MjEwMjUwWjBM
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3RvbjEUMBIGA1UEChMLUmF5
IFNydiBJbmMxEjAQBgNVBAMTCVJheVNydkluYzCBnzANBgkqhkiG9w0BAQEFAAOB
jQAwgYkCgYEA2Mv6DsylMQHjRWjanFZvHfdjCBdDpBucuDVDIqWsfoMQSi3INFYj
9l+29ZuMZSHTNQaFBovMXdk2B0xWQXmPTAKBF3A+ifVy3nAlSDoYJa9+LgH8eVCt
aDwkNpaga4NxOouHoph8N1ZOVYIGrek5bXiBV7d3XZHwHG708IryOu8CAwEAATAN
BgkqhkiG9w0BAQUFAAOBgQBw0XUTYzfiI0Fi9g4GuyWD2hjET3NtrT4Ccu+Jiivy
EvwhzHtVGAPhrV+VCL8sS9uSOZlmfK/ZVraDiFGpJLDMvPP5y5fwq5VGrFuZispG
X6bTBq2AIKzGGXxhwPqD8F7su7bmZDnZFRMRk2Bh16rv0mtzx9yHtqC5YJZ2a3JK
2g==
-----END CERTIFICATE-----

 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657f50>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657b90>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657c08>
ok
test_makefile_close (__main__.NetworkedTests) ...  server:  new connection from ::1:(47568, 0L, 0L)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657d70>
ok
test_non_blocking_connect_ex (__main__.NetworkedTests) ...  server:  new connection from ::1:(51540, 0L, 0L)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657d70>
ok
test_non_blocking_handshake (__main__.NetworkedTests) ...  server:  new connection from ::1:(35735, 0L, 0L)

Needed 5 calls to do_handshake() to establish session.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657e60>
ok
test_asyncore_server (__main__.ThreadedTests)
Check the example asyncore integration. ... 
 server:  new connection from ::1:(38307, 0L, 0L)
 client:  sending 'TEST MESSAGE of mixed case\n'...
 client:  read 'test message of mixed case\n'
 client:  closing connection.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657b90>
ok
test_echo (__main__.ThreadedTests)
Basic test of an SSL client connecting to a server ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('::1', 54784, 0L, 0L)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
 client:  sending 'FOO\n'...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending bytearray(b'FOO\n')...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending <memory at 0x7f75ab6f12b0>...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  closing connection.
 server: client closed connection
ok
test_empty_cert (__main__.ThreadedTests)
Connecting with an empty cert file ... 
SSLError is [(151441516, 'error:0906D06C:PEM routines:PEM_read_bio:no start line'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_getpeercert (__main__.ThreadedTests) ... 
{'notAfter': 'Jan 16 21:02:50 2024 GMT',
 'subject': ((('countryName', u'US'),),
             (('stateOrProvinceName', u'Washington'),),
             (('organizationName', u'Ray Srv Inc'),),
             (('commonName', u'RaySrvInc'),))}
Connection cipher is ('AES256-SHA', 'TLSv1/SSLv3', 256).
ok
test_handshake_timeout (__main__.ThreadedTests) ... ok
test_malformed_cert (__main__.ThreadedTests)
Connecting with a badly formatted certificate (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_malformed_key (__main__.ThreadedTests)
Connecting with a badly formatted key (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_nonexisting_cert (__main__.ThreadedTests)
Connecting with a non-existing cert file ... 
SSLError is [(336602136, 'error:14102418:SSL routines:dtls1_read_bytes:tlsv1 alert unknown ca')]
ok
test_protocol_dtlsv1 (__main__.ThreadedTests)
Connecting to a DTLSv1 server with various client options ... 
 DTLSv1->DTLSv1 CERT_NONE
 DTLSv1->DTLSv1 CERT_OPTIONAL
 DTLSv1->DTLSv1 CERT_REQUIRED
ok
test_recv_send (__main__.ThreadedTests)
Test recv(), send() and friends. ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('::1', 52369, 0L, 0L)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
ok
test_socketserver (__main__.ThreadedTests)
Using a SocketServer to create and manage SSL connections. ... 
 server (('::1', 40180, 0, 0):40180 None):
   [28/Feb/2017 05:50:43] "GET /keycert.pem HTTP/1.1" 200 -
 client: read 17 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 44 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 40 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 43 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 25 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 49 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 5 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 1024 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
 client: read 666 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:40180>>'
ok
test_starttls (__main__.ThreadedTests)
Switching from clear text to encrypted and back again. ...  server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('::1', 35259, 0L, 0L)

 client:  sending 'msg 1'...
 server: read 'msg 1' (unencrypted), sending back 'msg 1' (unencrypted)...
 client:  read 'msg 1' from server
 client:  sending 'MSG 2'...
 server: read 'MSG 2' (unencrypted), sending back 'msg 2' (unencrypted)...
 client:  read 'msg 2' from server
 client:  sending 'STARTTLS'...
 server: read STARTTLS from client, sending OK...
 client:  read 'OK\n' from server, starting TLS...
 client:  sending 'MSG 3'...
 server: read 'MSG 3' (encrypted), sending back 'msg 3' (encrypted)...
 client:  read 'msg 3' from server
 client:  sending 'msg 4'...
 server: read 'msg 4' (encrypted), sending back 'msg 4' (encrypted)...
 client:  read 'msg 4' from server
 client:  sending 'ENDTLS'...
 server: read ENDTLS from client, sending OK...
 client:  read 'OK\n' from server, ending TLS...
 client:  sending 'msg 5'...
 server: connection is now unencrypted...
 server: read 'msg 5' (unencrypted), sending back 'msg 5' (unencrypted)...
 client:  read 'msg 5' from server
 client:  sending 'msg 6'...
 server: read 'msg 6' (unencrypted), sending back 'msg 6' (unencrypted)...
 client:  read 'msg 6' from server
 client:  closing connection.
 server: client closed connection
ok
test_unreachable (__main__.ThreadedTests) ... ok

----------------------------------------------------------------------
Ran 25 tests in 1.318s

OK
Suite run: demux: routing, protocol: 2
test_ciphers (__main__.BasicSocketTests) ...  server:  new connection from 127.0.0.1:(39094,)
 server:  new connection from 127.0.0.1:(44667,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab659398>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab659410>
ok
test_constants (__main__.BasicSocketTests) ... ok
test_dtls_openssl_version (__main__.BasicSocketTests) ... ok
test_refcycle (__main__.BasicSocketTests) ... ok
test_wrapped_unconnected (__main__.BasicSocketTests) ... ok
test_sslwrap_simple (__main__.BasicTests) ... ok
test_connect (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(54386,)
 server:  new connection from 127.0.0.1:(44855,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab659398>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657320>
ok
test_connect_ex (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(45064,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6572a8>
ok
test_get_server_certificate (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(48502,)
 server:  new connection from 127.0.0.1:(51168,)
 server:  new connection from 127.0.0.1:(34099,)

Verified certificate is
-----BEGIN CERTIFICATE-----
MIICDTCCAXYCCQCxc2uXBLZhDjANBgkqhkiG9w0BAQUFADBKMQswCQYDVQQGEwJV
UzETMBEGA1UECBMKV2FzaGluZ3RvbjETMBEGA1UEChMKUmF5IENBIEluYzERMA8G
A1UEAxMIUmF5Q0FJbmMwHhcNMTQwMTE4MjEwMjUwWhcNMjQwMTE2MjEwMjUwWjBM
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3RvbjEUMBIGA1UEChMLUmF5
IFNydiBJbmMxEjAQBgNVBAMTCVJheVNydkluYzCBnzANBgkqhkiG9w0BAQEFAAOB
jQAwgYkCgYEA2Mv6DsylMQHjRWjanFZvHfdjCBdDpBucuDVDIqWsfoMQSi3INFYj
9l+29ZuMZSHTNQaFBovMXdk2B0xWQXmPTAKBF3A+ifVy3nAlSDoYJa9+LgH8eVCt
aDwkNpaga4NxOouHoph8N1ZOVYIGrek5bXiBV7d3XZHwHG708IryOu8CAwEAATAN
BgkqhkiG9w0BAQUFAAOBgQBw0XUTYzfiI0Fi9g4GuyWD2hjET3NtrT4Ccu+Jiivy
EvwhzHtVGAPhrV+VCL8sS9uSOZlmfK/ZVraDiFGpJLDMvPP5y5fwq5VGrFuZispG
X6bTBq2AIKzGGXxhwPqD8F7su7bmZDnZFRMRk2Bh16rv0mtzx9yHtqC5YJZ2a3JK
2g==
-----END CERTIFICATE-----

 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6572a8>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657668>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657140>
ok
test_makefile_close (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(53862,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657398>
ok
test_non_blocking_connect_ex (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(40837,)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657398>
ok
test_non_blocking_handshake (__main__.NetworkedTests) ...  server:  new connection from 127.0.0.1:(41020,)

Needed 5 calls to do_handshake() to establish session.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6576e0>
ok
test_asyncore_server (__main__.ThreadedTests)
Check the example asyncore integration. ... 
 server:  new connection from 127.0.0.1:(60205,)
 client:  sending 'TEST MESSAGE of mixed case\n'...
 client:  read 'test message of mixed case\n'
 client:  closing connection.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6572a8>
ok
test_echo (__main__.ThreadedTests)
Basic test of an SSL client connecting to a server ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('127.0.0.1', 53059)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
 client:  sending 'FOO\n'...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending bytearray(b'FOO\n')...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending <memory at 0x7f75ab6f1348>...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  closing connection.
 server: client closed connection
ok
test_empty_cert (__main__.ThreadedTests)
Connecting with an empty cert file ... 
SSLError is [(151441516, 'error:0906D06C:PEM routines:PEM_read_bio:no start line'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_getpeercert (__main__.ThreadedTests) ... 
{'notAfter': 'Jan 16 21:02:50 2024 GMT',
 'subject': ((('countryName', u'US'),),
             (('stateOrProvinceName', u'Washington'),),
             (('organizationName', u'Ray Srv Inc'),),
             (('commonName', u'RaySrvInc'),))}
Connection cipher is ('AES256-SHA', 'TLSv1/SSLv3', 256).
ok
test_handshake_timeout (__main__.ThreadedTests) ... ok
test_malformed_cert (__main__.ThreadedTests)
Connecting with a badly formatted certificate (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_malformed_key (__main__.ThreadedTests)
Connecting with a badly formatted key (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_nonexisting_cert (__main__.ThreadedTests)
Connecting with a non-existing cert file ... 
SSLError is [(336602136, 'error:14102418:SSL routines:dtls1_read_bytes:tlsv1 alert unknown ca')]
ok
test_protocol_dtlsv1 (__main__.ThreadedTests)
Connecting to a DTLSv1 server with various client options ... 
 DTLSv1->DTLSv1 CERT_NONE
 DTLSv1->DTLSv1 CERT_OPTIONAL
 DTLSv1->DTLSv1 CERT_REQUIRED
ok
test_recv_send (__main__.ThreadedTests)
Test recv(), send() and friends. ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('127.0.0.1', 41716)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
ok
test_socketserver (__main__.ThreadedTests)
Using a SocketServer to create and manage SSL connections. ... 
 server (('127.0.0.1', 35227):35227 None):
   [28/Feb/2017 05:50:45] "GET /keycert.pem HTTP/1.1" 200 -
 client: read 17 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 44 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 40 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 43 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 25 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 49 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 5 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 1024 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
 client: read 666 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:35227>>'
ok
test_starttls (__main__.ThreadedTests)
Switching from clear text to encrypted and back again. ...  server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('127.0.0.1', 56421)

 client:  sending 'msg 1'...
 server: read 'msg 1' (unencrypted), sending back 'msg 1' (unencrypted)...
 client:  read 'msg 1' from server
 client:  sending 'MSG 2'...
 server: read 'MSG 2' (unencrypted), sending back 'msg 2' (unencrypted)...
 client:  read 'msg 2' from server
 client:  sending 'STARTTLS'...
 server: read STARTTLS from client, sending OK...
 client:  read 'OK\n' from server, starting TLS...
 client:  sending 'MSG 3'...
 server: read 'MSG 3' (encrypted), sending back 'msg 3' (encrypted)...
 client:  read 'msg 3' from server
 client:  sending 'msg 4'...
 server: read 'msg 4' (encrypted), sending back 'msg 4' (encrypted)...
 client:  read 'msg 4' from server
 client:  sending 'ENDTLS'...
 server: read ENDTLS from client, sending OK...
 client:  read 'OK\n' from server, ending TLS...
 server: connection is now unencrypted...
 client:  sending 'msg 5'...
 server: read 'msg 5' (unencrypted), sending back 'msg 5' (unencrypted)...
 client:  read 'msg 5' from server
 client:  sending 'msg 6'...
 server: read 'msg 6' (unencrypted), sending back 'msg 6' (unencrypted)...
 client:  read 'msg 6' from server
 client:  closing connection.
 server: client closed connection
ok
test_unreachable (__main__.ThreadedTests) ... ok

----------------------------------------------------------------------
Ran 25 tests in 1.633s

OK
Suite run: demux: routing, protocol: 10
test_ciphers (__main__.BasicSocketTests) ...  server:  new connection from ::1:(46227, 0, 0)
 server:  new connection from ::1:(58066, 0, 0)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab65d398>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab65d410>
ok
test_constants (__main__.BasicSocketTests) ... ok
test_dtls_openssl_version (__main__.BasicSocketTests) ... ok
test_refcycle (__main__.BasicSocketTests) ... ok
test_wrapped_unconnected (__main__.BasicSocketTests) ... ok
test_sslwrap_simple (__main__.BasicTests) ... ok
test_connect (__main__.NetworkedTests) ...  server:  new connection from ::1:(42005, 0, 0)
 server:  new connection from ::1:(42818, 0, 0)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab65d398>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab65d500>
ok
test_connect_ex (__main__.NetworkedTests) ...  server:  new connection from ::1:(60527, 0, 0)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab65d1b8>
ok
test_get_server_certificate (__main__.NetworkedTests) ...  server:  new connection from ::1:(43722, 0, 0)
 server:  new connection from ::1:(35637, 0, 0)
 server:  new connection from ::1:(37017, 0, 0)

Verified certificate is
-----BEGIN CERTIFICATE-----
MIICDTCCAXYCCQCxc2uXBLZhDjANBgkqhkiG9w0BAQUFADBKMQswCQYDVQQGEwJV
UzETMBEGA1UECBMKV2FzaGluZ3RvbjETMBEGA1UEChMKUmF5IENBIEluYzERMA8G
A1UEAxMIUmF5Q0FJbmMwHhcNMTQwMTE4MjEwMjUwWhcNMjQwMTE2MjEwMjUwWjBM
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3RvbjEUMBIGA1UEChMLUmF5
IFNydiBJbmMxEjAQBgNVBAMTCVJheVNydkluYzCBnzANBgkqhkiG9w0BAQEFAAOB
jQAwgYkCgYEA2Mv6DsylMQHjRWjanFZvHfdjCBdDpBucuDVDIqWsfoMQSi3INFYj
9l+29ZuMZSHTNQaFBovMXdk2B0xWQXmPTAKBF3A+ifVy3nAlSDoYJa9+LgH8eVCt
aDwkNpaga4NxOouHoph8N1ZOVYIGrek5bXiBV7d3XZHwHG708IryOu8CAwEAATAN
BgkqhkiG9w0BAQUFAAOBgQBw0XUTYzfiI0Fi9g4GuyWD2hjET3NtrT4Ccu+Jiivy
EvwhzHtVGAPhrV+VCL8sS9uSOZlmfK/ZVraDiFGpJLDMvPP5y5fwq5VGrFuZispG
X6bTBq2AIKzGGXxhwPqD8F7su7bmZDnZFRMRk2Bh16rv0mtzx9yHtqC5YJZ2a3JK
2g==
-----END CERTIFICATE-----

 server:  closed connection <ssl.SSLSocket object at 0x7f75ab65d1b8>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657cf8>
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6579b0>
ok
test_makefile_close (__main__.NetworkedTests) ...  server:  new connection from ::1:(43986, 0, 0)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6578c0>
ok
test_non_blocking_connect_ex (__main__.NetworkedTests) ...  server:  new connection from ::1:(56873, 0, 0)
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6578c0>
ok
test_non_blocking_handshake (__main__.NetworkedTests) ...  server:  new connection from ::1:(32892, 0, 0)

Needed 5 calls to do_handshake() to establish session.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab657e60>
ok
test_asyncore_server (__main__.ThreadedTests)
Check the example asyncore integration. ... 
 server:  new connection from ::1:(45079, 0, 0)
 client:  sending 'TEST MESSAGE of mixed case\n'...
 client:  read 'test message of mixed case\n'
 client:  closing connection.
 server:  closed connection <ssl.SSLSocket object at 0x7f75ab6578c0>
ok
test_echo (__main__.ThreadedTests)
Basic test of an SSL client connecting to a server ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('::1', 44341, 0, 0)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
 client:  sending 'FOO\n'...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending bytearray(b'FOO\n')...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  sending <memory at 0x7f75ab6f1348>...
 server: read 'FOO\n' (encrypted), sending back 'foo\n' (encrypted)...
 client:  read 'foo\n'
 client:  closing connection.
 server: client closed connection
ok
test_empty_cert (__main__.ThreadedTests)
Connecting with an empty cert file ... 
SSLError is [(151441516, 'error:0906D06C:PEM routines:PEM_read_bio:no start line'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_getpeercert (__main__.ThreadedTests) ... 
{'notAfter': 'Jan 16 21:02:50 2024 GMT',
 'subject': ((('countryName', u'US'),),
             (('stateOrProvinceName', u'Washington'),),
             (('organizationName', u'Ray Srv Inc'),),
             (('commonName', u'RaySrvInc'),))}
Connection cipher is ('AES256-SHA', 'TLSv1/SSLv3', 256).
ok
test_handshake_timeout (__main__.ThreadedTests) ... ok
test_malformed_cert (__main__.ThreadedTests)
Connecting with a badly formatted certificate (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_malformed_key (__main__.ThreadedTests)
Connecting with a badly formatted key (syntax error) ... 
SSLError is [(151441508, 'error:0906D064:PEM routines:PEM_read_bio:bad base64 decode'), (336445449, 'error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib')]
ok
test_nonexisting_cert (__main__.ThreadedTests)
Connecting with a non-existing cert file ... 
SSLError is [(336602136, 'error:14102418:SSL routines:dtls1_read_bytes:tlsv1 alert unknown ca')]
ok
test_protocol_dtlsv1 (__main__.ThreadedTests)
Connecting to a DTLSv1 server with various client options ... 
 DTLSv1->DTLSv1 CERT_NONE
 DTLSv1->DTLSv1 CERT_OPTIONAL
 DTLSv1->DTLSv1 CERT_REQUIRED
ok
test_recv_send (__main__.ThreadedTests)
Test recv(), send() and friends. ... 
 server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('::1', 52086, 0, 0)
 server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
ok
test_socketserver (__main__.ThreadedTests)
Using a SocketServer to create and manage SSL connections. ... 
 server (('::1', 53117, 0, 0):53117 None):
   [28/Feb/2017 05:50:46] "GET /keycert.pem HTTP/1.1" 200 -
 client: read 17 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 44 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 40 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 43 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 25 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 49 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 5 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 1024 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
 client: read 666 bytes from remote server '<SocketServerHTTPSServer <HTTPSServerUDP localhost:53117>>'
ok
test_starttls (__main__.ThreadedTests)
Switching from clear text to encrypted and back again. ...  server:  wrapped server socket as <ssl.SSLSocket object at 0x7f75ad5d77d0>
 server:  new connection from ('::1', 38231, 0, 0)

 client:  sending 'msg 1'...
 server: read 'msg 1' (unencrypted), sending back 'msg 1' (unencrypted)...
 client:  read 'msg 1' from server
 client:  sending 'MSG 2'...
 server: read 'MSG 2' (unencrypted), sending back 'msg 2' (unencrypted)...
 client:  read 'msg 2' from server
 client:  sending 'STARTTLS'...
 server: read STARTTLS from client, sending OK...
 client:  read 'OK\n' from server, starting TLS...
 client:  sending 'MSG 3'...
 server: read 'MSG 3' (encrypted), sending back 'msg 3' (encrypted)...
 client:  read 'msg 3' from server
 client:  sending 'msg 4'...
 server: read 'msg 4' (encrypted), sending back 'msg 4' (encrypted)...
 client:  read 'msg 4' from server
 client:  sending 'ENDTLS'...
 server: read ENDTLS from client, sending OK...
 client:  read 'OK\n' from server, ending TLS...
 server: connection is now unencrypted...
 client:  sending 'msg 5'...
 server: read 'msg 5' (unencrypted), sending back 'msg 5' (unencrypted)...
 client:  read 'msg 5' from server
 client:  sending 'msg 6'...
 server: read 'msg 6' (unencrypted), sending back 'msg 6' (unencrypted)...
 client:  read 'msg 6' from server
 client:  closing connection.
 server: client closed connection
ok
test_unreachable (__main__.ThreadedTests) ... ok

----------------------------------------------------------------------
Ran 25 tests in 1.641s

OK

I'm not really clear on what's not working for you, since unless you were using an old version of Python, the unit tests themselves were not working, and you didn't say what you did to try to get them to run anyway.

So perhaps you should pull the commit, and then try again. If you're still having trouble, then let me know your platform details and steps I should use to reproduce the issue. From the above it looks like OpenSSL 1.0.2g works fine on Linux without touching MTU.

Thank you, BTW, for breaking your patch into smaller pieces. This will make it much easier for me to apply your pull requests.

mcfreis commented 7 years ago

Hi, I'm using Python 2.7.6. The unittests ran fine from your master. But with OpenSSL 1.0.2j they dont any more, because the mtu size gets pulled down to 270 bytes and with this setting, the handshake doesn't work anymore. I'm on a Windows 7 64bit edition with Python and OpenSSL 32-bit.

Now I'm getting into a mess, we ise 2.7.6 and you just upgraded to 2.7.12 ... I'll continue on my branch and come to that migration problem later on.

Using OpenSSL with V1.0.2j in DTLSv1 mode:

Suite run: demux: platform-native, protocol: 2
.........s...............
Suite run: demux: platform-native, protocol: 23
----------------------------------------------------------------------
Ran 25 tests in 4.836s

OK (skipped=1)
.........s...............
----------------------------------------------------------------------
Ran 25 tests in 2.683s

OK (skipped=1)

Skipped is the one with the makefile test.

Regards, Björn

rbit / pydtls

Added support for EC-Functions and DTLSv1.2 (Updated OpenSSL-Lib to V1.0.2j) #7