Open danechitoaie opened 8 years ago
I don’t know how requests is structured, unfortunately. I think the question may be better suited to ask the requests maintainers. Specifically, if the transport layer is modular. oscrypto.tls
in its current implementation controls the raw socket connection and provides methods to read and write data.
I do have the intention of using oscrypto with Sublime Text in the future, however my use case for oscrypto.tls
would be FTPS or a very basic HTTP 1.1 implementation. Requests is big enough and complicated enough that I’d prefer not to become dependent on it. Additionally, I currently still support Python 2.6 for ST2, which many Python packages are starting to phase out.
The other downside is that oscrypto does not support Windows XP (about 1% of Package Control users) or OS X 10.6. OS X 10.6 is only supported by ST2, and is probably a very small percentage of users.
I guess I'll have to dig in the requests source and see what can be done. Thanks.
It looks like requests uses urllib3, which has a contrib module for using the cryptography package as an alternative to the ssl module. That may be a basis upon which to add oscrypto support. http://urllib3.readthedocs.org/en/latest/contrib.html
Certainly let me know what you find out. I don't really have time to do development related to this, but I'd be happy to answer questions and try to remove any blockers.
Yes, if I make any progress I'll let you know.
Howdy! @wbond mentioned this was a thing in IRC, and probably didn't realise that I'm the current urllib3 lead maintainer, so didn't think to ask me what would be needed.
The relevant example is this contrib module. It shims PyOpenSSL into an interface that looks a lot like the standard library's ssl module. urllib3 currently codes to that interface, so that's the interface you need to shim into.
The specific requirements are:
SSLContext
object that takes one argument (the equivalent of Python's SSLv23_METHOD). You can probably get away with ignoring this argument for libraries that don't have anything meaningfully similar to it.
SSLContext
object must have a wrap_socket
method that acts as a TLS socket factory. This needs to match the API the PyOpenSSL context provides. Note that you can assume that server_side
is always False
, do_handshake_on_connect
is always True
, and suppress_ragged_eofs
is always True
, as we do not ever set those differently.load_verify_locations
to throw an exception.load_cert_chain
to throw an exception.set_ciphers
is mandatory. Annoyingly, this API uses an OpenSSL cipher string, so you'll probably need to include a translation. Happily, urllib3 uses a very restricted OpenSSL cipher string in regular use, though our users can and do customise them.set_default_verify_paths
must be implemented, but needn't do anything.verify_mode
must be implemented, and must accept ssl.CERT_NONE
and ssl.CERT_REQUIRED
. Supporting ssl.CERT_OPTIONAL
is, ironically, optional.options
must be implemented, and must accept OpenSSL option flags. Again, urllib3 mostly uses a restricted subset here, so you can probably avoid supporting many OpenSSL option flags._decref_socketios
, _reuse
, _drop
, and makefile
from the PyOpenSSL wrapper.recv
, recv_into
, settimeout
, sendall
, shutdown
, and close
(though we never use shutdown
).getpeercert
, which must be able to support both providing the binary form of the peer certificate and the wacky Python-data-structure parsed form. See what PyOpenSSL does here for guidance. You will need to be able to provide at least the common name of the subject and the subject alternative name field in a form that the Python match_hostname
function can handle.send
and setblocking
. Both of these are likely to be used in urllib3 v2, which is being actively developed.Assuming you do all of these things, everything should just work. If you start working on this in a concrete way, please let me know: I'm happy to do code review and testing, and if it gets into a good shape I'd also be happy to merge a contrib module into urllib3. That would make it possible for Requests to use it.
@Lukasa Thanks for the extensive info! From reading over it, I believe that everything exists right now in order to be able to implement what you described. Hopefully when I have some free time for hacking I can take a pass at this.
Out of curiosity did anyone ever give this a try?
Requests I believe uses urllib3. Urllib3 now has a MacOS-specific backend that is derived from the code in oscrypto, even though it doesn’t use it directly.
I don’t believe it has a Windows (SecureChannel) backend, though. I may be wrong about that.
I don’t believe it has a Windows (SecureChannel) backend, though. I may be wrong about that.
I don't believe so either because we (the company I work for) hacked together our own SSL context that uses OpenSSL's CAPI Engine to be able to do Windows Cert authentication. Unfortunately this breaks with TLS 1.2+ due to CAPI being deprecated in favor of CNG.
I was wondering if this could be an alternative. I'll try and spend a little time seeing if I can reproduce the same sort of hacking we did but with oscrypto.
I have a proof of concept working of using oscrypto to wrap the socket in an sslcontext and then providing that context to requests and it working well on Windows.
I want to clean up the code and debug it a little bit before sharing.
In particular for some sites I am getting the TLSError: SECURITY_STATUS error 0x80090327: The parameter is incorrect.
Is there any debug mode I can set in oscrypto for it to give me more info than that?
Heh, welcome to the world of debugging Windows APIs! In my experience you’ll need to look up the preceding API call and try to deduce what isn’t working properly.
There are a number of tests in oscrypto for the TLS layer. If you can provide some basic steps to reproduce, I can see what I can find. Even knowing a public site the error occurs on with what version of Windows and Python would be helpful.
If you can provide some basic steps to reproduce, I can see what I can find. Even knowing a public site the error occurs on with what version of Windows and Python would be helpful.
Oh dear, this will probably never be reproducible publicly. These sites are using SSO with smart-card certificates and an internal CA. I'm on Windows 10 and using Python 3.9 to test, I think it's on TLS 1.2, and the error happens during handshake. I'll keep debugging on my end and hope for the best.
Anyway, here is my proof of concept code to get requests
to use oscrypto
. If anyone uses this be aware I am not well versed in SSL/TLS logic, I've just shoved together a few APIs and hoped it worked, mostly based on @Lukasa 's excellent comment.
import requests
import ipaddress
import socket as socket_
import requests.adapters
from ssl import SSLContext
from oscrypto._errors import pretty_message
from oscrypto._types import type_name, str_cls
from oscrypto.tls import TLSSocket, TLSSession
SSL_WRITE_BLOCKSIZE = 16385
def is_ip(ip):
try:
ipaddress.ip_address(ip)
except ValueError:
return False
else:
return True
class OsCryptoWrappedSocket:
"""API-compatibility wrapper"""
def __init__(self, sock, session, timeout=10):
if sock:
ip, port = sock.getpeername()
self.oscrypto_socket = TLSSocket(ip, port, timeout, session)
else:
self.oscrypto_socket = TLSSocket(None, None, timeout, session)
self.suppress_ragged_eofs = True
self._io_refs = 0
self._closed = False
def fileno(self) -> int:
return self.oscrypto_socket._socket.fileno()
def _decref_socketios(self) -> None:
if self._io_refs > 0:
self._io_refs -= 1
if self._closed:
self.close()
def recv(self, bufsize, flags=None) -> bytes:
return self.oscrypto_socket.read(bufsize)
def recv_into(self, buffer, nbytes=None, flags=None) -> int:
buffer_len = len(buffer)
if nbytes:
max_length = min(buffer_len, nbytes)
else:
max_length = buffer_len
read_bytes = self.oscrypto_socket.read(max_length)
if not read_bytes:
return 0
buffer[:len(read_bytes)] = read_bytes
return len(read_bytes)
def settimeout(self, timeout: float) -> None:
return self.oscrypto_socket._socket.settimeout(timeout)
def _send_until_done(self, data: bytes) -> int:
self.oscrypto_socket.write(data)
return len(data)
def sendall(self, data: bytes) -> None:
total_sent = 0
while total_sent < len(data):
sent = self._send_until_done(
data[total_sent : total_sent + SSL_WRITE_BLOCKSIZE]
)
total_sent += sent
def shutdown(self) -> None:
self.oscrypto_socket.shutdown()
def close(self) -> None:
if self._io_refs < 1:
try:
self._closed = True
return self.oscrypto_socket.close()
except Exception:
return
else:
self._io_refs -= 1
def getpeercert(self, binary_form: bool = False):
if not self.oscrypto_socket.certificate:
return self.oscrypto_socket.certificate
if binary_form:
return self.oscrypto_socket.certificate.dump()
return {
"subject": ((("commonName", self.oscrypto_socket.certificate.subject.native["common_name"]),),),
"subjectAltName": (("IP Address" if is_ip(v) else "DNS", v)
for v in self.oscrypto_socket.certificate.subject_alt_name_value.native),
}
def version(self):
return self.oscrypto_socket.protocol
@classmethod
def wrap(cls, socket, hostname, session=None):
if not isinstance(socket, socket_.socket):
raise TypeError(pretty_message(
'''
socket must be an instance of socket.socket, not %s
''',
type_name(socket)
))
if not isinstance(hostname, str_cls):
raise TypeError(pretty_message(
'''
hostname must be a unicode string, not %s
''',
type_name(hostname)
))
if session is not None and not isinstance(session, TLSSession):
raise TypeError(pretty_message(
'''
session must be an instance of oscrypto.tls.TLSSession, not %s
''',
type_name(session)
))
new_socket = cls(None, session=session)
new_socket.oscrypto_socket._socket = socket
new_socket.oscrypto_socket._hostname = hostname
new_socket.oscrypto_socket._handshake()
return new_socket
OsCryptoWrappedSocket.makefile = socket_.socket.makefile
class OSCryptoSSLContext(SSLContext):
def wrap_socket(self,
sock,
server_side=False,
do_handshake_on_connect=True,
suppress_ragged_eofs=True,
server_hostname=None,
session=None):
return OsCryptoWrappedSocket.wrap(sock, server_hostname, session)
class HTTPAdapterOSCrpyto(requests.adapters.HTTPAdapter):
def init_poolmanager(self, connections, maxsize, block=..., **pool_kwargs):
return super().init_poolmanager(
connections, maxsize, block=block, ssl_context=OSCryptoSSLContext(),
**pool_kwargs
)
with requests.Session() as session:
session.mount("https://", HTTPAdapterOSCrpyto())
response = session.get("https://www.bbc.co.uk/")
print(response.status_code)
If I figure out the handshake issue I am getting I will update this sample code.
Oh dear, this will probably never be reproducible publicly. These sites are using SSO with smart-card certificates and an internal CA. I'm on Windows 10 and using Python 3.9 to test, I think it's on TLS 1.2, and the error happens during handshake. I'll keep debugging on my end and hope for the best.
A colleague has tracked it down to failing when this function is called: https://docs.microsoft.com/en-us/windows/win32/api/sspi/nf-sspi-initializesecuritycontextw
So some progress, except it has over 10 parameters and no real way to debug them yet, aha. If you have any pointers they'd be appreciated, otherwise we'll very slowly debug when we have time.
So for anyone reading this I'm fairly sure my code in https://github.com/wbond/oscrypto/issues/10#issuecomment-953416175 does get requests to use oscrypto, but probably because of https://github.com/wbond/oscrypto/issues/4 it doesn't work for my particular use case.
Can this be used with requests? As replacement for the built in SSL lib (currently I'm referring to a Sublime Text 3 plugin context).