search

trinodb / trino-python-client

Python client for Trino
Apache License 2.0
309 stars 151 forks source link

ResourceWarning: unclosed ssl.SSLSocket #294

Closed aksakalli closed 1 year ago

aksakalli commented 1 year ago

Expected behavior

I am running an unittest that executes a list of queries atomically (creating a new connection instance every time) through a HTTPS connection. Some function calls raises ResourceWarning: unclosed <ssl.SSLSocket .... I expect Trino to close all existing SSL sockets in conn.close() and cur.close(). I tried both context manager and explicit close calls, the result is the same.

conn = connect(
  host="trino.mydomain.net",
  port=443,
  http_scheme="https",
  catalog="hive",
  ...
)
cur = conn.cursor()
cur.execute('SELECT * FROM hive.myschema.mytable')
rows = cur.fetchall()
cur.close()
conn.close()

or

with connect(...) as conn:
    cur = conn.cursor()
    cur.execute('SELECT * FROM hive.myschema.mytable')
    rows = cur.fetchall()

Actual behavior

Receiving ResourceWarning: unclosed <ssl.SSLSocket ... warning during python -m unittest discover.

Steps To Reproduce

Initially, I wanted to write an integration test method under trino-python-client/tests/integration to reproduce it but the connection to the docker container are not using SSL.

create a SSL connection to a Trino host and loop through a list of queries:

import tracemalloc
import unittest

from src.my_model import my_execute_method

class TestMyModule(unittest.TestCase):

  def test_queries(self):
    tracemalloc.start(25)
    for q in queries:
      my_execute_method(q)

my_execute_method fetches the results as explained previously.

Log output

I enabled tracemalloc and here is the output:

/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/requests/structures.py:58: ResourceWarning: unclosed <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.0.146', 57287), raddr=('6.6.6.6', 443)>
  return (casedkey for casedkey, mappedvalue in self._store.values())
Object allocated at (most recent call last):
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/unittest/suite.py", lineno 84
    return self.run(*args, **kwds)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/unittest/suite.py", lineno 122
    test(result)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/unittest/case.py", lineno 650
    return self.run(*args, **kwds)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/unittest/case.py", lineno 591
    self._callTestMethod(testMethod)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/unittest/case.py", lineno 549
    method()
  File "/Users/caksakalli/projects/fsl-access-manager/tests/test_dpr_parser.py", lineno 48
    print(f"{t}: {table_exists(t)}")
  File "/Users/caksakalli/projects/fsl-access-manager/src/helpers.py", lineno 104
    res = trino_execute(
  File "/Users/caksakalli/projects/fsl-access-manager/src/helpers.py", lineno 83
    cur.execute(query, params=params)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/trino/dbapi.py", lineno 415
    self._prepare_statement(operation, statement_name)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/trino/dbapi.py", lineno 315
    query.execute()
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/trino/client.py", lineno 743
    response = self._request.post(self._sql, additional_http_headers)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/trino/client.py", lineno 506
    http_response = self._post(
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/trino/client.py", lineno 822
    result = func(*args, **kwargs)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/requests/sessions.py", lineno 635
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/requests/sessions.py", lineno 587
    resp = self.send(prep, **send_kwargs)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/requests/sessions.py", lineno 701
    r = adapter.send(request, **kwargs)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/requests/adapters.py", lineno 489
    resp = conn.urlopen(
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/urllib3/connectionpool.py", lineno 703
    httplib_response = self._make_request(
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/urllib3/connectionpool.py", lineno 386
    self._validate_conn(conn)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/urllib3/connectionpool.py", lineno 1042
    conn.connect()
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/urllib3/connection.py", lineno 414
    self.sock = ssl_wrap_socket(
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/urllib3/util/ssl_.py", lineno 449
    ssl_sock = _ssl_wrap_socket_impl(
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/site-packages/urllib3/util/ssl_.py", lineno 493
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/ssl.py", lineno 513
    return self.sslsocket_class._create(
  File "/Users/caksakalli/miniconda3/envs/fsl/lib/python3.10/ssl.py", lineno 1034
    self = cls.__new__(cls, **kwargs)

I masked raddr.

Operating System

macos

Trino Python client version

0.319.0

Trino Server version

380

Python version

3.10

Are you willing to submit PR?