bryanyang0528 / ksql-python

A python wrapper for the KSQL REST API.
MIT License
159 stars 67 forks source link

HTTP/2 Configurable Timeout to avoid socket.timeout error #120

Open Robbie-Palmer opened 1 year ago

Robbie-Palmer commented 1 year ago

I am getting a socket timeout error on calling inserts_stream, and I am unable to increase the default socket timeout The timeout parameter in KSQLAPI's constructor does not affect inserts_stream

timeout is used to construct a SimplifiedAPI object This is only used in the parent class BaseApi in its _request function, which uses HTTP/1 with urllib These HTTP/1 calls are used by the query and ksql functions

The timeout is not used by HTTP/2 calls The HTTP/2 calls depend on Hyper, which had an issue requesting support for timeouts, but it appears it wasn't addressed before the repo was deprecated This affects the inserts_stream function and query when use_http2=True

Due to hyper's deprecation it is advised to migrate to httpx which is discussed in #102 This was started but abandoned in #107 This was published to pypi as https://pypi.org/project/pykSQL/0.11.0/, but it doesn't work as calling inserts_stream causes a bad request from ksql saying '{"@type":"generic_error","error_code":40004,"message":"This endpoint is only available when using HTTP2"}'

If the migration to httpx is completed, then it is simple to pass the same timeout for HTTP/2 requests as httpx supports this response = httpx.post(url, data=row, headers=headers, timeout=self.timeout)

sheinbergon commented 1 month ago

Hello @Robbie-Palmer

I've forked the code, modernized it (replaced hyper with HTTPX, bumped all dependencies, updated to Python 3.9 as the minimal version, and fixed all the tests to behave with latest KSQL DB versions (0.29.0). Could you check if it's still present in my version?

https://github.com/sheinbergon/ksql-python-ng

I'll be making a PYPI release soon enough. In the meanwhile, you can test it by running

pipx install git+https://github.com/sheinbergon/ksql-python-ng.git --include-deps

or

pip install git+https://github.com/sheinbergon/ksql-python-ng.git