search

google / timesketch

Collaborative forensic timeline analysis
Apache License 2.0
2.52k stars 577 forks source link

Timeout with API Client for retrieving sketch information after upgrading both Timesketch server and API Client #3018

Open clowe-r7 opened 5 months ago

clowe-r7 commented 5 months ago

Describe the bug After upgrading to the latest version of Timesketch and the Timesketch API Client, a request fails when attempting to get information about a particular Sketch. The request ends up failing here with a timeout and a 308 status code using the resource uri defined at init.

To Reproduce Steps to reproduce the behavior:

  1. Spin up a Timesketch server using the latest version (I upgraded a currently running Timesketch instance. The request worked on a local development Timesketch server using the docker compose file in the docker/dev directory)
  2. Run the following script as described in the Timesketch API Client documentation 
    print(f"Listing Sketches")
sketches = client.list_sketches()
for sketch in sketches:
print('[{0:d}] {1:s} <{2:s}>'.format(sketch.id, sketch.name, sketch.description))

Expected behavior A list of all sketches with the id, name and description printed out.

Desktop (please complete the following information):

Additional context I was able to fix this in my local virtual environment by appending a forward slash to the sketch resource uri at init of the class here so instead of

super().__init__(api=api, resource_uri=f"sketches/{self.id}")

it became

super().__init__(api=api, resource_uri=f"sketches/{self.id}/")

and the request succeeded.

Full stacktrace

Traceback (most recent call last):
  File "/venv/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/venv/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.9/3.9.18_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/venv/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/venv/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x11c4e8100>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='https://timesketch.myowndomain.com', port=80): Max retries exceeded with url: /api/v1/sketches/1/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11c4e8100>: Failed to establish a new connection: [Errno 60] Operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/timesketch.py", line 21, in <module>
    print('[{0:d}] {1:s} <{2:s}>'.format(sketch.id, sketch.name, sketch.description))
  File "/venv/lib/python3.9/site-packages/timesketch_api_client/sketch.py", line 112, in description
    sketch = self.lazyload_data()
  File "/venv/lib/python3.9/site-packages/timesketch_api_client/resource.py", line 44, in lazyload_data
    self.resource_data = self.api.fetch_resource_data(self.resource_uri)
  File "/venv/lib/python3.9/site-packages/timesketch_api_client/client.py", line 376, in fetch_resource_data
    response = self.session.get(resource_url, params=params)
  File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 677, in send
    history = [resp for resp in gen]
  File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 677, in <listcomp>
    history = [resp for resp in gen]
  File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
    resp = self.send(
  File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/venv/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='https://timesketch.myowndomain.com', port=80): Max retries exceeded with url: /api/v1/sketches/1/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11c4e8100>: Failed to establish a new connection: [Errno 60] Operation timed out'))
clowe-r7 commented 5 months ago

Turns out this is an issue with the interaction between an AWS ALB, the Flask app running behind it and I think the werkzeug upgrade. Essentially the ALB is using HTTPs and the Flask app responded with a redirect but for HTTP. I was able to follow the fix on this reddit post https://www.reddit.com/r/flask/comments/11luchr/redirect_is_using_http_instead_of_https/ to get it working again without modifying the API Client. I added app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1, x_port=1) here and it's working again as expected.