Quotecast API freezes after exactly 1 hour since August 30, 2022

Background

We have run our script successfully on a daily basis for the past months. It uses the Quotecast API to fetch the latest price data of a ticker every 1 second, during the 6.5 market hours of a Ney York trading day.

Issue

However, since August 30, 2022 we don't get any price updates anymore after 1 hour of the start of the Quotecast API.

The issue occurred on version 2.0.16 of degiro-connector that we still use. But I've tested that it also occurs on the latest 2.0.21.

What I have tried:

Login in DeGiro web portal with the same account, before starting the script. Keep the web session active while the script is running, by clicking on some links every now and then. No difference; even if the portal is still active after 1 hour, the script freezes.
Call connect() on the QuotecastAPI after 40 minutes, to somehow refresh or reset a timer. But no success; 20 minutes later (so 1 hour after the start) the API freezes and doesn't report any new data.

Reproducible

You can use the test script below, to observe this behaviour in a reproducible way.

Test results

When starting the script, the following output is shown:

DEBUG:degiro_connector.quotecast.api:setup_one_action : connect
DEBUG:degiro_connector.quotecast.api:setup_one_action : fetch_data
DEBUG:degiro_connector.quotecast.api:setup_one_action : get_chart
DEBUG:degiro_connector.quotecast.api:setup_one_action : subscribe
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): degiro.quotecast.vwdservices.com:443
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "POST /CORS/request_session?version=1.0.20201211&userToken=2979984 HTTP/1.1" 200 52
INFO:degiro_connector.quotecast.actions.action_connect:get_session_id:response_dict: {'sessionId': '99989670-027b-4c3d-a4c7-ad5ccfc88d2c'}
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
INFO:degiro_connector.quotecast.actions.action_subscribe:subscribe:data {"controlData":"a_req(612967.LastDate);a_req(612967.LastTime);a_req(612967.LastPrice);a_req(612967.L
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "POST /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 0
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 179
DEBUG:root:[{"m":"a_req","v":["612967.LastDate",493]},{"m":"us","v":[493,"2022-09-02"]},{"m":"a_req","v":["612967.LastTime",492]},{"m":"a_req","v":["612967.LastPrice",496]},{"m":"a_req","v":["612967.LastVolume",405982]},{"m":"a_req","v":["612967.AskPrice",495]},{"m":"a_req","v":["612967.BidPrice",494]},{"m":"un","v":[495,9.680000]},{"m":"un","v":[494,9.672000]},{"m":"un","v":[405982,100]},{"m":"un","v":[496,9.672000]},{"m":"us","v":[492,"11:26:59"]}]
INFO:root:             response_datetime  request_duration  vwd_id      LastDate  AskPrice  BidPrice  LastVolume  LastPrice  LastTime
0  2022-09-02T09:27:01.848645Z          1.033442  612967  1.662070e+09      9.68     9.672       100.0      9.672   41219.0
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.

Let's make a mental note of the first time stamp T09:27:01.

Every second the fetch is repeated, either responded by new quote data or an empty data frame. Let the script run for the next hour.

Notice that the last received update has time stamp T10:26:27 (so nearly 1 hour), followed by a few empty data frames:

DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 87
DEBUG:root:[{"m":"un","v":[495,9.672000]},{"m":"un","v":[405982,600]},{"m":"un","v":[496,9.670000]},{"m":"us","v":[492,"12:26:25"]}]
INFO:root:             response_datetime  request_duration  vwd_id  AskPrice  LastVolume  LastPrice  LastTime
0  2022-09-02T10:26:27.001690Z          3.045704  612967     9.672       600.0       9.67   44785.0
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/99989670-027b-4c3d-a4c7-ad5ccfc88d2c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread

Then the last DEBUG log degiro_connector.core.models.model_session:session:getter: MainThread is shown, which stays unanswered forever.

Questions

What is happening here, why is the API freezing?
How to resolve this?

Test script

The script is based on this example, extended with exception handling and debug log level. I've added 2 possible vwd_ids for tickers on both the European and North American market hours, so you get valid real-time data depending your testing time.

Let the script run for 1 hour, and observe what happens after 1 hour (see section Test Results above).

# Based on: https://github.com/Chavithra/degiro-connector/blob/main/examples/quotecast/realtime_poller.py

# IMPORTATIONS
import json
import logging
import pandas as pd
import time

from degiro_connector.quotecast.api import API as QuotecastAPI
from degiro_connector.quotecast.models.quotecast_parser import QuotecastParser
from degiro_connector.quotecast.models.quotecast_pb2 import Quotecast

# SETUP LOGGING
logging.basicConfig(level=logging.DEBUG)

# SETUP CONFIG DICT
with open("config/config.json") as config_file:
    config_dict = json.load(config_file)

# SETUP CREDENTIALS
user_token = config_dict.get("user_token")  # HERE GOES YOUR USER_TOKEN

# SETUP API
quotecast_api = QuotecastAPI(user_token=user_token)

# CONNECTION
quotecast_api.connect()

# SUBSCRIBE TO METRICS
request = Quotecast.Request()

# Subscribe a ticker, e.g. "612967" (ABN AMRO) during European market hours, or "AAPL.BATS,E" (Apple Inc.) during North American market hours:
request.subscriptions["612967"].extend(
    [
        "LastDate",
        "LastTime",
        "LastPrice",
        "LastVolume",
        "AskPrice",
        "BidPrice",
    ]
)
quotecast_api.subscribe(request=request)

# SETUP JSON PARSER
quotecast_parser = QuotecastParser()

while True:
    try:
        # FETCH DATA
        ticker_df = pd.DataFrame()
        try:
            quotecast = quotecast_api.fetch_data()

            # DISPLAY RAW JSON
            logging.debug(quotecast.json_data)

            # Get PANDAS.DATAFRAME
            quotecast_parser.put_quotecast(quotecast=quotecast)
            ticker_df = quotecast_parser.ticker_df
        except TimeoutError:
            logging.warning("DeGiro Quotecast API session did timeout.")
        except Exception as exception:
            logging.warning("DeGiro Quotecast API failed to receive/parse server data with exception: {0}".format(exception))

        # Parse DataFrame when NOT empty:
        if not ticker_df.empty:
            logging.info(ticker_df)
        else:
            logging.warning("DeGiro Quotecast API returned an empty data frame.")

        # REMOVE THIS LINE TO RUN IT IN LOOP
        # USE : CTRL+C TO QUIT
        # break

        # Fetch every second:
        time.sleep(1)

    except Exception as e:
        logging.warning("DeGiro Quotecast API failed with exception: {0}".format(e))
        break
    except KeyboardInterrupt:
        logging.info("KeyboardInterrupt")
        break

Analysis

While analysing the source code of the Quotecast API, in particular fetch_data.py, it appears that the requests do not have a timeout specified:

https://github.com/Chavithra/degiro-connector/blob/073646150cd72f8a09341df43b411d8fb6b96da9/degiro_connector/quotecast/actions/action_fetch_data.py#L54

This might cause the freeze of the API after 1 hour in case a timeout would occur.

Change

Add the timeout parameter to the send() call, with a value of 14 seconds, so line 54 becomes:

response = session.send(request=prepped, timeout=14)

Results

When running the above test script, now after an hour the exception is thrown: ✅

DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/9f09a1b8-3e95-4ed9-a8c2-52bbaa39570d HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
WARNING:root:DeGiro Quotecast API failed to receive/parse server data with exception: HTTPSConnectionPool(host='degiro.quotecast.vwdservices.com', port=443): Read timed out. (read timeout=14)
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout.
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout.
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout.

Now that the API no longer freezes and throws the exception, it's possible to heal the issue by adding reconnect logic to the test script:

except TimeoutError:
logging.warning("DeGiro Quotecast API session did timeout. Reconnecting...")

# Reconnect and subscribe tickers:
quotecast_api.connect()
quotecast_api.subscribe(request=request)

Put these 2 together and finally after an hour a reconnect can be done and real-time data resumes: ✅

DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/b26b871b-c7d5-421d-ba2c-71f948b3938c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
WARNING:root:DeGiro Quotecast API failed to receive/parse server data with exception: HTTPSConnectionPool(host='degiro.quotecast.vwdservices.com', port=443): Read timed out. (read timeout=13)
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout. Reconnecting...
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): degiro.quotecast.vwdservices.com:443
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "POST /CORS/request_session?version=1.0.20201211&userToken=2979984 HTTP/1.1" 200 52
INFO:degiro_connector.quotecast.actions.action_connect:get_session_id:response_dict: {'sessionId': '84819a6f-ee7e-4e20-81ac-decb829471d6'}
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
INFO:degiro_connector.quotecast.actions.action_subscribe:subscribe:data {"controlData":"a_req(612967.LastDate);a_req(612967.LastTime);a_req(612967.LastPrice);a_req(612967.L
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "POST /CORS/84819a6f-ee7e-4e20-81ac-decb829471d6 HTTP/1.1" 200 0
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/84819a6f-ee7e-4e20-81ac-decb829471d6 HTTP/1.1" 200 190
DEBUG:root:[{"m":"a_req","v":["612967.LastDate",2748]},{"m":"us","v":[2748,"2022-09-05"]},{"m":"a_req","v":["612967.LastTime",2747]},{"m":"us","v":[2747,"17:35:22"]},{"m":"a_req","v":["612967.LastPrice",2751]},{"m":"un","v":[2751,9.662000]},{"m":"a_req","v":["612967.LastVolume",783031]},{"m":"a_req","v":["612967.AskPrice",2750]},{"m":"un","v":[2750,9.720000]},{"m":"a_req","v":["612967.BidPrice",2749]},{"m":"un","v":[2749,9.640000]},{"m":"un","v":[783031,20938]}]
INFO:root:             response_datetime  request_duration  vwd_id      LastDate  LastTime  LastPrice  AskPrice  BidPrice  LastVolume
0  2022-09-05T18:35:01.902509Z          1.117626  612967  1.662329e+09   63322.0      9.662      9.72      9.64     20938.0
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/84819a6f-ee7e-4e20-81ac-decb829471d6 HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.

Discussion

What is causing the timeout after exactly 1 hour?
This feels as an initial work-around, since part of the solution (the reconnect) has to be implemented in the client code. Ideally, the solution should be entirely implemented in the connector. Can we prevent the timeout at all?
If implementing the timeout parameter in the send() method is part of the solution, the entire connector should be analysed for this usage. For example it's also missing in get_chart.py: https://github.com/Chavithra/degiro-connector/blob/073646150cd72f8a09341df43b411d8fb6b96da9/degiro_connector/quotecast/actions/action_get_chart.py#L306
If implementing the timeout parameter in the send() method is part of the solution, what should the value be? I've now chosen for 14 sec, since the docu claims it times out at 15 sec.

Chavithra / degiro-connector