Open funnel20 opened 2 years ago
@Chavithra Can you please have a look into this, since it seems a serious stopper.
While analysing the source code of the Quotecast API, in particular fetch_data.py, it appears that the requests do not have a timeout specified:
This might cause the freeze of the API after 1 hour in case a timeout would occur.
Add the timeout
parameter to the send()
call, with a value of 14 seconds, so line 54 becomes:
response = session.send(request=prepped, timeout=14)
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/9f09a1b8-3e95-4ed9-a8c2-52bbaa39570d HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
WARNING:root:DeGiro Quotecast API failed to receive/parse server data with exception: HTTPSConnectionPool(host='degiro.quotecast.vwdservices.com', port=443): Read timed out. (read timeout=14)
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout.
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout.
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout.
Now that the API no longer freezes and throws the exception, it's possible to heal the issue by adding reconnect logic to the test script:
except TimeoutError:
logging.warning("DeGiro Quotecast API session did timeout. Reconnecting...")
# Reconnect and subscribe tickers:
quotecast_api.connect()
quotecast_api.subscribe(request=request)
Put these 2 together and finally after an hour a reconnect can be done and real-time data resumes: ✅
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/b26b871b-c7d5-421d-ba2c-71f948b3938c HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
WARNING:root:DeGiro Quotecast API failed to receive/parse server data with exception: HTTPSConnectionPool(host='degiro.quotecast.vwdservices.com', port=443): Read timed out. (read timeout=13)
WARNING:root:DeGiro Quotecast API returned an empty data frame.
WARNING:root:DeGiro Quotecast API session did timeout. Reconnecting...
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): degiro.quotecast.vwdservices.com:443
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "POST /CORS/request_session?version=1.0.20201211&userToken=2979984 HTTP/1.1" 200 52
INFO:degiro_connector.quotecast.actions.action_connect:get_session_id:response_dict: {'sessionId': '84819a6f-ee7e-4e20-81ac-decb829471d6'}
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
INFO:degiro_connector.quotecast.actions.action_subscribe:subscribe:data {"controlData":"a_req(612967.LastDate);a_req(612967.LastTime);a_req(612967.LastPrice);a_req(612967.L
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "POST /CORS/84819a6f-ee7e-4e20-81ac-decb829471d6 HTTP/1.1" 200 0
WARNING:root:DeGiro Quotecast API returned an empty data frame.
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/84819a6f-ee7e-4e20-81ac-decb829471d6 HTTP/1.1" 200 190
DEBUG:root:[{"m":"a_req","v":["612967.LastDate",2748]},{"m":"us","v":[2748,"2022-09-05"]},{"m":"a_req","v":["612967.LastTime",2747]},{"m":"us","v":[2747,"17:35:22"]},{"m":"a_req","v":["612967.LastPrice",2751]},{"m":"un","v":[2751,9.662000]},{"m":"a_req","v":["612967.LastVolume",783031]},{"m":"a_req","v":["612967.AskPrice",2750]},{"m":"un","v":[2750,9.720000]},{"m":"a_req","v":["612967.BidPrice",2749]},{"m":"un","v":[2749,9.640000]},{"m":"un","v":[783031,20938]}]
INFO:root: response_datetime request_duration vwd_id LastDate LastTime LastPrice AskPrice BidPrice LastVolume
0 2022-09-05T18:35:01.902509Z 1.117626 612967 1.662329e+09 63322.0 9.662 9.72 9.64 20938.0
DEBUG:degiro_connector.core.models.model_session:session:getter: MainThread
DEBUG:urllib3.connectionpool:https://degiro.quotecast.vwdservices.com:443 "GET /CORS/84819a6f-ee7e-4e20-81ac-decb829471d6 HTTP/1.1" 200 11
DEBUG:root:[{"m":"h"}]
WARNING:root:DeGiro Quotecast API returned an empty data frame.
timeout
parameter in the send()
method is part of the solution, the entire connector should be analysed for this usage. For example it's also missing in get_chart.py:
https://github.com/Chavithra/degiro-connector/blob/073646150cd72f8a09341df43b411d8fb6b96da9/degiro_connector/quotecast/actions/action_get_chart.py#L306timeout
parameter in the send()
method is part of the solution, what should the value be? I've now chosen for 14 sec, since the docu claims it times out at 15 sec.
Background
We have run our script successfully on a daily basis for the past months. It uses the Quotecast API to fetch the latest price data of a ticker every 1 second, during the 6.5 market hours of a Ney York trading day.
Issue
However, since August 30, 2022 we don't get any price updates anymore after 1 hour of the start of the Quotecast API.
The issue occurred on version 2.0.16 of
degiro-connector
that we still use. But I've tested that it also occurs on the latest 2.0.21.What I have tried:
connect()
on theQuotecastAPI
after 40 minutes, to somehow refresh or reset a timer. But no success; 20 minutes later (so 1 hour after the start) the API freezes and doesn't report any new data.Reproducible
You can use the test script below, to observe this behaviour in a reproducible way.
Test results
When starting the script, the following output is shown:
Let's make a mental note of the first time stamp
T09:27:01
.Every second the fetch is repeated, either responded by new quote data or an empty data frame. Let the script run for the next hour.
Notice that the last received update has time stamp
T10:26:27
(so nearly 1 hour), followed by a few empty data frames:Then the last DEBUG log
degiro_connector.core.models.model_session:session:getter: MainThread
is shown, which stays unanswered forever.Questions
Test script
The script is based on this example, extended with exception handling and debug log level. I've added 2 possible
vwd_id
s for tickers on both the European and North American market hours, so you get valid real-time data depending your testing time.Let the script run for 1 hour, and observe what happens after 1 hour (see section Test Results above).