Lightstreamer exception with invalid literal for int() with base 10: '' and stops receiving data

[marcipops]

Reopening this issue as it was closed without resolution.

Please refer to: https://github.com/ig-python/ig-markets-api-python-library/issues/200

Lightstreamer.py throws an exception and stops responding to new data after a few days of running successfully.

I am using version 3 and streaming TICK data.

This is the error produced:

Exception in thread STREAM-CONN-THREAD-0: Traceback (most recent call last): File "/home/marc/miniconda3/envs/marc/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/home/marc/miniconda3/envs/marc/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "/home/marc/miniconda3/envs/marc/lib/python3.9/site-packages/trading_ig/lightstreamer.py", line 381, in _receive self._forward_update_message(message) File "/home/marc/miniconda3/envs/marc/lib/python3.9/site-packages/trading_ig/lightstreamer.py", line 325, in forward update_message table, item = int(tok[0]), tok[1] ValueError: invalid literal for int() with base 10: ''

I have found it only does this over a weekend, i.e. when MARKETSTATE is not TRADEABLE.

Is it possible to put in some code before line 325, to check the validity of tok[0] and manage any errors? Many thanks.

[bug-or-feature]

Please create a PR

[dario-weswit]

Since the error may be caused by unexpected data received, it seems difficult to put a workaround by fixing the data. If anything, you could catch the exception, identify that it is of this type, and discard the current connection and open a new one.

As Lightstreamer Support, we would like to see the actual data. The error may be, for instance, the consequence of some wrong decoding of the previous line. This requires a DEBUG log. Is it possible for you to collect the log in some rolling fashion, so that you don't have to keep days of log but only the last 100 lines?

[zansibal]

If anything, you could catch the exception, identify that it is of this type, and discard the current connection and open a new one.

This is how I have done it (for years).

[bug-or-feature]

If anything, you could catch the exception, identify that it is of this type, and discard the current connection and open a new one. This is how I have done it (for years).

Could you create a PR so other people can benefit? Or share a code snippet, and I'll do it

[zansibal]

Sorry, I misread. I confused this issue with a similar problem when BID or OFR contains empty strings in the stream.

I close down the connection over weekends to avoid this (here) issue. Not very helpful, I know.

[galburn]

I've hit this same issue last week. Unfortunately I let the debug log file get too large for me to open (23Gb), so I cannot see the preceding "Received update message..." message to see what is in update_message. Can anybody say what data is causing the issue? I'll devise a simple catch of the exception and create a PR for it. That should, at least, allow the process to continue executing.

[dario-weswit]

Could you manage to extract something from the log file using some lightweight tool? For instance "grep -n" to find the error line and then "head" and "tail" to extract a few hundred lines before the issue?

[galburn]

The log messages just before the event are:- 2022-07-15 23:11:48,306 Received message ---> <> 2022-07-15 23:11:48,306 Received update message ---> <>

There is already a test for a "None" message, but not for a zero length string. I've added a simple test for that into lightstreamer.py around line 347:- if message is None: receive = False log.warning("No new message received") elif message == "":

Skip a sporadic zero length string message

            log.debug("Null message")

This is currently under test with an IG demo & live account. Hopefully the new code will be exercised late Friday evening.

[dario-weswit]

From Lightstreamer point of view, there is no explanation for this empty message. Since the message is read by a "readline", it should correspond to an empty line coming from the Server. This is unexpected, so we cannot assume that it is a "sporadic zero length string messsage", but it would be a bugged input from the Server with no guarantee on what will come next. However, it would be difficult to trace exactly what arrives on the wire. Is it possible that this empty line is another way in which the "readline" notifies an interruption of the stream? We hope that the ongoing tests can shed more light on what happens after you skip the empty message.

[galburn]

Results from the logfiles (edited for brevity):-

---

LIVE server with BST timezone ... 2022-07-19 09:19:07,599 Creating new v2 session for user 'xxxxxx' at 'https://api.ig.com/gateway/deal' 2022-07-19 09:19:07,599 Starting new HTTPS connection (1): api.ig.com:443 2022-07-19 09:19:07,880 https://api.ig.com:443 "POST /gateway/deal/session HTTP/1.1" 200 361 2022-07-19 09:19:07,880 POST '/session', resp 200 2022-07-19 09:19:07,880 Starting connection with https://apd.marketdatasystems.com 2022-07-19 09:19:08,052 Waiting for a new message 2022-07-19 09:19:08,177 Server response ---> 2022-07-19 09:19:08,177 422 StreamStart subscribe finished 2022-07-19 09:19:08,193 Received message ---> <1,4|6224.9|6226.7|1658218747365> 2022-07-19 09:19:08,193 Received update message ---> <1,4|6224.9|6226.7|1658218747365> 2022-07-22 21:59:58,796 Waiting for a new message 2022-07-22 21:59:58,937 Received message ---> <1,10|#|#|#> 2022-07-22 21:59:58,937 Received update message ---> <1,10|#|#|#> 2022-07-22 21:59:58,968 Waiting for a new message 2022-07-22 21:59:58,968 Received message ---> <1,3|||> 2022-07-22 21:59:58,968 Received update message ---> <1,3|||> 2022-07-22 23:30:51,750 Waiting for a new message 2022-07-22 23:30:56,765 Received message ---> 2022-07-22 23:30:56,765 PROBE message 2022-07-22 23:30:56,765 Waiting for a new message ... No further messages received ... ig_stream_service.unsubscribe_all() 2022-07-23 10:26:01,368 Server response ---> 2022-07-23 10:26:01,368 Server error ... ig_stream_service.disconnect() 2022-07-23 10:26:01,477 Server response ---> 2022-07-23 10:26:01,477 Server error

Demo server with BST timezone... ... similar startup lines as the LIVE server 2022-07-22 21:59:58,406 Waiting for a new message 2022-07-22 21:59:58,750 Received message ---> <1,10|13171.3|13178.3|1658523599847> 2022-07-22 21:59:58,750 Received update message ---> <1,10|13171.3|13178.3|1658523599847> 2022-07-22 21:59:58,921 Received message ---> <1,10|#|#|#> 2022-07-22 21:59:58,921 Received update message ---> <1,10|#|#|#> 2022-07-22 21:59:58,984 Waiting for a new message 2022-07-22 21:59:59,015 Received message ---> <1,9|||> 2022-07-22 21:59:59,546 Waiting for a new message 2022-07-22 22:06:16,437 Received message ---> 2022-07-22 22:06:16,437 PROBE message 2022-07-22 22:06:16,437 Waiting for a new message 2022-07-22 22:06:19,312 Received message ---> <> 2022-07-22 22:06:19,312 Null message 2022-07-22 22:06:19,312 Waiting for a new message 2022-07-22 22:06:19,312 Received message ---> <> 2022-07-22 22:06:19,312 Null message ... millions of identical messages 2022-07-23 10:25:54,883 Waiting for a new message 2022-07-23 10:25:54,883 Received message ---> <> 2022-07-23 10:25:54,899 Waiting for a new message 2022-07-23 10:25:54,899 Received message ---> <> 2022-07-23 10:25:54,899 Null message 2022-07-23 10:25:54,899 Null message ... Out of disk space for logfile

---

This is my theory which is pure conjecture... IG are shutting down the LIVE server around 23:30 in a non-graceful way without letting the lightstreamer clients know. Hence the SYNC error when I attempt un unsubscribe/disconnect. They are also shutting down the demo server around 22:06, but for some reason this is producing millions of null messages in lightstreamer.py. No idea why, but if a graceless shutdown is performed unpredictable results are to be expected.

I will be amending my code to shutdown the Lightstreamer connection shortly after the Fri 22:00 end-of-week so as to avoid the issue that I observed this week.

This does not address the original bug report, so I will look out for these null messages arriving during the trading week. I'll try to think of some method of preventing millions of messages filling my log drive. Suggestions welcome.

[RRMXkun]

@galburn this is useful thanks for sharing. I think it would be trivial to check whether the content that is being received is empty, and raise an exception if so. That way users can handle the exception and decide what to do (try to reconnect, exit, etc.). Meanwhile IG should be made aware that their maintenance procedure does not cater for Lightstreamer shutdown.

[dario-weswit]

Note that Lightstreamer Server has no strong requirement for shutdown. Although an ordered closure is better than a process and/or connections abrupt interruption, the clients should be able to cope with any type of connection interruption. In this case, it seems to me that it is just python's "readline" which behaves in an inconsistent way, by reporting an empty line in a loop upon connection interruption. But I'm not familiar with python, so mine is just a hint, not a claim.

[marcipops]

Using IG's demo server, always over a weekend (Fri-Sun) but interestingly not every weekend, my client invariably goes "to sleep forever" whilst waiting for new data to arrive.

I've managed to trap the latest error from last Friday (2022-09-02 22:09:02 with info below).

Following previous advice above, I have created an error log attached, using the statement: logging.basicConfig(filename='MH_Error.log', level=logging.ERROR, format='%(asctime)s: %(funcName)s(): %(message)s')

For brevity, I have left out everything before 2022-09-02 21:50:00,018.

Anomalies?: 2022-09-02 22:07:06,219: _receive(): Received message ---> <1,1|$|$|$|$|$|$|$|$|$||$> 2022-09-02 22:24:18,337: _receive(): Received message ---> <1,1|1662153840000||||||||||0> 2022-09-02 22:43:36,093: _receive(): Received message ---> <>

My client logs nothing past this point, and waits forever for data. Essentially I have to kill the process and restart application.

Coinciding with the last received message, I also have an exception message at 2022-09-22 22:43:

Exception in thread STREAM-CONN-THREAD-0: Traceback (most recent call last): File "/home/marc/miniconda3/envs/marc/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/home/marc/miniconda3/envs/marc/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "/home/marc/miniconda3/envs/marc/lib/python3.9/site-packages/trading_ig/lightstreamer.py", line 381, in _receive self._forward_update_message(message) File "/home/marc/miniconda3/envs/marc/lib/python3.9/site-packages/trading_ig/lightstreamer.py", line 325, in _forward_update_message table, item = int(tok[0]), tok[1] ValueError: invalid literal for int() with base 10: ''

It looks identical to the error message I originally reported when I reopened this issue - but this time I am not collecting TICK data but 1 minute data - at least there seems to be something consistent.

Anway, following advice from dario-weswit (https://github.com/ig-python/ig-markets-api-python-library/issues/252#issuecomment-1153585774) I've tried to catch the exception.

FYI a fragment of my main function relevant to exception handling is:

subscription_prices.addlistener(on_prices_update)              # Adding the "on_prices_update" function to Subscription
#-------------------------------------------------------------------
ig_rest_service = IGService(username, password, api_key, acc_type, acc_number=acc_number)

print("Starting program ... " + datetime.datetime.now().strftime('%y/%m/%d %H:%M'))
while True:
    Exception = False

    ig_stream_service = IGStreamService(ig_rest_service)
    ig_stream_service.create_session(version='2')
    ig_stream_service.ls_client.subscribe(subscription_prices)     # Registering the Subscription

    while not Exception:
        try:
            time.sleep(1000000000)
        except:
            print("Exception: " + datetime.datetime.now().strftime('%y/%m/%d %H:%M'))
            ig_stream_service.unsubscribe_all()
            ig_stream_service.disconnect()
            Exception = True

I must have written this code incorrectly, as it doesn't seem to catch the exception (no output from print() statement).

Hope this all helps. Any suggestions gratefully received as how to proceed?

`` MH_Error_v0.2.log

[galburn]

It appears that the main thread will be busy sleeping, so your exception handling will not be triggered. Well that's my understanding of it which may well be wrong. I'm not an expert on threading. Your main issue is due to the LS server closing the connection, and the python readline() returning a null line. Trying to do an int() on a null is going to raise an exception.
The simplest option for you might be to close the LS subscription at 10:01PM on Friday when the markets close. Any data after that is going to have little value. This is what I now do.

[marcipops]

Hi Galburn, thanks for your reply.

I was trying to follow dario-weswit's advice from the 'defensive programming' school - make the client app robust. My app does nothing except receiving data in the callback i.e. the main loop does nothing. Thus leading me to put it to sleep rather than spin around consuming resources unnecessarily.

Is there any way of dealing with the root cause of the issue, i.e. handle the null line? Presumably, this will be editing the ig-markets-api-python-library?

If not, then perhaps we only have two options left: 1) trapping the exception, so enabling us to disconnect/reconnect - except how to resolve the 'sleep unresponsive to exceptions issue'? 2) force a disconnect (like you have done). Then will have to manage time zones, trading days and times to observe etc.

Thanks!

[marcipops]

Just revisited my logging: 2022-09-02 22:07:06,219: _receive(): Received message ---> <1,1|$|$|$|$|$|$|$|$|$||$> 2022-09-02 22:24:18,337: _receive(): Received message ---> <1,1|1662153840000||||||||||0> 2022-09-02 22:43:36,093: _receive(): Received message ---> <>

An exception is not thrown at the point: 2022-09-02 22:07:06,219: _receive(): Received message ---> <1,1|$|$|$|$|$|$|$|$|$||$>

So what would be the best way of detecting this?

Many thanks

[galburn]

Some pointers:- • You would NOT expect an exception at 22:07:06.219 because the first two values in the message are both ints. • The try/except needs to surround the place where the exception occurs (lightstreamer.py, line 325), not in the main loop. That is why it is not being triggered. This is what I have tried before I changed to the scheduled shutdown:- • You could also try matching to a null string and handling that gracefully:- Lines 357 to 359

[marcipops]

@dario-weswit I have raised the issue with support.ig.com on 3/8 and their reply was: " Thanks for your email and the downtime will be diffeent every weekend depending on what gets released so we unfortunately can't provide maintenance schedule in advance. Saturdays are for maintenance so if this inconvenience causes too much trouble then the best way would be to avoid connecting on Saturdays. "

I have reported it again today indicating that the current issue is occurring on Fridays.

[marcipops]

Some pointers:- • You would NOT expect an exception at 22:07:06.219 because the first two values in the message are both ints. • The try/except needs to surround the place where the exception occurs (lightstreamer.py, line 325), not in the main loop. That is why it is not being triggered. This is what I have tried before I changed to the scheduled shutdown:- • You could also try matching to a null string and handling that gracefully:- Lines 357 to 359

@galburn Thanks for the info - makes absolute sense. Did this work? Wondering why you have gone for scheduled shutdown instead?

Thanks again for your help and insights!

[dario-weswit]

@marcipops let me insist on the points expressed in my Jul 25 post. There is no need for an ordered shutdown of Lightstreamer Server. In any case, the python client would see an interruption of the socket, which is something that can happen regardless of the Server, also in case of network issues. So, the client should be resilient to socket interruptions. And this is what "receive" in lightstreamer.py actually does, by testing for "_read_from_stream" returning null or throwing an exception. The problem here is that "_read_from_stream" returns an empty string, which corresponds to receiving "\r\n" from the stream, which is not true. I don't know how this originates, but the case can be easily detected in "receive" and treated the same as a null, that is, as a socket interruption. I confirm that, in a correct stream from Lightstreamer Server, an empty line can never occur, hence it can always be treated as a stream interruption.

[marcipops]

Thanks @dario-weswit. Yes, agree that the change @galburn has suggested to receive() in lightstreamer.py (line 357-359 above) will ignore the null string, thus receive() would not call forward_update_message() therefore not creating an exception on int(tok[0]), tok[1].

Is the repo going to be updated with this fix, please?

Thanks

[galburn]

As @dario-weswit points out, my workaround fails to detect a network connection disconnect. It would be much better to throw an exception so that a new connection can be made. If you would like to make a patch to do this I would be happy to test it before a PR is created.

[dario-weswit]

@galburn there must be a misunderstanding. I think that your workaround should work. I didn't even notice it yesterday, when answering about Lightstreamer Server shutdown. My impression is that _read_from_stream returns an empty string where it was supposed to return null. So, it makes sense to treat the two outcomes in the same way. That said, you may evaluate if throwing an exception is better. As far as I'm concerned, I won't change the code, because I'm not familiar with Python.

[marcipops]

@galburn, @dario-weswit Thanks both.

Yesterday I put in the change galburn suggested, based on matching a null string, but without an exception (as if the matching works then it wouldn't throw an exception) - and am testing now to see if it copes over the weekend.

Also I've just had a response back from IG support, essentially saying to disconnect at the end of trading hours (just as galburn has done).

I want to test to see whether the change will handle the situation without having to manage time zones/market hours/summer time etc and do the disconnecting/reconnecting.

Thank you again both!

[galburn]

Sorry @dario-weswit, I was not reading back through the thread properly.
The suggestion from @zansibal is probably what I was thinking of. Yes, raising an exception would be a much more robust solution. @marcipops, I'll be interested to see how the "null match" works over the weekend when IG shutdown the LS servers. FYI, I shutdown my client at 22:01GMT on Friday and restart it at 22:00GMT on Sunday during the winter. When BST is active I change this to 21:00 GMT. This is a fairly trivial configuration maintenance task which I perform when changing my wall clocks. No real need for complex time zone calculations.

[galburn]

Testing of a connection break is difficult. When I attempt to simulate a LD disconnect by pulling out my RJ45 ethernet cable on my workstation, I get this:-

  File "C:\Users\galburn\IG1\ig1.py", line 2538, in <module>
    main()
  File "C:\Users\galburn\IG1\ig1.py", line 2118, in main
    eventloop()
  File "C:\Users\galburn\IG1\ig1.py", line 1633, in eventloop
    StreamRestart()
  File "C:\Users\galburn\IG1\ig1.py", line 501, in StreamRestart
    StreamStop()
  File "C:\Users\galburn\IG1\ig1.py", line 486, in StreamStop
    ig_stream_service.unsubscribe_all()
  File "c:\users\galburn\IG1\lib\site-packages\trading_ig\stream.py", line 49, in unsubscribe_all
    self.ls_client.unsubscribe(subcription_key)
  File "c:\users\galburn\IG1\lib\site-packages\trading_ig\lightstreamer.py", line 306, in unsubscribe
    server_response = self._control(
  File "c:\users\galburn\IG1\lib\site-packages\trading_ig\lightstreamer.py", line 166, in _control
    response = self._call(self._control_url, CONTROL_URL_PATH, params)
  File "c:\users\galburn\IG1\lib\site-packages\trading_ig\lightstreamer.py", line 149, in _call
    return _urlopen(url, data=self._encode_params(body))
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 519, in open
    response = self._open(req, data)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1776.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

Anybody have a suggestion on how to simulate a LS connection drop?

[dario-weswit]

With a browser client I would use a proxy, like fiddler, on which you can close sockets selectively. But I don't know if the Python runtime detects proxies seamlessly like browsers.

[marcipops]

@marcipops, I'll be interested to see how the "null match" works over the weekend when IG shutdown the LS servers.

@galburn - it has survived the weekend without crashing. The only change I put in was the one from your second image (check for null string). I'll keep it running to see how long it can go without crashing. I am logging just in case.

[marcipops]

@galburn - it has survived the second weekend without crashing, and all the data looks correct, so this simple fix is looking OK to go ahead with.

[marcipops]

@galburn - quick update as requested. Now third weekend but has errored out with:

2022-09-24 01:49:32,303 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-09-24 01:49:37,303 trading_ig.lightstreamer DEBUG Received message ---> 2022-09-24 01:49:37,304 trading_ig.lightstreamer DEBUG PROBE message 2022-09-24 01:49:37,304 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-09-24 01:49:42,304 trading_ig.lightstreamer DEBUG Received message ---> 2022-09-24 01:49:42,304 trading_ig.lightstreamer DEBUG PROBE message 2022-09-24 01:49:42,304 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-09-24 01:49:47,159 trading_ig.lightstreamer DEBUG Received message ---> <> 2022-09-24 01:49:47,160 trading_ig.lightstreamer ERROR Empty message received 2022-09-24 01:49:47,161 trading_ig.lightstreamer DEBUG Closing connection

All I have done is to update lightstreamer.py as you suggested above: elif message == "": #null string received receive = False log.error("Empty message received")

My client app has not produced any more log entries since this - so not sure what it is doing.

Which function and how best to handle this exception i.e. reestablish the connection?

[marcipops]

So has been running successfully for a few weeks but then stopped receiving data and stopped logging - the last few messages are below.

2022-10-13 23:41:50,028 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-10-13 23:41:51,025 trading_ig.lightstreamer DEBUG Received message ---> <1,1|||6893.1||6893.1||6889.1||6889.1||> 2022-10-13 23:41:51,025 trading_ig.lightstreamer DEBUG Received update message ---> <1,1|||6893.1||6893.1||6889.1||6889.1||> 2022-10-13 23:41:51,025 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-10-13 23:41:56,026 trading_ig.lightstreamer DEBUG Received message ---> 2022-10-13 23:41:56,026 trading_ig.lightstreamer DEBUG PROBE message 2022-10-13 23:41:56,026 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-10-13 23:42:00,028 trading_ig.lightstreamer DEBUG Received message ---> <1,1|||6893.4||6893.4||6889.4||6889.4||1> 2022-10-13 23:42:00,028 trading_ig.lightstreamer DEBUG Received update message ---> <1,1|||6893.4||6893.4||6889.4||6889.4||1> 2022-10-13 23:42:00,029 trading_ig.lightstreamer DEBUG Waiting for a new message

So it looks like it was stuck waiting for the next message. I killed the task and re-ran the program and it's now receiving messages and logging again.

I am wondering now whether the only route forward is to set up a separate scheduled task that runs at 12:00 midnight and kills the main task off and then re-runs it. Does seem a very poor situation not to be able to cater for this programmatically. Any suggestions? Thanks

[btodac]

@marcipops I have been having the same issue. It happens at random times of the day.
I tried

elif message == "": #null string received rebind = True receive = False log.error("Null string received")

to see if rebinding the connection solves the issue but leads to the following:

DEBUG:trading_ig.lightstreamer:Waiting for a new message DEBUG:trading_ig.lightstreamer:Received message ---> <> DEBUG:trading_ig.lightstreamer:Null string recieved DEBUG:trading_ig.lightstreamer:Binding to this active session ERROR:trading_ig.lightstreamer:Server response error: SYNC ERROR Exception in thread STREAM-CONN-THREAD-0:

@dario-weswit Am I correct in thinking the lightstreamer session has been terminated by the host?

[btodac]

From what I can see the empty string is returned by the readline function in LSClient._read_from_stream (From the python docs: 'if f.readline() returns an empty string, the end of the file has been reached'). The file in this case is a temporary file used by the urlib response class ('class addbase(tempfile._TemporaryFileWrapper):')

[dario-weswit]

ERROR:trading_ig.lightstreamer:Server response error: SYNC ERROR Exception in thread STREAM-CONN-THREAD-0:

@dario-weswit Am I correct in thinking the lightstreamer session has been terminated by the host?

Yes, SYNC ERROR means session not found. Anyway, in general, if a streaming connection is interrupted abruptly, issuing a rebind to the same session is not advisable. Even if the session was still alive (for instance, because of a network outage that the Server has not detected yet) and the rebind was successful, it would leave a hole in the update flow. BTW, there is a proper syntax to recover a session by resuming from a specific point, but it is not supported by the protocol in use here.

[marcipops]

So are we saying there is no programmatic solution, and the only reliable solution is manually killing the task after market close and rerunning on market open? thanks

[RRMXkun]

So are we saying there is no programmatic solution, and the only reliable solution is manually killing the task after market close and rerunning on market open? thanks

I was working on a "watcher" thread last year, which would monitor connection status and re-connect (not rebind, recreate a brand new connection). However I have not had the time to complete and property test that code. But the way I did it was by adding an extra layer (in stream.py) that would manage the connection and subscriptions.

[btodac]

I'm testing an edit to the end of _receive() using restart flag that is set when an empty string occurs and copying the subscriptions to the new LS connection. I think it will work as long as the security tokens are valid.

[marcipops]

Quick update, it's been running flawlessly since I restarted it from above. However, it stopped receiving any messages over the weekend again. The last one was 21/10 @ 22:00.

My logs are just filling up with debug messages re empty message:

2022-10-24 09:19:03,861 trading_ig.lightstreamer DEBUG Received message ---> <> 2022-10-24 09:19:03,861 trading_ig.lightstreamer ERROR Empty message 2022-10-24 09:19:03,861 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-10-24 09:19:03,861 trading_ig.lightstreamer DEBUG Received message ---> <> 2022-10-24 09:19:03,861 trading_ig.lightstreamer ERROR Empty message 2022-10-24 09:19:03,861 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-10-24 09:19:03,861 trading_ig.lightstreamer DEBUG Received message ---> <> 2022-10-24 09:19:03,861 trading_ig.lightstreamer ERROR Empty message 2022-10-24 09:19:03,861 trading_ig.lightstreamer DEBUG Waiting for a new message 2022-10-24 09:19:03,861 trading_ig.lightstreamer DEBUG Received message ---> <>

It seems to get stuck, once it receives an empty message, then regardless of legal messages that are being sent, the task just ignores them.

@dario-weswit thanks for the info - looks like we need to create a new session? @btodac your solution looks promising, as long as the task doesn't continually use up resources (with the old session state(s) still present)? Otherwise @RRMXkun's solution might be the only real way forward - at least it is the cleanest (but most draconian) way - kill and restart the task with a separate watchdog process (either based on flag on empty message, or only run during market hours).

Many thanks all.

[bug-or-feature]

I'm closing this, it's old. The latest release uses the official Lightstreamer Python client SDK, which does auto-reconnections, auto-resubscriptions. See #302