[Bug]: TSL3.1 Takes a Ton of Time to Process Data in >3.x

[JTF4]

What happened?

When you use the TSL3.1 source type, something is causing tally data to be processed slowly and easily get backed up. Often times it will just stop working for a few seconds then send a all the tally data within a second.

Version

3.0.1

Distribution

Desktop

OS

Windows

What browsers are you seeing the problem on?

No response

If applicable, What Listener Clients are You Using?

No response

TallyArbiter configuration

No response

Relevant log output

No response

Error stacktrace (if applicable)

No response

[josephdadams]

Can confirm this is still happening. We get delays across the board because the tally data is not being processed fast enough for some reason

[josephdadams]

Is only a problem in TA 3.0 or greater

[JTF4]

@MatteoGheza is there some synchronous process in the TSL 3 source type that could be holding everything up?

[The-Sizzler]

This might be what is causing my problem on Mac that I posted about in the FB Group.

[MatteoGheza]

@MatteoGheza is there some synchronous process in the TSL 3 source type that could be holding everything up?

I'm sorry, IDK how that protocol works and I didn't work on TSL.ts. I tried to use the NodeJS Profiler, but it don't work very well.

@hrueger can you take a look at this?

[hrueger]

Yes, I can take a look at it. I just don't know when I'll have time for that as I'm currently in my last year of school - which means lots of tests and the finals are also arriving.

[The-Sizzler]

Switching from TCP to UDP solved my issues.

[JTF4]

I've been working the TSL source today and I've seen that as well. Once I switch to the UDP type, the delay is greatly decreased. It looks like there's something in the TCP protocol that's slowing down tally decoding to a crawl.

There is still an issue with Ross carbonites where every now and then the carbonite will send two different tally statuses for preview that will cause the standard TSL source type to miss one update every now and then.

I'm digging into the TCP issue now. I know that TCP is normally slower than UDP, but I've never seen it run this slow.

[hrueger]

Thanks @JTF4 for your recent contributions. I'll be able to work on TA again in a couple months, just have to finish school now with all the finals ;-) but than I'll hopefully have more time for Open-Source work.

[josephdadams]

I created a quick script to send TSL 3.1 (both TCP and UDP, selectable), for testing purposes: https://github.com/josephdadams/tsl-tester

I spun up a local instance of TA and it looks like, whether using TCP or UDP, the data hangs up every 3-4 seconds for 1-2 seconds and then repeats. While it's hung up, the tally states are frozen.

@JTF4

I suspect it could be related to the parser library, I will see if I can remove this or implement in another way.

[josephdadams]

I implemented a temp solution that does not rely on parser and the issue remains, so I will look further up the chain.

[josephdadams]

It does not seem delayed in the logs that show up in the UI. But the tally states in the UI are delayed. I am going to try increasing the speed of the test mode to determine if Test Mode has the same problem.

[josephdadams]

Yes, internal test mode has the same problem. You can easily see it by changing the interval from 1000 (1 second) to <300ms.

[josephdadams]

https://github.com/josephdadams/TallyArbiter/assets/2559091/c7183249-1f93-4ff4-8980-f2b875c352a1

Here is a quick video of the problem. This is with internal test mode on set to 100ms.

[hrueger]

What happens if you console.log the incoming websocket messages? It that logging also delayed or on time? Then, we'd have a hint whether it is a ui update issue or something with actually pushing out the messages in the backend.

[josephdadams]

I console.log'd on this.socket.on('tally_data') and it hits the browser console instantly. So the delay is happening somewhere after that.

@JTF4 I suppose this means that the data coming out of TA to listener clients and back to the MV of the Carbonite might not be delayed and that this is purely a UI issue

[hrueger]

I just tried it - works just fine for me using Firefox as well as using Chrome on macos...

https://github.com/josephdadams/TallyArbiter/assets/18656830/84e07e4d-388a-4c96-af09-bd363d7f78d5

[josephdadams]

Interesting. @JTF4 we need to test with a computer outside of our organization to rule out any business policy apps

[hrueger]

Did you test with a browser or in electron? I wonder whether something like power save mode kicks in, maybe also if the application is not in focus.

[josephdadams]

In Chrome browser on Macos. I'll test in other browsers

[josephdadams]

Same in Firefox over here, but I need to test on a non business machine just to rule that out.

[josephdadams]

FWIW, I am seeing the same delay on my TSL client - the tally data that goes back out to my UMD display. If the TA interface is lagging, the UMD matches.

[josephdadams]

I haven't been able to figure out what is causing the browser lag, but I've been able to determine that if you have a tsl_client with only test mode, there is no delay going out to the tsl_client, however, the moment you add a source of type TSL (at least TSL 3.1 TCP as that is all I tested), now you will see a delay going out that matches what the browser is doing. So perhaps something directly in the TSL source type implementation

[josephdadams]

Disabling the reconnect (setting reconnect: false before calling the super) in TSL.ts seems to help. I will test this against the switcher but my tsl test sript, it was rather performant

[josephdadams]

well that was fun. The TSL protocol was not properly implemented in TCP which was creating lots of race conditions, hence the false tally states when data comes in really quickly. Some switchers like the Ross Carbonite also send tally data in chunks so you have to process that in individual items or else the tally data gets kinda screwy which is why it was acting slowly. I will push out a release soon but it's fixed in master.

[hrueger]

That makes sense, thanks for debugging! I hadn't added any souces other than the test mode source, that's probably why it did work for me.