Add metrics for how many delta's is each user generating

[Frooxius]

Is your feature request related to a problem? Please describe.

Some sessions can be susceptible to queued packets, particularly with a number of users. Currently the diagnostic tools for this are very limited, especially to help narrow down which users are more responsible for more network traffic, compared to others.

Describe the solution you'd like

We will add additional metrics for how much delta/correction messages are generated per second by each user in the session. Potentially we can also add metrics for different types of messages, like how much of the generated traffic is stream data.

Describe alternatives you've considered

We can also add diagnostics for which parts of hierarchy are responsible for how much data, but this is both more complicated to implement and will require more complicated tools to provide the information and would also have significant performance impact.

This way is quicker to add, so we'll be doing this first.

Additional Context

This originally came in a session of Blood On The Clocktower and having issues with a number of users, potentially due to avatars generating significant amount of network traffic.

[5H4D0W-X]

When you talk about delta messages, do you mean the Delta flux node or something else related to the data model?

[Banane9]

When you talk about delta messages, do you mean the Delta flux node or something else related to the data model?

He means the underlying data model changes and network traffic. Think: diagnostics to determine if there is anything or anyone in particular causing people to have queued packets.

This would be very helpful to evaluate whether items people create are well localized too - that could've avoided the first BeatLink release :'D

Currently the diagnostic tools for this are very limited, especially to help narrow down which users are more responsible for more network traffic, compared to others.

We will add additional metrics for how much delta/correction messages are generated per second by each user in the session.

[Frooxius]

These have been added in 2024.1.23.176!

[bontebok]

I've been working with the three new nodes and I have some observations and questions about the usefulness of the values they produce. It appears that the amount of change of the Delta Messages node is limited to about one update per frame, this leads me to believe that the underlying volume of changes are still masked under the value.

For example, equipping a brush tool tends to immediately increase the Delta Messages to a value slightly above the number of frames per second, while opening your dash and letting your laser rest on the canvas causes roughly the same increase (not sure why either of these activities creates Delta Messages exactly). However, doing both activities at the same time still results in the same increase of Delta Messages, not the sum of the two activities.

If at most one Delta Messages accumulates per each frame and the underlying changes are masked underneath, how useful is the Delta Messages node in determining the volume of delta changes in the world or the overall load?

[Frooxius]

This is intentional. Generally you don't care about bursts of delta messages like that - they level out quickly. In most cases what causes issues is sustained high-rate generation of delta messages over time, because that is what can end up causing the queue to be backed up - if users are constantly generating lots of messages constantly.

For that the accumulated number of messages over each second is more than sufficient - you want to generate large intervals too - how much delta's is the user generating per 10 seconds on average? How much per minute?

If we updated the amount every update, then this diagnostic would actually end up generating significantly more network traffic on itself and would end up hurting the network performance more.

Like mentioned in the initial post, the goal of this addition isn't to provide a very fine grained diagnostic, that would be a separate feature, but rather a quick to add mechanism that lets you diagnose the overall rate.

[bontebok]

Ok, that makes sense, thanks for providing the insight - I just wanted to understand the goal of the nodes as we noticed in a few sessions some users already maxed out the Delta Messages just by existing in their personal avatars. Since I can't tell if that's 1 or 300 changes, it's hard to say how much they are contributing to queueing overall.

Regarding the other observations we've made to what can cause constant Delta Messages, like placing a skinned mesh tooltip on your toolshelf, equipping a skinned mesh tooltip, equipping a brush tool, or resting your laser onto your dash leading to constant Delta Messages, should I report those findings on GitHub?

[Frooxius]

Generally you'd want stuff that's "just existing" to generate as few delta changes as possible if things aren't really visibly changing - this might be good indicator that more things should be driven.

The things that you report are completely normal and expected behaviors. Anything that changes state of the world will generate a delta message. I would not really report that, because we'll just close that as "by design", because that's perfectly normal behavior - delta messages aren't "harmful" in the sense that you want to eliminate every single one.

EDIT: If you think about it, equipping tool or placing it or changing you laser are all sudden changes of state - it is expected those would generate delta messages.

Typically stuff like movement, voice and such, are by default synchronized using stream messages (which do not queue), so existing like that will not cause delta changes.

Writing values often will though.

[bontebok]

Thanks Froox, just to clarify I wasn't referring to actions that lead to a burt of Delta messages (the scenario you described makes sense) but ones that send continuous delta messages.

Perhaps you can clear up one example from the list, when I open my Dash and let my laser rest upon it, why would this lead to a continuous delta message sent to the host until my laser is moved off of my dash or my dash is closed, given that the act of pointing my laser at my dash does not resulting in a world data model change?

Thanks!