Scaling Issues with Dashing in the Enterprise

[divan-moller]

We are using Dashing extensively as a monitoring presentation layer at our company and it has enjoyed widespread adoption. Quick numbers: over 800 widgets across more than 150+ dashboards company wide.

Here is the problem:

Dashing broadcasts widget data to all connected dashboards regardless if the dashboard actually contains the widget. The end result? Dashboards that burn through 10+ GB a day... per dashboard.

Any suggestions to a fix?

source.addEventListener('message', function(e) receives all widget updates and it then selectively passes it on to widget.receiveData(data)

[divan-moller]

To help minimise the bandwidth consumed could gzip compression be enabled?

Chrome suggested in the audit: events could save ~59.7 MB (this was for a dashboard that ran for ten minutes)

[terraboops]

To really solve this problem, it would require re-engineering the event service to supply data to various widget endpoints and then have the dashboard subscribe to only those endpoints.

I don't know Batman well enough to say how difficult that would be, but it certainly would not be a small change. That being said, this sounds like an important and interesting problem to solve.

In the mean-time, it seems like your suggestion to gzip the event service output has merit:

Concerned by high overhead of UTF-8 on the wire? An SSE connection is a streaming HTTP response, which means that it can be compressed (i.e., gziped), just as any other HTTP response while in flight! While SSE is not meant for delivery of binary data, it is nonetheless an efficient transport: ensure that your server is applying gzip compression on the SSE stream. Source: High Performance Browser Networking

[terraboops]

I've thought about this a bit more, there may be a simpler solution to this problem. Also, the solution I suggested before may actually make the problem worse because of the additional HTTP overhead associated with each request (1 request per widget vs 1 request per dashboard). This would depend on how many widgets you have per dashboard (more than 4 / dashboard and this becomes useless).

For example, the event service could optionally take a list of widgets as a parameter and supply only data for those widgets. This would impact performance however, as the connection to the SSE endpoint could only be establish once the DOM is parsed and Dashing.widgets is filled with data.

If this is implemented, it should be a setting that must be turned on a per-dashboard basis.

[tijszwinkels]

I'm not saying that this solution will work for everybody, but we have chosen to solve a similar problem as following:

• Real-time data is published on an XMPP api through pubsub. (running ejabberd)
• The XMPP api is also exposed through BOSH (so it can be accessed from browsers.)
• We are using a browser-only version of Dashing. (I've actually just put the source up on Github. ) This also uses browser-side jobs for data gathering.
• These in-browser dashing jobs connect to the XMPP api using the Strophe.js library. They only subscribe to the pubsub-nodes that are relevant to them.

Advantages:

• This moves the problem of scalable data-distribution to XMPP. XMPP has been designed with scalable data-distribution in mind.
• We avoid some headaches with having Dashing / Batman deal with different data for different users on the server-side. Instead, we use an existing client-side javascript framework to deal with authorisation and authentication. The jobs only need to know with what authentication data to subscribe to which nodes.
• Coming from a situation where the XMPP-infrastructure was already in place, it saves us from having to deploy and maintain another piece of infrastructure (the dashing server). The current solution can run from any webserver serving static files to clients.

[v0lkan]

bump!

That's really an important issue for large-scale enterprise usage.

one workaround would be to create multiple dashing sub-sites (i.e., dasbhoard-foo.example.com, dashboard-bar.example.com etc) -- At least that's how I'm planning to solve it, or similar to @tijszwinkels hack the mesage queue and replace with something else.