Dashboards with a large number of widgets no longer return all data

[mikewalkerblackpatch]

Since the change introduced by https://github.com/Smashing/smashing/pull/173# some of our dashboards, which contain a huge number of widgets don't return all the data.

The GET request now being made (included below) does not contain all the data id's. There are 104 in the request shown below but we have 168 on this particular dashboard.

127.0.0.1 - - [02/Mar/2021:13:11:22 +0000] "GET /events?ids=applivery%2Capplivery.sit%2Capplivery.uat%2Capplivery.preprod%2Capplivery.live%2Cclient-api%2Cclient-api.sit%2Cclient-api.uat%2Cclient-api.preprod%2Cclient-api.live%2Ccollection-streams-processor%2Ccollection-streams-processor.sit%2Ccollection-streams-processor.uat%2Ccollection-streams-processor.preprod%2Ccollection-streams-processor.live%2Ce2e-parcel-events-connector%2Ce2e-parcel-events-connector.sit%2Ce2e-parcel-events-connector.uat%2Ce2e-parcel-events-connector.preprod%2Ce2e-parcel-events-connector.live%2Ce2e-tour-reporting-sink-connector%2Ce2e-tour-reporting-sink-connector.sit%2Ce2e-tour-reporting-sink-connector.uat%2Ce2e-tour-reporting-sink-connector.preprod%2Ce2e-tour-reporting-sink-connector.live%2Cemail-sender%2Cemail-sender.sit%2Cemail-sender.uat%2Cemail-sender.preprod%2Cemail-sender.live%2Chht-auth-service%2Chht-auth-service.sit%2Chht-auth-service.uat%2Chht-auth-service.preprod%2Chht-auth-service.live%2Chht-config-api%2Chht-config-api.sit%2Chht-config-api.uat%2Chht-config-api.preprod%2Chht-config-api.live%2Chht-event-feed%2Chht-event-feed.sit%2Chht-event-feed.uat%2Chht-event-feed.preprod%2Chht-event-feed.live%2Chht-parcel-api%2Chht-parcel-api.sit%2Chht-parcel-api.uat%2Chht-parcel-api.preprod%2Chht-parcel-api.live%2Chht-routing-api%2Chht-routing-api.sit%2Chht-routing-api.uat%2Chht-routing-api.preprod%2Chht-routing-api.live%2Chht-tour-api%2Chht-tour-api.sit%2Chht-tour-api.uat%2Chht-tour-api.preprod%2Chht-tour-api.live%2Ckafka-delayed-topics%2Ckafka-delayed-topics.sit%2Ckafka-delayed-topics.uat%2Ckafka-delayed-topics.preprod%2Ckafka-delayed-topics.live%2Cmars-et-services%2Cmars-et-services.sit%2Cmars-et-services.uat%2Cmars-et-services.preprod%2Cmars-et-services.live%2Cmars-opunits-connector%2Cmars-opunits-connector.sit%2Cmars-opunits-connector.uat%2Cmars-opunits-connector.preprod%2Cmars-opunits-connector.live%2Cmars-parcel-feed%2Cmars-parcel-feed.sit%2Cmars-parcel-feed.uat%2Cmars-parcel-feed.preprod%2Cmars-parcel-feed.live%2Cmars-parcel-feed-jdbc-connectors%2Cmars-parcel-feed-jdbc-connectors.sit%2Cmars-parcel-feed-jdbc-connectors.uat%2Cmars-parcel-feed-jdbc-connectors.preprod%2Cmars-parcel-feed-jdbc-connectors.live%2Cmars-parcel-target-hist-connector%2Cmars-parcel-target-hist-connector.sit%2Cmars-parcel-target-hist-connector.uat%2Cmars-parcel-target-hist-connector.preprod%2Cmars-parcel-target-hist-connector.live%2Cparcel-date-processor%2Cparcel-date-processor.sit%2Cparcel-date-processor.uat%2Cparcel-date-processor.preprod%2Cparcel-date-processor.live%2Croute-analysis-connector%2Croute-analysis-connector.sit%2Croute-analysis-connector.uat%2Croute-analysis-connector.preprod%2Croute-analysis-connector.live%2Croute-analysis-tour-data-connector%2Croute-analysis-tour-data-connector.sit%2Croute-analysis-tour-data-connector.uat%2Croute-analysis-tour-data-connector.preprod HTTP/1.1" 200 - 90.1307

Thanks to @toy an @kinow for there help so far.

[toy]

The fact that you were still receiving data for part of widget didn't give me rest, so I've generated a dashboard with 160 widgets:

<% content_for :title do %>My super sweet dashboard<% end %>

<script type='text/javascript'>
$(function() {
  Dashing.widget_base_dimensions = [50, 50]
  Dashing.numRows = 10
  Dashing.numColumns = 16
});
</script>

<div class="gridster">
  <ul>
    <% 10.times do |a| %>
      <% 16.times do |b| %>
        <li data-row="<%= a + 1 %>" data-col="<%= b + 1 %>" data-sizex="1" data-sizey="1">
          <div data-id="test-<%= a %>-<%= b %>" data-view="Text" data-moreinfo="<%= a %>×<%= b %>"></div>
        </li>
      <% end %>
    <% end %>
  </ul>
</div>

Looking at console I realised that EventSource is created too early and after failing to find anything in documentation I dug through batman.js code and found apparently the only event that should work without changing any other logic.

A bit more explanation in #182

[kinow]

@mikewalkerblackpatch releasing 1.3.3 now, if you could test your dashboard again with that fix. @toy 's PR linked this issue, but I am not sure if that fix will resolve the issue with large number of widgets that have also long names. Even if you manage to shorten the name of the widget ID's, we should still add the new flag to have the /events/ endpoint returning all the events without ids=abc,def etc. Thanks!

[mikewalkerblackpatch]

@kinow & @toy I updated this afternoon and 1.3.4 came down. My dashboards all appear to be rendering correctly and I haven't even needed to shorten any id's. Thank you so much for the incredibly quick turn around on this. I am very happy to close out this bug now. Many thanks again

[kinow]

Thank you for your patience and help with this issue @mikewalkerblackpatch . All props go to @toy for his work on the past two releases :1st_place_medal: