rstudio / shinycannon

Load generation tool, part of shinyloadtest
https://rstudio.github.io/shinyloadtest/
15 stars 6 forks source link

Bump `receiveQueue` limit from 5 to 50 #63

Closed schloerke closed 2 years ago

schloerke commented 2 years ago

Example

2021-11-22 13:06:27.713 ERROR [thread06] - Playback failed: receiveQueue is full (max = 5)
java.lang.Exception: receiveQueue is full (max = 5)

This can occur when custom messages are received out of order. (Non-deterministic. Very plausible situation.)

Previously, the queue had a size of 5. Now it is set to 50.


If app authors are sending enough non-deterministic custom messages to hit the new limit, I'd love to see what the app is doing and how we could address it.

schloerke commented 2 years ago

Example debug.log snippet showing that the increased queue size is working past limit = 5. (It would have broken before this PR)

2021-11-24 11:27:57.312 DEBUG [ReadingThread] - %%% Received: a["1E#0|m|{\"custom\":{\"my-custom-message\":{\"i\":20}}}"]
2021-11-24 11:27:57.312 DEBUG [ReadingThread] - %%% Message queue length: 1
2021-11-24 11:27:57.312 DEBUG [ReadingThread] - %%% Received: a["1F#0|m|{\"custom\":{\"my-custom-message\":{\"i\":21}}}"]
2021-11-24 11:27:57.312 DEBUG [ReadingThread] - %%% Message queue length: 2
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Received: a["20#0|m|{\"custom\":{\"my-custom-message\":{\"i\":22}}}"]
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Message queue length: 3
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Received: a["21#0|m|{\"custom\":{\"my-custom-message\":{\"i\":23}}}"]
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Message queue length: 4
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Received: a["22#0|m|{\"custom\":{\"my-custom-message\":{\"i\":24}}}"]
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Message queue length: 5
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Received: a["23#0|m|{\"custom\":{\"my-custom-message\":{\"i\":25}}}"]
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Message queue length: 6
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Received: a["24#0|m|{\"custom\":{\"my-custom-message\":{\"i\":26}}}"]
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Message queue length: 7
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Received: a["25#0|m|{\"custom\":{\"my-custom-message\":{\"i\":27}}}"]
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Message queue length: 8
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Received: a["26#0|m|{\"custom\":{\"my-custom-message\":{\"i\":28}}}"]
2021-11-24 11:27:57.313 DEBUG [ReadingThread] - %%% Message queue length: 9
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["27#0|m|{\"custom\":{\"my-custom-message\":{\"i\":29}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 10
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["28#0|m|{\"custom\":{\"my-custom-message\":{\"i\":30}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 11
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["29#0|m|{\"custom\":{\"my-custom-message\":{\"i\":31}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 12
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["2A#0|m|{\"custom\":{\"my-custom-message\":{\"i\":32}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 13
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["2B#0|m|{\"custom\":{\"my-custom-message\":{\"i\":33}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 14
2021-11-24 11:27:57.314 DEBUG [thread01] - WS_RECV received: a["1D#0|m|{\"custom\":{\"my-custom-message\":{\"i\":19}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["2C#0|m|{\"custom\":{\"my-custom-message\":{\"i\":34}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 14
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["2D#0|m|{\"custom\":{\"my-custom-message\":{\"i\":35}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 15
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Received: a["2E#0|m|{\"custom\":{\"my-custom-message\":{\"i\":36}}}"]
2021-11-24 11:27:57.314 DEBUG [ReadingThread] - %%% Message queue length: 16
2021-11-24 11:27:57.314 DEBUG [thread01] - WS_RECV received: a["1E#0|m|{\"custom\":{\"my-custom-message\":{\"i\":20}}}"]
2021-11-24 11:27:57.315 DEBUG [ReadingThread] - %%% Received: a["2F#0|m|{\"custom\":{\"my-custom-message\":{\"i\":37}}}"]
2021-11-24 11:27:57.315 DEBUG [ReadingThread] - %%% Message queue length: 16

(It quickly reduced itself to 0 after this.)

Reprex app:

library(shiny)

ui <- fluidPage(
  tags$style("td { border: 1px black solid; }"),
  tags$script(htmltools::HTML('
    // This receives messages of type "my-custom-message" from the server.
    Shiny.addCustomMessageHandler("my-custom-message",
      function(message) {
        console.log(JSON.stringify(message));
        $("#code").text(JSON.stringify(message));

        if (message.i == 1) {
          setTimeout(function() {
            Shiny.setInputValue("testOutput", 2);
          }, Math.floor(Math.random() * (1000 / 5) * 0.8))
        }
      }
    );
  ')),
  h1("Non-deterministic custom message app"),
  p("Number of messages: ", textOutput("n1", inline = TRUE)),
  tags$pre(id = "code"),
  p("This app sends ", textOutput("n2", inline = TRUE), " custom messages from the server. The UI will also set a value at a set timeout. Due to natural timing errors, the recorded location of the set input value will be in a different location each time."),
  p("In this situation, the fixed branch should not throw an error for up to 50 out-of-order messages. Where as the `main` branch should work only for up to 5 out-of-order-messages (rare)."),
  markdown('
    |out-of-order count|Works on `main`| Works on `PR`|
    |--|--|--|
    |`<= 5`|Yes|Yes|
    |`6-50`|**No**|**\\*\\*Yes\\*\\*** |
    |`>50`|No|No|
  '),
  tags$code("app.R"),
  verbatimTextOutput("app")
)

server <- function(input, output, session) {

  n <- 100

  output$n1 <- renderText(n)
  output$n2 <- renderText(n)

  output$app <- renderText(paste0(readLines("app.R"), collapse = "\n"))

  send_msgs <- function() {
    lapply(seq_len(n), function(i) {
      later::later(function() {
        session$sendCustomMessage(
          type = 'my-custom-message',
          message = list(i = i)
        )
      }, delay = 1 + (i/100) / 5)
    })
  }
  send_msgs()
}

shinyApp(ui, server)
schloerke commented 2 years ago

For completeness, here is a debug.log snippet of a failing queue using the reprex app above:

2021-11-24 11:35:29.449 DEBUG [ReadingThread] - %%% Received: a["34#0|m|{\"custom\":{\"my-custom-message\":{\"i\":42}}}"]
2021-11-24 11:35:29.449 DEBUG [ReadingThread] - %%% Received: a["35#0|m|{\"custom\":{\"my-custom-message\":{\"i\":43}}}"]
2021-11-24 11:35:29.449 DEBUG [ReadingThread] - %%% Received: a["36#0|m|{\"custom\":{\"my-custom-message\":{\"i\":44}}}"]
2021-11-24 11:35:29.449 DEBUG [ReadingThread] - %%% Received: a["37#0|m|{\"custom\":{\"my-custom-message\":{\"i\":45}}}"]
2021-11-24 11:35:29.449 DEBUG [ReadingThread] - %%% Received: a["38#0|m|{\"custom\":{\"my-custom-message\":{\"i\":46}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["39#0|m|{\"custom\":{\"my-custom-message\":{\"i\":47}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["3A#0|m|{\"custom\":{\"my-custom-message\":{\"i\":48}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["3B#0|m|{\"custom\":{\"my-custom-message\":{\"i\":49}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["3C#0|m|{\"custom\":{\"my-custom-message\":{\"i\":50}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["3D#0|m|{\"custom\":{\"my-custom-message\":{\"i\":51}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["3E#0|m|{\"custom\":{\"my-custom-message\":{\"i\":52}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["3F#0|m|{\"custom\":{\"my-custom-message\":{\"i\":53}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["40#0|m|{\"custom\":{\"my-custom-message\":{\"i\":54}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["41#0|m|{\"custom\":{\"my-custom-message\":{\"i\":55}}}"]
2021-11-24 11:35:29.450 DEBUG [ReadingThread] - %%% Received: a["42#0|m|{\"custom\":{\"my-custom-message\":{\"i\":56}}}"]
2021-11-24 11:35:29.451 DEBUG [ReadingThread] - %%% Received: a["43#0|m|{\"custom\":{\"my-custom-message\":{\"i\":57}}}"]
2021-11-24 11:35:29.451 DEBUG [ReadingThread] - %%% Received: a["44#0|m|{\"custom\":{\"my-custom-message\":{\"i\":58}}}"]
2021-11-24 11:35:29.451 DEBUG [ReadingThread] - %%% Received: a["45#0|m|{\"custom\":{\"my-custom-message\":{\"i\":59}}}"]
2021-11-24 11:35:29.451 ERROR [thread01] - Playback failed: receiveQueue is full (max = 5)
java.lang.Exception: receiveQueue is full (max = 5)
    at com.rstudio.shinycannon.Event$WS_OPEN$handle$2$$special$$inlined$also$lambda$1.onTextMessage(Events.kt:280)
    at com.neovisionaries.ws.client.ListenerManager.callOnTextMessage(ListenerManager.java:353)
    at com.neovisionaries.ws.client.ReadingThread.callOnTextMessage(ReadingThread.java:266)
    at com.neovisionaries.ws.client.ReadingThread.callOnTextMessage(ReadingThread.java:244)
    at com.neovisionaries.ws.client.ReadingThread.handleTextFrame(ReadingThread.java:969)
    at com.neovisionaries.ws.client.ReadingThread.handleFrame(ReadingThread.java:752)
    at com.neovisionaries.ws.client.ReadingThread.main(ReadingThread.java:108)
    at com.neovisionaries.ws.client.ReadingThread.runMain(ReadingThread.java:64)
    at com.neovisionaries.ws.client.WebSocketThread.run(WebSocketThread.java:45)