Disconnected from the server but no errors

[a-velt]

Hi everyone,

I'm trying to deploy my shiny app with the open-source shiny-server version.

You can see it here.

http://138.102.159.39/vitisimageviewer/

The code :

https://gitlab.com/avelt/vitisimageviewer

It's a matter of making filters on the table, then clicking on the "View images" button and the images are displayed as images in the shiny app using ggdraw and grid.arrange.

The application works to display for example 40 images, but above 100 images, I am disconnected from the server.

It is not a problem of timeout because I added the necessary options, here is my shiny-server configuration file.

# Instruct Shiny Server to run applications as the user "shiny"
run_as shiny;
http_keepalive_timeout 30000;
sanitize_errors false;
preserve_logs true;

# Define a server that listens on port 3838
server {
  listen 80;

  # Define a location at the base URL
  location / {

    app_idle_timeout 0;
    app_init_timeout 25000;

    # Host the directory of Shiny Apps stored in this directory
    site_dir /srv/shiny-server;

    # Log all Shiny output to files in this directory
    log_dir /var/log/shiny-server;

    # When a user visits the base URL rather than a particular application,
    # an index of the applications available in this directory will be shown.
    directory_index on;
  }
}

I have no errors in my logs. I have some warnings but I have the same when I run the application locally on my rstdudio. Locally, the application works very well, even requesting 1000 images for example.

shiny-server.log :

[2022-01-06T11:32:30.884] [INFO] shiny-server - Shiny Server v1.5.17.973 (Node.js v12.22.6)
[2022-01-06T11:32:30.886] [INFO] shiny-server - Using config file "/etc/shiny-server/shiny-server.conf"
[2022-01-06T11:32:30.938] [INFO] shiny-server - Starting listener on http://[::]:80

vitisimageviewer-shiny-20220106-113239-43797.log :

su: ignoring --preserve-environment, it's mutually exclusive with --login
Warning in loadSupport(appDir, renv = sharedEnv, globalrenv = NULL) :
  Loading R/ subdirectory for Shiny application, but this directory appears to contain an R package. Sourcing files in R/ may cause unexpected behavior.
ℹ Loading VitisImageViewer
Warning: replacing previous import ‘EBImage::combine’ by ‘dplyr::combine’ when loading ‘VitisImageViewer’
Warning: replacing previous import ‘dplyr::combine’ by ‘gridExtra::combine’ when loading ‘VitisImageViewer’
Warning: replacing previous import ‘shiny::runExample’ by ‘shinyjs::runExample’ when loading ‘VitisImageViewer’
Warning: replacing previous import ‘shinyWidgets::alert’ by ‘shinyjs::alert’ when loading ‘VitisImageViewer’
Warning: replacing previous import ‘EBImage::show’ by ‘shinyjs::show’ when loading ‘VitisImageViewer’
Warning: replacing previous import ‘shinyjs::runExample’ by ‘shiny::runExample’ when loading ‘VitisImageViewer’

Listening on http://127.0.0.1:43797
Warning: Good news!
You don't need to call `useShinyalert()` anymore. Please remove this line from your code.
If you really want to pre-load {shinyalert} to the UI for any reason, use:
        `useShinyalert(force = TRUE)`

My shiny-server is hosted on a 20gb ram/4 cpus VM. My personal computer has 16gb ram and the app is working fine so i don't believe it is a "RAM problem". However, I tried to monitor the RAM during the image processing in the shiny app, with "top", but the RAM was far from being completely used up. I'm out of ideas to debug this ... Anyone have an idea?

Thank you very much for your help !

Best,

Amandine

PS : My R version is R version 3.6.3 (2020-02-29)

I installed it with

sudo apt-get install r-base

on my ubuntu VM.

PS2 :

I deployed my shiny app on a server with 300Gb of RAM and a different version of shiny-server open-source and R, same problem of disconnection after trying to show > 100 images ... Definitely it's not a memory usage problem ...

[lvalnegri]

hi there,

Why don't you just paginate the images view in blocks of 10/20/50? is there any reason you want to show 100s images at any single time?

[a-velt]

Because users want to filter the table on more than one criteria and therefore may potentially want to display more than 100 images if their filters match more than 100 images ... But this might not often happen, actually. I just want to let the possibility to do this.

How can I do this pagination of images? In fact I struggled a bit to find a way to display the images without taking up too much space on the page, by putting them in groups of 3 on a graphic, but I am ok to take other ideas that I do not know.

[lvalnegri]

you can try the slickR package

[a-velt]

Thank you very much I will look at that package.

However, I don't think this is the origin of my problem with shiny-server or yes it can be ? I do not understand why it works with 100 images and not 200, knowing that I have enough memory on my VM ...

[lvalnegri]

I'm not such an expert, AFAIK it can even be a client (browser) problem (and that could be the reason why shiny server disconnect without showing problems of its own)

On another note, you could separate table and images on two different tabs, so table can be taller (you can also add a scroller extension) and you can show more images. Moreover, you can add viewing images on selection as well as filtering

[jcheng5]

When Shiny applications run locally, they use WebSockets, which are supported by modern browsers and if you have a good connection are an extremely simple and robust way to send messages back and forth between the browser and the server. When run over Shiny Server, they use the SockJS library, which is designed to work better over unreliable connections and/or older network proxies that can't reliably deal with WebSocket connections.

This latter problem is quite complex, and involves some heuristics where SockJS makes assumptions about how long it should reasonably take some event or other to occur. For example, the SockJS server will periodically ping the browser to make sure it's still listening; if it doesn't hear back within 10 seconds, the server assumes the browser is gone and severs the connection.

The problem here could be that the 200 images you're generating are keeping the browser busy for more than 10 continuous seconds, without the slightest pause that would let the SockJS client respond to the ping. Maybe the 100 images only take 6 seconds to process, and 200 images take 12 seconds, for example. (Note that these times are different than the extremely long server-side processing times involved; we're interested here in the longest uninterrupted stretch of JavaScript execution.)

There are ways to deal with that by tweaking Shiny Server config settings, but in this case, I think there are probably much much faster ways to render these images that will work at least as well. Here's a quick sketch, it should show the images in seconds not minutes:

library(shiny)

# This call tells Shiny to serve the extdata/images_reduites/ directory at the
# /images_reduites/ URL path.
addResourcePath("images_reduites", system.file("extdata/images_reduites", package = "VitisImageViewer"))

image_df <- read.csv(system.file("extdata/Liste_photos.csv", package = "VitisImageViewer"))

show_images <- function(filenames) {
  # Break the filenames into groups of 3
  images_per_row <- 3
  rows <- ceiling(length(filenames) / images_per_row)
  groups <- inverse.rle(list(values = seq_len(rows), lengths = rep_len(images_per_row, rows)))
  groups <- head(groups, length(filenames))

  # image_rows is a list, where each element is (up to) 3 filenames
  image_rows <- split(filenames, groups)

  tags$div(class = "container-fluid",
    # Create a fluidRow() for each row
    lapply(image_rows, function(image_row) {
      fluidRow(
        lapply(image_row, function(filename) {
          # Put the image in a column of width 4, so they fill the row (which has 12 units total)
          column(4, class = "text-center",
            tags$img(
              src = file.path("images_reduites", filename),
              style = "max-width: 100%; margin-bottom: 2em;",
              loading = "lazy"
            )
          )
        })
      )
    })
  )
}

ui <- fluidPage(
  DT::DTOutput("table"),
  actionButton("view_images", "View images"),
  uiOutput("images")
)

server <- function(input, output, session) {
  output$table <- DT::renderDT({
    DT::datatable(image_df,
      # to allow filtering table on each column
      filter = 'top',
      # allow the user to edit table cells
      editable = TRUE,
      extensions = 'Buttons',
      rownames= TRUE,
      options = list(
        # to keep the filters of the user on the table
        # and access the filtered table with input$table_state
        stateSave = TRUE,
        paging = TRUE,
        searching = TRUE,
        fixedColumns = TRUE,
        autoWidth = TRUE,
        ordering = TRUE,
        dom = 'Bfrtip',
        scrollX=TRUE,
        pageLength = 5,
        deferRender=TRUE,
        scrollCollapse=TRUE,
        # in order to remove the rownames after clicking the download button
        columnDefs = list(
          list(width = '200px', targets = list(3,6,8,12)),
          list(targets = list(13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31), visible = FALSE)
        ),
        buttons = list(
          I('colvis'),
          list(extend = "excel", text = "Download Table", title = NULL,filename = "data",
            exportOptions = list(
              modifier = list(page = "all")
            )
          )
        )
      )
    )
  })

  images_to_show <- reactiveVal()

  observeEvent(input$view_images, {
    images_to_show(image_df[input$table_rows_all, , drop = FALSE])
  })
  observeEvent(input$table_rows_all, {
    images_to_show(NULL)
  })
  output$images <- renderUI({
    # Don't show anything if there's nothing to show
    req(!is.null(images_to_show()))
    req(nrow(images_to_show()) > 0)

    show_images(images_to_show()$Photo)
  })
}

shinyApp(ui, server)

[a-velt]

Woooh ! I integrated your code in my app, it works very well and it's ....... really really faster ! Thank you very much ! I will test that on shiny-server but for sure, the problem will disappear !

[a-velt]

Thank you again, the problem is resolved ! :)

[jcheng5]

So glad to hear it!