search

exaexa / scattermore

very fast scatterplots for R
https://exaexa.github.io/scattermore/
GNU General Public License v3.0
243 stars 7 forks source link

Odd slowdown/crash with `coord_cartesian` in {shiny} with `geom_scattermost` #3

Closed davemcg closed 4 years ago

davemcg commented 4 years ago

This may be some ggplot or shiny oddity, but I thought I'd just bring it to your attention. As it is borking my app.

Zoom example adapted from: https://gallery.shinyapps.io/105-plot-interaction-zoom/

What I've learned:

  1. only an issue with scattermost
  2. Top left "zoom" is fast.
  3. Bottom right "zoom" is crazy slow or crashes computer
  4. ONLY HAPPENS WHEN pointsize is above 0 
    
library(ggplot2)
library(Cairo)   # For nicer ggplot2 output when deployed on Linux
library(scattermore)
d <- cbind(rnorm(5e5),rnorm(5e5)) %>% data.frame()
colnames(d) <- c('X','Y')
ui <- fluidPage(
fluidRow(
column(width = 4, class = "well",
       h4("Brush and double-click to zoom"),
       plotOutput("plot1", height = 300,
                  dblclick = "plot1_dblclick",
                  brush = brushOpts(
                    id = "plot1_brush",
                    resetOnNew = TRUE
                  )
       )
)
)
)

server <- function(input, output) {

-------------------------------------------------------------------

Single zoomable plot (on left)

ranges <- reactiveValues(x = NULL, y = NULL)

output$plot1 <- renderPlot({ ggplot() + geom_scattermost(cbind(d[,1],d[,2]), pointsize = 3) + coord_cartesian(xlim = ranges$x, ylim = ranges$y, expand = FALSE) })

When a double-click happens, check if there's a brush on the plot.

If so, zoom to the brush bounds; if not, reset the zoom.

observeEvent(input$plot1_dblclick, { brush <- input$plot1_brush if (!is.null(brush)) { ranges$x <- c(brush$xmin, brush$xmax) ranges$y <- c(brush$ymin, brush$ymax)

} else {
  ranges$x <- NULL
  ranges$y <- NULL
}

})

}

shinyApp(ui, server)


# Session Info

devtools::session_info() ─ Session info ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── setting value
version R version 3.6.2 (2019-12-12) os macOS Mojave 10.14.6
system x86_64, darwin15.6.0
ui RStudio
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/New_York
date 2020-04-20

─ Packages ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── package version date lib source
ape 5.3 2019-03-17 [1] CRAN (R 3.6.0)
assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.0)
backports 1.1.5 2019-10-02 [1] CRAN (R 3.6.0)
BiocManager 1.30.10 2019-11-16 [1] CRAN (R 3.6.0)
bit 1.1-15.2 2020-02-10 [1] CRAN (R 3.6.0)
bit64 0.9-7 2017-05-08 [1] CRAN (R 3.6.0)
blob 1.2.1 2020-01-20 [1] CRAN (R 3.6.0)
broom 0.5.5 2020-02-29 [1] CRAN (R 3.6.0)
Cairo 1.5-12 2020-04-11 [1] CRAN (R 3.6.2)
callr 3.4.2 2020-02-12 [1] CRAN (R 3.6.0)
cellranger 1.1.0 2016-07-27 [1] CRAN (R 3.6.0)
cli 2.0.2 2020-02-28 [1] CRAN (R 3.6.0)
colorspace 1.4-1 2019-03-18 [1] CRAN (R 3.6.0)
cowplot 1.0.0 2019-07-11 [1] CRAN (R 3.6.0)
crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.0)
crosstalk 1.0.0 2016-12-21 [1] CRAN (R 3.6.0)
DBI 1.1.0 2019-12-15 [1] CRAN (R 3.6.0)
dbplyr 1.4.2 2019-06-17 [1] CRAN (R 3.6.0)
desc 1.2.0 2018-05-01 [1] CRAN (R 3.6.0)
devtools 2.2.2 2020-02-17 [1] CRAN (R 3.6.0)
dichromat 2.0-0 2013-01-24 [1] CRAN (R 3.6.0)
digest 0.6.25 2020-02-23 [1] CRAN (R 3.6.0)
dplyr 0.8.4 2020-01-31 [1] CRAN (R 3.6.0)
DT 0.12 2020-02-05 [1] CRAN (R 3.6.0)
ellipsis 0.3.0 2019-09-20 [1] CRAN (R 3.6.0)
fansi 0.4.1 2020-01-08 [1] CRAN (R 3.6.0)
farver 2.0.3 2020-01-16 [1] CRAN (R 3.6.0)
fastmap 1.0.1 2019-10-08 [1] CRAN (R 3.6.0)
forcats 0.5.0 2020-03-01 [1] CRAN (R 3.6.2)
fs 1.3.1 2019-05-06 [1] CRAN (R 3.6.0)
generics 0.0.2 2018-11-29 [1] CRAN (R 3.6.0)
ggplot2 3.3.0 2020-03-05 [1] CRAN (R 3.6.0)
ggrepel 0.9.0 2020-04-15 [1] Github (slowkow/ggrepel@3941cf1)
ggtree 2.1.3 2020-03-22 [1] Github (YuLab-SMU/ggtree@cd229bf)
glue 1.3.1 2019-03-12 [1] CRAN (R 3.6.0)
gridExtra 2.3 2017-09-09 [1] CRAN (R 3.6.0)
gtable 0.3.0 2019-03-25 [1] CRAN (R 3.6.0)
haven 2.2.0 2019-11-08 [1] CRAN (R 3.6.0)
hms 0.5.3 2020-01-08 [1] CRAN (R 3.6.0)
htmltools 0.4.0 2019-10-04 [1] CRAN (R 3.6.0)
htmlwidgets 1.5.1 2019-10-08 [1] CRAN (R 3.6.0)
httpuv 1.5.2 2019-09-11 [1] CRAN (R 3.6.0)
httr 1.4.1 2019-08-05 [1] CRAN (R 3.6.0)
jsonlite 1.6.1 2020-02-02 [1] CRAN (R 3.6.0)
labeling 0.3 2014-08-23 [1] CRAN (R 3.6.0)
later 1.0.0 2019-10-04 [1] CRAN (R 3.6.0)
lattice 0.20-38 2018-11-04 [1] CRAN (R 3.6.2)
lazyeval 0.2.2 2019-03-15 [1] CRAN (R 3.6.0)
lifecycle 0.1.0 2019-08-01 [1] CRAN (R 3.6.0)
lubridate 1.7.4 2018-04-11 [1] CRAN (R 3.6.0)
magrittr 1.5 2014-11-22 [1] CRAN (R 3.6.0)
mapproj 1.2.7 2020-02-03 [1] CRAN (R 3.6.0)
maps 3.3.0 2018-04-03 [1] CRAN (R 3.6.0)
memoise 1.1.0 2017-04-21 [1] CRAN (R 3.6.0)
mime 0.9 2020-02-04 [1] CRAN (R 3.6.0)
modelr 0.1.6 2020-02-22 [1] CRAN (R 3.6.0)
munsell 0.5.0 2018-06-12 [1] CRAN (R 3.6.0)
nlme 3.1-142 2019-11-07 [1] CRAN (R 3.6.2)
pals 1.6 2019-12-04 [1] CRAN (R 3.6.0)
patchwork 1.0.0 2019-12-01 [1] CRAN (R 3.6.0)
pillar 1.4.3 2019-12-20 [1] CRAN (R 3.6.0)
pkgbuild 1.0.6 2019-10-09 [1] CRAN (R 3.6.0)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 3.6.0)
pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.6.0)
pool 0.1.4.3 2019-10-03 [1] CRAN (R 3.6.0)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 3.6.0)
processx 3.4.2 2020-02-09 [1] CRAN (R 3.6.0)
promises 1.1.0 2019-10-04 [1] CRAN (R 3.6.0)
ps 1.3.2 2020-02-13 [1] CRAN (R 3.6.0)
purrr 0.3.3 2019-10-18 [1] CRAN (R 3.6.0)
R6 2.4.1 2019-11-12 [1] CRAN (R 3.6.0)
Rcpp 1.0.3 2019-11-08 [1] CRAN (R 3.6.0)
readr 1.3.1 2018-12-21 [1] CRAN (R 3.6.0)
readxl 1.3.1 2019-03-13 [1] CRAN (R 3.6.0)
remotes 2.1.1 2020-02-15 [1] CRAN (R 3.6.0)
reprex 0.3.0 2019-05-16 [1] CRAN (R 3.6.0)
rlang 0.4.4 2020-01-28 [1] CRAN (R 3.6.0)
rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.0)
RSQLite 2.2.0 2020-01-07 [1] CRAN (R 3.6.0)
rstudioapi 0.11 2020-02-07 [1] CRAN (R 3.6.0)
rvcheck 0.1.8 2020-03-01 [1] CRAN (R 3.6.2)
rvest 0.3.5 2019-11-08 [1] CRAN (R 3.6.0)
scales 1.1.0 2019-11-18 [1] CRAN (R 3.6.0)
scattermore 0.6 2020-02-09 [1] Github (exaexa/scattermore@c3968fc) sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.0)
shiny 1.4.0 2019-10-10 [1] CRAN (R 3.6.0)
shinythemes 1.1.2 2018-11-06 [1] CRAN (R 3.6.0)
stringi 1.4.6 2020-02-17 [1] CRAN (R 3.6.0)
stringr 1.4.0 2019-02-10 [1] CRAN (R 3.6.0)
svgPanZoom 0.3.4 2020-02-15 [1] CRAN (R 3.6.0)
testthat 2.3.1 2019-12-01 [1] CRAN (R 3.6.0)
tibble 2.1.3 2019-06-06 [1] CRAN (R 3.6.0)
tictoc 1.0 2014-06-17 [1] CRAN (R 3.6.0)
tidyr 1.0.2 2020-01-24 [1] CRAN (R 3.6.0)
tidyselect 1.0.0 2020-01-27 [1] CRAN (R 3.6.0)
tidytree 0.3.2 2020-03-12 [1] CRAN (R 3.6.0)
tidyverse 1.3.0 2019-11-21 [1] CRAN (R 3.6.0)
treeio 1.10.0 2019-10-29 [1] Bioconductor
usethis 1.5.1 2019-07-04 [1] CRAN (R 3.6.0)
vctrs 0.2.3 2020-02-20 [1] CRAN (R 3.6.0)
viridis 0.5.1 2018-03-29 [1] CRAN (R 3.6.0)
viridisLite 0.3.0 2018-02-01 [1] CRAN (R 3.6.0)
withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.0)
xml2 1.2.2 2019-08-09 [1] CRAN (R 3.6.0)
xtable 1.8-4 2019-04-21 [1] CRAN (R 3.6.0)
yaml 2.2.1 2020-02-01 [1] CRAN (R 3.6.0)

[1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

davemcg commented 4 years ago

I can work around by using scattermore, but would prefer using scattermost as it is 50-100% faster. Except when I zoom in the bottom right. Haven't fully quantified, but it seems to get slower as you go right and down.

As a stupid fix I tried making the coordinates all positive. Made no difference. Some kind of buffer overflow thing? I'm way outside my comfort zone....

exaexa commented 4 years ago

Hello! would you be able to run R with a debugger and produce a backtrace from the crash?

  1. you need some unix, Mac probably works too (you will need to install gdb). I have no idea how to debug R on windows
  2. instead of R run R -d gdb; after executing it write a single r to the GDB shell (followed by enter) to start the actual R
  3. when the crash happens, type bt full in the gdb console and paste the whole output.

I'll try to reproduce the issue here ASAP. It seems like out-of-bounds array access, which is IMHO very possible.

Also, thanks for the report!

davemcg commented 4 years ago

I think I'm wrong on the crash part - the slowdown was so dramatic I assumed it had crashed. But if I wait minutes it finishes.

exaexa commented 4 years ago

I am able to reproduce the problem; guessing it originates in some (weird) kind of integer over/underflow. scattermore internally uses integer arithmetic only for speed, the downside is the amount of corner cases.

Will know more in a few minutes.

exaexa commented 4 years ago

PS. this is the first time in ~20 years of programming when a corner case happens in actual corner. :tada: :fireworks:

davemcg commented 4 years ago

So glad you verified it. Took me quite a while to reproduce outside my app.

I started with the simple examples and could only get the damn thing to happen in a {shiny} app.

davemcg commented 4 years ago

For example this seems fine

library(tictoc)
library(scattermore)
library(ggplot2)
d <- cbind(rnorm(5e5),rnorm(5e5))
color_range <- range(d[,1])
scattermost_plot <- ggplot() +
  geom_scattermost(cbind(d[,1],d[,2]), pointsize = 10,
                   color = viridis::magma(100, alpha=0.3)
                   [1+99*(d[,1]-color_range[1])/diff(color_range)]) +
  ggtitle("geom_scattermost")
tic(); scattermost_plot; toc() #3.1 seconds
tic(); scattermost_plot + coord_cartesian(xlim=c(-5,-3),ylim=c(3,5)); toc() #1.2 seconds
tic(); scattermost_plot + coord_cartesian(xlim=c(3,5),ylim=c(-3,-5)); toc() #1.9 seconds
exaexa commented 4 years ago

Seems very much like a problem with the radii computation and integer overflows, it actually crashes. Will fix ASAP.

exaexa commented 4 years ago

Can you please try the version from git now?

devtools::install_github('exaexa/scattermore');
exaexa commented 4 years ago

Anyway, this was a mistake in integer logic as expected. I'll convert the whole thing to a similar system as GigaScatter soon (see https://github.com/LCSB-BioCore/GigaScatter.jl -- it expands the points in a separate phase, to get quite a bit of speedup for large number of big points). That should avoid this kind of trouble for good.

davemcg commented 4 years ago

BOOM YES THANK YOU SO FAST

exaexa commented 4 years ago

:boom: O GREAT

Thanks for the report!