search

ATFutures / upthat

Urban planning and transport health assessment tool
4 stars 1 forks source link

shiny eats memory #10

Closed mpadge closed 4 years ago

mpadge commented 5 years ago

@symbolixAU A mapdeck issue for you, if i may. This repo is a shiny app that uses mapdeck, and has a toggle to switch between cities. Each switch then auto-moves to mapdeck a bunch of paths. The problem is something is aggregating in memory, so each toggle progressively eats up memory. The cause is this (from the "Memory Dominators" report in Firefox): Screenshot from 2019-07-26 13-31-26 This seems to my naive understanding to suggest that the array objects that are eating it are the deck.gl data. Any advice or help would be greatly appreciated!

Here is the server script, and you can reproduce the issues by locally loading big enough data (presuming you're in the root dir of this repo):

f_accra <- "https://github.com/ATFutures/who-data/releases/download/v0.0.5-moveability/accra-moveability-sf.Rds"
f_kath <- "https://github.com/ATFutures/who-data/releases/download/v0.0.5-moveability/kathmandu-moveability-sf.Rds"
download.file (f_accra, file.path (here::here(), "inst", "accra-moveability-sf.Rds")) 
download.file (f_kath, file.path (here::here(), "inst", "kath-moveability-sf.Rds"))

Then just upthat::runUpthat(). (Oh, and you'll first have to ensure your mapbox token is called just "MAPBOX", then it's auto-found so this whole thing can auto-deploy from docker.) Your help in reclaiming the eaten memory would be hugely appreciated! Thanks

Robinlovelace commented 5 years ago

Just to add: it's hard/impossible to open it up on a smartphone. I suspect the size of the network is a cause. No comments on the memory eating... To clarify, are you talking about memory eating on the client side? I imagine so based on the logs from the Firefox web client.

mpadge commented 5 years ago

Yeah, deck.gl doesn't deliver well on mobile devices, but hey, not delivering well is still miles better than leaflet & co, which don't deliver at all

mpadge commented 5 years ago

@symbolixAU I tried with latest dev of all packages (mapdeck, spatialwidget, colouorvalues, jsonify), but the whole thing failed. Downgraded to CRAN mapdeck (0.2.1) then worked again, but it still eats memory. Note that its not a leak, it is clearly just that something - deck.gl? - is holding all the vector data from each click in memory, so memory usage aggregates with each successive click of toggle.

SymbolixAU commented 5 years ago

I've run this a few times and I can't say I'm seeing the issue you're describing. How many times do you toggle between cities before you notice it slow down?

One thing I have found is in the Kathmandu data, row 24869 has a LINESTRING with two coordinates, both of which appear to be the same. Deck.gl errors when it finds this.

options( digits = 8 )
> unclass( net[ 24869, ]$geometry[[1]] )
          [,1]      [,2]
[1,] 85.342642 27.751105
[2,] 85.342642 27.751105

And, you don't need to call clear_path() if you're overwriting the same layer (layer_id) with new data; deck.gl will handle that for you.

I'm also using the latest dev versions of all my packages, and using Chrome

devtools::session_info("mapdeck")
─ Session info ─────────
 setting  value                       
 version  R version 3.6.0 (2019-04-26)
 os       macOS Mojave 10.14.5        
 system   x86_64, darwin15.6.0        
 ui       RStudio                     
 language (EN)                        
 collate  en_AU.UTF-8                 
 ctype    en_AU.UTF-8                 
 tz       Australia/Melbourne         
 date     2019-07-29                  

─ Packages ────────────
 ! package         * version    date       lib source                                   
   BH                1.69.0-1   2019-01-07 [1] CRAN (R 3.6.0)                           
   colourvalues      0.3.0      2019-07-04 [1] Github (SymbolixAU/colourvalues@074fc54) 
   crayon            1.3.4      2017-09-16 [1] CRAN (R 3.6.0)                           
   curl              4.0        2019-07-22 [1] CRAN (R 3.6.0)                           
   digest            0.6.20     2019-07-04 [1] CRAN (R 3.6.0)                           
   geojsonsf         1.3.2      2019-04-28 [1] Github (SymbolixAU/geojsonsf@94eb225)    
   googlePolylines   0.7.2      2018-11-23 [1] CRAN (R 3.6.0)                           
   htmltools         0.3.6      2017-04-28 [1] CRAN (R 3.6.0)                           
   htmlwidgets       1.3        2018-09-30 [1] CRAN (R 3.6.0)                           
   httpuv            1.5.1      2019-04-05 [1] CRAN (R 3.6.0)                           
   jsonify           0.2.10002  2019-07-24 [1] Github (SymbolixAU/jsonify@1313e23)      
   jsonlite          1.6        2018-12-07 [1] CRAN (R 3.6.0)                           
   later             0.8.0      2019-02-11 [1] CRAN (R 3.6.0)                           
   magrittr          1.5        2014-11-22 [1] CRAN (R 3.6.0)                           
   mapdeck         * 0.2.1006   2019-07-28 [1] local                                    
   mime              0.7        2019-06-11 [1] CRAN (R 3.6.0)                           
   promises          1.0.1      2018-04-13 [1] CRAN (R 3.6.0)                           
   R6                2.4.0      2019-02-14 [1] CRAN (R 3.6.0)                           
   rapidjsonr        1.1        2018-09-22 [1] CRAN (R 3.6.0)                           
 V Rcpp              1.0.1      2019-07-25 [1] CRAN (R 3.6.0)                           
   rlang             0.4.0.9000 2019-07-09 [1] Github (r-lib/rlang@a3d7e47)             
   shiny           * 1.3.2      2019-04-22 [1] CRAN (R 3.6.0)                           
   sourcetools       0.1.7      2018-04-25 [1] CRAN (R 3.6.0)                           
   spatialwidget     0.2.1006   2019-07-24 [1] Github (SymbolixAU/spatialwidget@dbe2449)
   xtable            1.8-4      2019-04-21 [1] CRAN (R 3.6.0)                           
   yaml              2.2.0      2018-07-25 [1] CRAN (R 3.6.0)
mpadge commented 4 years ago

Thanks @SymbolixAU. Here's some latest attempts from my side, toggling around half a dozen times in between each memory profile. Chrome: Screenshot from 2019-07-29 09-23-23 And firefox: Screenshot from 2019-07-29 09-26-59

All actually looks okay now, so happy to close. I'll remove the clear_path, which I just inserted in an attempt to possibly improve things - thanks.

Note also that I don't get any error with that linestring, possibly because:

duplicated(as.matrix(net$geometry[[24869]]))
#> FALSE FALSE

The values aren't identical. Thanks for looking into this - much appreciated!