Efficiency of Census Tract File Load

[lindsaypoirier]

Swapping between states is currently quite slow because the all of the census tract shapefiles are not set up load by default, but instead are set up to be unzipped and extracted when a new state is selected.

See comments in the server function for more detail on how this is working.

I'm not sure this is the most efficient way to do things. So any help with speeding this up would be appreciated.

[lindsaypoirier]

Try unzipping files by default

[lindsaypoirier]

After some testing, it seems as though it's not the unzipping that's slowing things down, but the loading of the shape file. Not sure if there's any libraries that could speed up shape file reading.

[benmillam]

Update:

• On my machine I used the profvis package to profile the app
• The census tract loading takes ~1 second
• The plotting of the census polygons takes about ~2.5 seconds (for California, ~8K tracts)
• The app still takes 15-30 seconds total on my machine when selecting e.g. CA as a state!
• My machine has a major slowdown in a call to renderWidget() that's part of shiny dealing with output$dist_plot, in which it appears to spend a lot of time converting data types for some kind of JSON format

This is my interpretation of the profvis profile, however, when I remove the census tracts from the leaflet plot (remove the addPolygons() call), the app is fast again (in line with @lindsaypoirier's original observation of the census tracts being the issue.)

Possible next steps:

• dig into why renderWidget() is taking so long (I hesitate, because I can't find anyone experiencing similar issues on this specific front, at least on a cursory search)
• confront the possibility that ~8K polygons is too much for Shiny (a comment I read somewhere said 20k-30k is max in their experience); note, I tried rmapshaper::ms_simplify to simplify the polygons, and that may be a marginal performance tweak to include later
• consult someone with more Shiny/leaflet/profvis experience; this is my first time interpreting a profvis profile; or I can post on StackOverflow

[benmillam]

Here's an annotated screenshot of my app profvis profile: If interested, I'd recommend running profiles on your machine (or the server); (removing the tidyverse piping %>% from the plotting helped for clarity in the profvis output, too.)

[benmillam]

Update: during Tues meeting, Ben to post on RStudio Community