Speed up app using arrow ?

[SanderDevisscher]

The lack of speed when starting the app keeps causing displeasure. Do you have any experience with the arrow - package ? maybe it can speed up the start up ?

What do you think ?

[mvarewyck]

What do you think ?

Is it both for the private and public app? I think there are multiple options to speed up the startup of the app:

• Add shinyproxy feature minimumSeatsAvailable: 1. This is a new feature that is available on your ECS deployment. It basically prepares a container that has all the data loaded and 'waits' for the user to take it. Should be straightforward for the public app but will need some modification for the private app. Downside is that you will have permanently one container open and thus pay for it.
• Review the R-code: Evaluate which processes/data files slow down the app using profvis. Given these results, we might consider different file storage (e.g. using arrow), different data processing (e.g. using data.table), preprocess the data and save summarized data on S3 (e.g. for each WBE) or some other changes.
• R package arrow: I have used arrow before for rectangular data files, but not for spatial data and in context of S3, so I can't predict how much time we will gain, but it is certainly worth a try given https://arrow.apache.org/docs/r/articles/fs.html and https://cran.r-project.org/web/packages/sfarrow But I would first want to identify why it is slow.

[SanderDevisscher]

• Ok so you are saying if we switch to "new" hosting technology as currently used by the alien species portal & setting minimumSeatsAvailable: 1 should possibly speed up the app. I would like to test this first with the alien species portal first, which is seriously slow atm. @TheJenne18 also mentioned this when talking about ECS.

• In case the implementation of ECS for reporting grofwild doesn't go as smooth as anticipated or gets delayed like such things tend to do I suggest walking a parallel path of code enhancement. Can you do an attempt to detect what is currently a bottleneck?

[mvarewyck]

• Can you do an attempt to detect what is currently a bottleneck?

Sure, I can

[mvarewyck]

I had a quick look using profvis for the public app. Originally: time before results are displayed is 30s.

• Using parquet files (ie. using arrow) for eco and geo data: gain 13s. Not much faster in loading data, but due to avoiding gc() for data processing - data is not in-memory. This is a serious gain, so I'll investigate how to do this for the spatial data (shapes and schade data)
• The upper map on 'Grofwild' page is rendered multiple times; taking about 6s. I'll look into the R code whether we can prevent the map to be rendered too early (and thus being re-rendered multiple times when more inputs come in)
• All dashboard outputs are rendered upon starting the app, this is for the report to have all outputs even though they are hidden; taking about 5s. However, barDraagkracht() calls take most time -> this might be improved in the R code