Bundled packages but app can't read metadata.rds, re-downloads all packages

[Luke-Symes-Tsy]

I am testing out the new functionality of bundling WebR packages with my r-shinylive app. However, when I try and test it locally offline, it seems to be trying to download at least stringi (and gives some readRDS errors before that):

[Luke-Symes-Tsy]

Very start of loading the app in case it's useful:

Locally I'm also running R 4.4.1. So bit confused about why the metadata RDS seemed to fail to read on WebR 4.4.1, which complains it "cannot read unreleased workspace version -1120096660 written by experimental R -26382.-161.-74".

[georgestagg]

Thanks, would it be possible to also see the shiny source for the app, so I can try to reproduce the problem here?

Do you happen to know which versions of the shinylive R app (output of packageVersion("shinylive")) and the shinylive assets (output of shinylive::assets_info()) you have installed?

Can you try removing the output directory (e.g. "site") and regenerating the site from scratch using e.g. shinylive::export("myapp", "site")? It's possible the pre-downloaded package metadata has become corrupt.

[Luke-Symes-Tsy]

Source code is here: https://github.com/Treasury-Analytics-and-Insights/IncomeExplorer

Locally I'm doing this:

packageVersion("shinylive") [1] ‘0.2.0’ shinylive::assets_info() shinylive R package version: 0.2.0 shinylive web assets version: 0.5.0

There are a lot of dependencies, I'm re-running the export now & will let you know.

As an aside - loading the app (no bundled packages) the first time takes 4 minutes with a fast connection, have seen it take up to 10 minutes.

[Luke-Symes-Tsy]

Just finished the export, and am getting the same error messages.

[georgestagg]

Thanks, I will take a look at the source when I can!

Just finished the export, and am getting the same error messages.

Okay, one last thing to try: Can you try private browsing, or another browser? Or, if you are using Chrome, can you open the Dev Tools, Right click the refesh button, then select "Empty Cache and Hard Reload".

We find this sometimes helps with strange caching issues.

There are a lot of dependencies [...] loading the app the first time takes 4 minutes with a fast connection, have seen it take up to 10 minutes.

Yes, apps with many dependencies will take a long time to launch. At some point every dependency must be downloaded onto the user's machine - it is a downside fundamental to this way of deploying an app. A silver lining is that once deployed an app will cache downloaded packages for that particular deployment URL.

[Luke-Symes-Tsy]

Thanks, I just tried the Chrome "Empty Cache and Hard Reload". Still getting the same errors.

Good to know it will cache for the deployment URL once deployed, we're hoping it's not a huge barrier for people to use it.

[Luke-Symes-Tsy]

Here's a minimal working example:

library(shiny)
library(data.table)
ui <- fluidPage(
  "Hello world"
)
server <- function(input, output, session) {
}
shinyApp(ui, server)

[Luke-Symes-Tsy]

Associated metadata file (zipped) after running shinylive::export() - metadata-before-loading-site.zip

[Luke-Symes-Tsy]

Looks like metadata.rds is successfully read in, but the mounted <package>/Meta/package.rds files are not. This means that installed.packages() fails to detect the bundled packages, so they are downloaded again. Here is a minimal example that outputs the base64enc of an example package.rds file that fails to be read by readRDS:

library(shiny)
library(data.table)
ui <- fluidPage(
  textOutput("problematic_file")
)
server <- function(input, output, session) {
    problematic_file_path <- "/shinylive/webr/packages/data.table/Meta/package.rds"
    test <- tryCatch({readRDS(problematic_file_path)}, error = function(e){warning("We hit the error")})
    output$problematic_file <- renderText(base64enc::base64encode(problematic_file_path))
}
shinyApp(ui, server)

[marcelo-g-simas]

We are experiencing the same issue.

[georgestagg]

Thank you for providing these self-contained examples! Unfortunately both of the examples work OK for me, and so I am still unable to reproduce the problem to debug.

Can you zip up your entire exported app for me to take a look at? Or, otherwise just the site/shinylive/webr/packages directory? This points to .js.metadata corruption, but I fail to see why that would be.

Could you also drop in a comment the output of packageDescription("data.table"), on your local machine, so that I can grab the exact same version?

[Luke-Symes-Tsy]

Here is the site/shinylive/webr/packages directory (full exported app zip is 45MB, too large for GitHub's 25MB limit): packages.zip

Here is the output of packageDescription("data.table") on my local machine:

> packageDescription("data.table")
Package: data.table
Version: 1.15.4
Title: Extension of `data.frame`
Depends: R (>= 3.1.0)
Imports: methods
Suggests: bit64 (>= 4.0.0), bit (>= 4.0.4), R.utils, xts, zoo (>= 1.8-1), yaml, knitr, markdown
Description: Fast aggregation of large data (e.g. 100GB in RAM), fast ordered joins, fast add/modify/delete of columns by group using
               no copies at all, list columns, friendly and fast character-separated-value read/write. Offers a natural and flexible
               syntax, for faster development.
License: MPL-2.0 | file LICENSE
URL: https://r-datatable.com, https://Rdatatable.gitlab.io/data.table, https://github.com/Rdatatable/data.table
BugReports: https://github.com/Rdatatable/data.table/issues
VignetteBuilder: knitr
ByteCompile: TRUE
Authors@R: c( person("Tyson","Barrett", role=c("aut","cre"), email="t.barrett88@gmail.com"), person("Matt","Dowle", role="aut",
               email="mattjdowle@gmail.com"), person("Arun","Srinivasan", role="aut", email="asrini@pm.me"), person("Jan","Gorecki",
               role="aut"), person("Michael","Chirico", role="aut"), person("Toby","Hocking", role="aut", comment =
               c(ORCID="0000-0002-3146-0865")), person("Pasha","Stetsenko", role="ctb"), person("Tom","Short", role="ctb"),
               person("Steve","Lianoglou", role="ctb"), person("Eduard","Antonyan", role="ctb"), person("Markus","Bonsch", role="ctb"),
               person("Hugh","Parsonage", role="ctb"), person("Scott","Ritchie", role="ctb"), person("Kun","Ren", role="ctb"),
               person("Xianying","Tan", role="ctb"), person("Rick","Saporta", role="ctb"), person("Otto","Seiskari", role="ctb"),
               person("Xianghui","Dong", role="ctb"), person("Michel","Lang", role="ctb"), person("Watal","Iwasaki", role="ctb"),
               person("Seth","Wenchel", role="ctb"), person("Karl","Broman", role="ctb"), person("Tobias","Schmidt", role="ctb"),
               person("David","Arenburg", role="ctb"), person("Ethan","Smith", role="ctb"), person("Francois","Cocquemas", role="ctb"),
               person("Matthieu","Gomez", role="ctb"), person("Philippe","Chataignon", role="ctb"), person("Nello","Blaser",
               role="ctb"), person("Dmitry","Selivanov", role="ctb"), person("Andrey","Riabushenko", role="ctb"), person("Cheng","Lee",
               role="ctb"), person("Declan","Groves", role="ctb"), person("Daniel","Possenriede", role="ctb"),
               person("Felipe","Parages", role="ctb"), person("Denes","Toth", role="ctb"), person("Mus","Yaramaz-David", role="ctb"),
               person("Ayappan","Perumal", role="ctb"), person("James","Sams", role="ctb"), person("Martin","Morgan", role="ctb"),
               person("Michael","Quinn", role="ctb"), person("@javrucebo","", role="ctb"), person("@marc-outins","", role="ctb"),
               person("Roy","Storey", role="ctb"), person("Manish","Saraswat", role="ctb"), person("Morgan","Jacob", role="ctb"),
               person("Michael","Schubmehl", role="ctb"), person("Davis","Vaughan", role="ctb"), person("Leonardo","Silvestri",
               role="ctb"), person("Jim","Hester", role="ctb"), person("Anthony","Damico", role="ctb"), person("Sebastian","Freundt",
               role="ctb"), person("David","Simons", role="ctb"), person("Elliott","Sales de Andrade", role="ctb"),
               person("Cole","Miller", role="ctb"), person("Jens Peder","Meldgaard", role="ctb"), person("Vaclav","Tlapak",
               role="ctb"), person("Kevin","Ushey", role="ctb"), person("Dirk","Eddelbuettel", role="ctb"),
               person("Benjamin","Schwendinger", role="ctb"), person("Tony","Fischetti", role="ctb"), person("Ofek","Shilon",
               role="ctb"), person("Vadim","Khotilovich", role="ctb"), person("Hadley","Wickham", role="ctb"),
               person("Bennet","Becker", role="ctb"), person("Kyle","Haynes", role="ctb"), person("Boniface Christian","Kamgang",
               role="ctb"), person("Olivier","Delmarcell", role="ctb"), person("Josh","O'Brien", role="ctb"), person("Dereck","de
               Mezquita", role="ctb"), person("Michael","Czekanski", role="ctb") )
NeedsCompilation: yes
Packaged: 2024-03-28 05:25:20 UTC; tysonbarrett
Author: Tyson Barrett [aut, cre], Matt Dowle [aut], Arun Srinivasan [aut], Jan Gorecki [aut], Michael Chirico [aut], Toby Hocking [aut]
               (<https://orcid.org/0000-0002-3146-0865>), Pasha Stetsenko [ctb], Tom Short [ctb], Steve Lianoglou [ctb], Eduard
               Antonyan [ctb], Markus Bonsch [ctb], Hugh Parsonage [ctb], Scott Ritchie [ctb], Kun Ren [ctb], Xianying Tan [ctb], Rick
               Saporta [ctb], Otto Seiskari [ctb], Xianghui Dong [ctb], Michel Lang [ctb], Watal Iwasaki [ctb], Seth Wenchel [ctb],
               Karl Broman [ctb], Tobias Schmidt [ctb], David Arenburg [ctb], Ethan Smith [ctb], Francois Cocquemas [ctb], Matthieu
               Gomez [ctb], Philippe Chataignon [ctb], Nello Blaser [ctb], Dmitry Selivanov [ctb], Andrey Riabushenko [ctb], Cheng Lee
               [ctb], Declan Groves [ctb], Daniel Possenriede [ctb], Felipe Parages [ctb], Denes Toth [ctb], Mus Yaramaz-David [ctb],
               Ayappan Perumal [ctb], James Sams [ctb], Martin Morgan [ctb], Michael Quinn [ctb], @javrucebo [ctb], @marc-outins [ctb],
               Roy Storey [ctb], Manish Saraswat [ctb], Morgan Jacob [ctb], Michael Schubmehl [ctb], Davis Vaughan [ctb], Leonardo
               Silvestri [ctb], Jim Hester [ctb], Anthony Damico [ctb], Sebastian Freundt [ctb], David Simons [ctb], Elliott Sales de
               Andrade [ctb], Cole Miller [ctb], Jens Peder Meldgaard [ctb], Vaclav Tlapak [ctb], Kevin Ushey [ctb], Dirk Eddelbuettel
               [ctb], Benjamin Schwendinger [ctb], Tony Fischetti [ctb], Ofek Shilon [ctb], Vadim Khotilovich [ctb], Hadley Wickham
               [ctb], Bennet Becker [ctb], Kyle Haynes [ctb], Boniface Christian Kamgang [ctb], Olivier Delmarcell [ctb], Josh O'Brien
               [ctb], Dereck de Mezquita [ctb], Michael Czekanski [ctb]
Maintainer: Tyson Barrett <t.barrett88@gmail.com>
Repository: CRAN
Date/Publication: 2024-03-30 23:50:02 UTC
Built: R 4.4.1; x86_64-w64-mingw32; 2024-08-02 00:32:30 UTC; windows
Archs: x64

-- File: C:/Users/SymesL/AppData/Local/R/win-library/4.4/data.table/Meta/package.rds

[georgestagg]

Thank you for your patience and for uploading these files, I finally know what is going on!

Your copy of the Wasm R package data has been written with Windows-style line endings. That is, \n characters have been replaced with \r\n! I guess you are using a Windows machine? The change breaks the package metadata because it works based on offsets into the .data file.

I think this is probably because we were using R's download.file() function without setting mode = "wb". On Unix and macOS machines it does not matter, but on Windows it does. The documentation even warns about this, d'oh!

I have made a change to the Shinylive R package to enforce downloading assets as a binary format. Please can you try installing the latest development version of the Shinylive package from GitHub with:

pak::pak("posit-dev/r-shinylive")

Then, after restarting R, removing your output site directory and trying to regenerate from scratch once more using e.g. shinylive::export("myapp", "site")?

[marcelo-g-simas]

In our case the issue was that the app we were bundling depended on an internal package we manage that is not on CRAN. We worked around it by including the code we were depending on in the ShinyLive application and that made the publish/build process generate artifacts that worked.

If you can point us towards resources on how to solve this w/o bundling the dependency's code we can give it a try in a future project.

[Luke-Symes-Tsy]

Thank you for your patience and for uploading these files, I finally know what is going on!

Your copy of the Wasm R package data has been written with Windows-style line endings. That is, \n characters have been replaced with \r\n! I guess you are using a Windows machine? The change breaks the package metadata because it works based on offsets into the .data file.

I think this is probably because we were using R's download.file() function without setting mode = "wb". On Unix and macOS machines it does not matter, but on Windows it does. The documentation even warns about this, d'oh!

I have made a change to the Shinylive R package to enforce downloading assets as a binary format. Please can you try installing the latest development version of the Shinylive package from GitHub with:

pak::pak("posit-dev/r-shinylive")

Then, after restarting R, removing your output site directory and trying to regenerate from scratch once more using e.g. shinylive::export("myapp", "site")?

Just tested this on my minimal example, and it works now! Many thanks for tracking down the cause and fixing it.

[georgestagg]

We worked around it by including the code we were depending on in the ShinyLive application and that made the publish/build process generate artifacts that worked.

Glad you found a solution that works for you!

If you can point us towards resources on how to solve this w/o bundling the dependency's code we can give it a try in a future project.

The easiest way to build custom R packages for WebAssembly is to create an R-Universe personal repository. R-Universe will automatically build R packages that can be used with webR and R Shinylive.

---

Just tested this on my minimal example, and it works now!

Great to hear! I'll close this issue as resolved.