As Armadillo developer I want to understand why Armadillo crashes when I load a large workspace (BLOCKED: Rock)

[marikaris]

Demo:

• Armadillo doesn't break anymore when loading large resource

Acceptance criteria:

• Save large workspace
• Try to reload it to reproduce issue with breaking armadillo
• Find out why it's breaking (something to do with something not streaming?)
• Fix it
• Acceptable to gracefully crash if things get really complicated

todo:

• [x] reproduced in R
• [ ] reproduced in Opal

Code for comparing Opal and Armadillo (in progress)

library(DSMolgenisArmadillo)
library(DSI)
library(DSOpal)
library(MolgenisAuth)
library(devtools)
# install_github("molgenis/molgenis-r-auth")
library(MolgenisAuth)
library(dsBaseClient)

armadillo_url <- "http://localhost:8080/"
opal_url <- "https://demo-opal.molgenis.net/repo"

token <- armadillo.get_token(armadillo_url)

builder <- newDSLoginBuilder()

builder$append(
  url = armadillo_url,
  server = "armadillo",
  token = token,
  driver = "ArmadilloDriver")

builder$append(
  url = opal_url,
  server = "opal",
  user = "administrator",
  password = [in vault: 'customers/armadillo/demo-opal.molgenis.net/OPAL_ADMINISTRATOR_PASSWORD'],
  driver = "OpalDriver")

logindata <- builder$build()

conns <- datashield.login(logins = logindata, assign = FALSE)

datashield.assign.table(conns["armadillo"], "non_rep", "lifecycle/core/nonrep")
datashield.assign.table(conns["opal"], "non_rep", "test-workspace-save.nonrep")

ds.colnames("non_rep")

ds.glmSLMA(
  formula = "preg_ht ~ 1 + agebirth_p_d + dia_bf + sibling_pos + breastfed_ever + cats_quant_preg", 
  family = "gaussian",
  dataName = "non_rep",
  newobj = "reg_out")

for(i in 1:10000) {
  datashield.assign.expr(conns, paste0("reg_out", i), "reg_out")
}

mem_used_before_save <- ds.memUsed(conns)
datashield.workspace_save(conns, "test_workspace")

conns <- datashield.login(logins = logindata, restore = "test_workspace")
mem_used_after_save <- ds.memUsed(conns)

Remaining to do:

This will work up to the for loop. For the rest to work:

• In opal, install the package we made with the test memory function (https://github.com/timcadman/dsBase/tree/feat/mem-used/R, ref = "feat/mem-used"). Might want to check first that works, or maybe Mariska has a local copy with just the one function included.
• Install locally the clientside version of this package from the repo (https://github.com/timcadman/dsBaseClient, ref = "feat/mem-used) or if it isn't working then Mariska's version

[marijevdgeest]

Probably an R issue