404 message when saving workspace

[timcadman]

datashield.workspace_save(conns, "bmi_poc_sec_2")
Error: Client error: (404) Not Found

[mswertz]

@timcadman is this still the case?

[timcadman]

No I've worked this one out - it was an error triggered by their not being sufficient disk space to save the workspace. It would be helpful if it could state this in the error (though not sure if you have control over this).

[timcadman]

To reproduce

Log into a DS session

demo_url <- "https://armadillo-demo.molgenis.net/"
demo_token <- armadillo.get_token(demo_url)

builder <- DSI::newDSLoginBuilder()

builder$append(
  server = "armadillo",
  url = demo_url,
  profile = "xenon",
  driver = "ArmadilloDriver",
  token = demo_token
)

logindata <- builder$build()
conns <- DSI::datashield.login(logins = logindata, assign = F)

Create an object

ds.rep(x1 = 4,
       times = 10000,
       length.out = NA,
       each = 1,
       source.x1 = "clientside",
       source.times = "c",
       source.length.out = NULL,
       source.each = "c",
       x1.includes.characters = FALSE,
       newobj = "rep.seq",
       datasources = conns)

ds.ls()

'$armadillo $armadillo$environment.searched [1] "R_GlobalEnv"

$armadillo$objects.found [1] "rep.seq"'

Save the workspace and log out

datashield.workspace_save(conns, "test_save")
datashield.logout(conns)

Log back in and see that object still exists

conns <- datashield.login(logindata, restore = "test_save")
ds.ls()

'$armadillo $armadillo$environment.searched [1] "R_GlobalEnv"

$armadillo$objects.found [1] "rep.seq"'

SSH into server and create a file that uses up entire disk space

fallocate -l 10g /usr/share/armadillo/data/test.img

Save the workspace using the same file name

datashield.workspace_save(conns, "test_save") Observe that command executes with no error returned

Log out

datashield.logout(conns)

Remove the file on the server to free up space

rm /usr/share/armadillo/data/test.img

Attempt to log back in restoring workspace with the same name

conns <- datashield.login(logindata, restore = "test_save")

Logging into the collaborating servers Login armadillo [==================================>-----------------------------------] 50% / 0sError: Internal server error: org.molgenis.r.exceptions.RExecutionException: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "{"status":"400","key":"REvaluationRuntimeException","args":[],"message":"Error in base::load(file = \".RData\", envir = .GlobalEnv) : \n empty (zero-byte) input file\n"}"

[clemens-tolboom]

We have to check while saving a workspace we have enough storage

1. org.molgenis.armadillo.controller.DataController#saveUserWorkspace
2. https://github.com/molgenis/molgenis-service-armadillo/blob/e64e2994f245969889e689b8c245d5bee572c6c2/armadillo/src/main/java/org/molgenis/armadillo/command/impl/CommandsImpl.java#L197-L208
3. https://github.com/molgenis/molgenis-service-armadillo/blob/390d683a1f70043790e190e065e9545bf3243e86/armadillo/src/main/java/org/molgenis/armadillo/storage/LocalStorageService.java#L110-L126

[timcadman]

Dick has checked and the largest workspace currently saved on BiB server is 1.3gb

[timcadman]

I don't think there is anyway to know how large the workspace will be prior to saving it. I think this leaves two options:

1. We write in a 'pre-save', e.g. saving to a different file name first and copying across to desired name if succeeds. This would be messy code and take longer, but be most reliable.
2. We start giving warnings/errors if remaining disk space drops below a certain amount (e.g. 2Gb). Disadvantage is this is still vulnerable to people saving exceptionally large files.

[StuartWheater]

You could try the algorithm:

1. Write the shadow workspace
2. Cause a "filesystem sync" to persistent storage
3. If (1) and (2) successful: Remove original workspace
4. If (3) successful: Rename shadow workspace to original workflow
5. Cause a "filesystem sync" to persistent storage
6. Repeat (4) and (5) until successful

This will require the running of a crash recovery process on machine restart, to ensure all shadow workspace have been renamed.