container rebuild and download problems

[colindaven]

Doing now with branch master. We need to check if download functionality works (for me it has never worked in the container).

[colindaven]

Hi @mariusrueve

image downloads still aren't working from the container AFAIK have never worked for anybody.

It just hangs here. Please test! " Please wait! The creation of files (.png and .pdf) can take a while. The download will start, once everything is ready. "

[tglomb]

I can just confirm that the download in the singularity is also not working for me.

[mariusrueve]

Is it possible that the app has no permission to create the files (.pdf + .png) in the wd?

Should I try switching to tempdir again?

owd = setwd(tempdir())
on.exit(setwd(owd))

[mariusrueve]

Just took a look into singularity_app_recipe_dev.txt. Does it have anyimpact, that library(zip) is no part of this?

%runscript
    #!/bin/bash
    exec echo "Container version 0.1 dev August 2020"

    library(shiny) 
    library(Seurat) 
    library(ggplot2) 
    library(readxl)

[colindaven]

Good point. Current container is running off master, not dev, but these both need to be updated.

https://github.com/MHH-RCUG/scrnaseq_app/blob/master/singularity/singularity_app_recipe.txt

[colindaven]

Explicitly loading all the libs as you suggested has not made a difference in my test.

Heres the output.

You can write some code before the figure saving part to test if saving to the current directory works and provide an error message if not. Singularity should be able to write to the current directory (environment var $PWD in Linux). Alternatively, if that fails, you can try writing to

/home/<currentuser>

eg in bash

echo /home/$USER

or

~

ngsssd1/rcug/singularity/sameith_app/marius/scrnaseq_app/singularity$ singularity exec ../../scrnaseq_app.simg     R -e "options('shiny.port'=3838,shiny.host='0.0.0.0'); shiny::runApp('/data/scrnaseq_app/scrnaseq_app.R')"

R version 4.0.3 (2020-10-10) -- "Bunny-Wunnies Freak Out"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> options('shiny.port'=3838,shiny.host='0.0.0.0'); shiny::runApp('/data/scrnaseq_app/scrnaseq_app.R')
Loading required package: shiny

Attaching package: ‘shinyalert’

The following object is masked from ‘package:shiny’:

    runExample

Attaching package: ‘shinyjs’

The following object is masked from ‘package:shinyalert’:

    runExample

The following object is masked from ‘package:shiny’:

    runExample

The following objects are masked from ‘package:methods’:

    removeClass, show

Attaching package: ‘zip’

The following objects are masked from ‘package:utils’:

    unzip, zip

Listening on http://0.0.0.0:3838
Warning: Using `as.character()` on a quosure is deprecated as of rlang 0.3.0.
Please use `as_label()` or `as_name()` instead.
This warning is displayed once per session.
Warning: Error in grid.newpage: could not open file 'FeaturePlot_IFNG_ENSG00000111537.png'
  [No stack trace available]

[mariusrueve]

@colindaven @tglomb Is it a viable option to remove on.exit(unlink(files)) from the code? Through this change the files will not be deleted on exit. If there are still no files in the directory after this change, the problem is the creation of those files. Therefore the zip function has no files to work with in the first place.

[mariusrueve]

In addition I added this:

        if (file.exists(paste0("FeaturePlot_", input$genes[[1]], ".png"))){
          print(paste0("FeaturePlot_", input$genes[[1]], ".png"))
        }

Commit: 021834e7b853d98ca097744dd65fc1cc5c68b23a

I there any way I can test this on/ with my pc?

[colindaven]

Not really, unless you have an linux PC with singularity installed ?

I don't see what your changes are going to tell us about the bug. I would recommend putting in a lot more debugging code, such as querying (with GUI output)

• is current location writable
• creating fig1
• creating fig2
• creating fig3 etc
• writing zip
• check zip exists

You could put these inside a debug boolean if (debug): print lots of debug info

Debug should be a (developer or user-set) variable at the start of the script.

I think those changes might help us a lot more to find out whats going on. I'd rebuild the container after that, if this is ok ?

Colin

[colindaven]

@colindaven @tglomb Is it a viable option to remove on.exit(unlink(files)) from the code? Through this change the files will not be deleted on exit. If there are still no files in the directory after this change, the problem is the creation of those files. Therefore the zip function has no files to work with in the first place.

I don't know. Maybe you can try copying them to the /home/user directory before that .... ? Maybe we also need a singularity container writable global directory on our network infrastructure for this ( /tmp should be ok, but it's not working apparently, or we'll see if you add all that debug code I suggested above).

Thanks Colin

[tglomb]

@colindaven @tglomb Is it a viable option to remove on.exit(unlink(files)) from the code? Through this change the files will not be deleted on exit. If there are still no files in the directory after this change, the problem is the creation of those files. Therefore the zip function has no files to work with in the first place.

What about getting directory/file info during download process after various steps (plot creation, zipping, ...)?
https://stat.ethz.ch/R-manual/R-devel/library/base/html/list.files.html https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/file.info

[mariusrueve]

Added list.files () in dd0d425c27b0edf2b2ad76aeea2bef11354fe53c

Output example:

[1] "Files before zip():"
 [1] "CHANGELOG.md"                              "DotPlot.pdf"                              
 [3] "DotPlot.png"                               "FeaturePlot_NOC2L_ENSG00000188976.png"    
 [5] "FeaturePlot.pdf"                           "README.md"                                
 [7] "RidgePlot_Norm_NOC2L_ENSG00000188976.png"  "RidgePlot_Norm.pdf"                       
 [9] "RidgePlot_Raw_NOC2L_ENSG00000188976.png"   "RidgePlot_Raw.pdf"                        
[11] "scrnaseq_app.R"                            "singularity"                              
[13] "ViolinPlot_Norm_NOC2L_ENSG00000188976.png" "ViolinPlot_Norm.pdf"                      
[15] "ViolinPlot_Raw_NOC2L_ENSG00000188976.png"  "ViolinPlot_Raw.pdf"                       
[17] "www"                                      

[colindaven]

Thanks Marius, container being rebuilt.

[colindaven]

Ok, container running again, please test. Thanks.

Edit: I don't see any changes in the "Files while rendering plots" output. Maybe a permissions problem ?

Where are those files being written , which path? Can we not just write to /home/user as I suggested some time ago, as containers are read only by default. We could create a writable container as well .... ? What is your preference Marius ?

Listening on http://0.0.0.0:3838 [1] "Files while rendering plots:" [1] "CHANGELOG.md" "pbmc_2020-06-08.rds" "README.md" [4] "scrnaseq_app.R" "singularity" "www" Warning: Using as.character() on a quosure is deprecated as of rlang 0.3.0. Please use as_label() or as_name() instead. This warning is displayed once per session. [1] "Files while rendering plots:" [1] "CHANGELOG.md" "pbmc_2020-06-08.rds" "README.md" [4] "scrnaseq_app.R" "singularity" "www" [1] "Files while rendering plots:" [1] "CHANGELOG.md" "pbmc_2020-06-08.rds" "README.md" [4] "scrnaseq_app.R" "singularity" "www" Picking joint bandwidth of 3.4e-06 Picking joint bandwidth of 3.82e-06 Picking joint bandwidth of 3.9e-06 [1] "Files while rendering plots:" [1] "CHANGELOG.md" "pbmc_2020-06-08.rds" "README.md" [4] "scrnaseq_app.R" "singularity" "www" Picking joint bandwidth of 3.4e-06 Picking joint bandwidth of 3.82e-06 Picking joint bandwidth of 3.9e-06 [1] "Files while rendering plots:" [1] "CHANGELOG.md" "pbmc_2020-06-08.rds" "README.md" [4] "scrnaseq_app.R" "singularity" "www" Picking joint bandwidth of 3.4e-06 Picking joint bandwidth of 3.82e-06 Picking joint bandwidth of 3.9e-06 Picking joint bandwidth of 0.379 Picking joint bandwidth of 0.0337 Warning: Error in grid.newpage: could not open file 'FeaturePlot_PLEKHN1_ENSG00000187583.png' [No stack trace available]

[colindaven]

@Oliver-D-B FYI

So I uncommented Marius temp directory code below and committed to the dev branch.

I can build a container out of this with a working download. Yay. Thanks for the code Marius.

       #temporarily switch to the temp dir, in case you do not have write permission to the current working directory
       owd = setwd(tempdir())
       on.exit(setwd(owd))

Please can others briefly test this container download functionality ? Thanks

[Oliver-D-B]

If have just tested it and it also worked for me as well, great job @mariusrueve and @colindaven

[mariusrueve]

Sorry for the late answer! Thanks a lot @colindaven! 🙏🏼 I can confirm that the download is working.