error running mash-docker

[stephens999]

@pcarbo @gaow any ideas on this?

Matthews-MacBook-Air-2:gtexresults stephens$ mash-docker sos run workflows/gtex6_mash_analysis.ipynb
INFO: Executing default_0: Run all analysis in this notebook
INFO: input:   []
INFO: Executing export_0: Export notebook to HTML file
INFO: input:   [file_target('/Users/stephens/git/gtexresults/workflows/gtex6_mash_analysis.ipynb')]
ERROR: [24760332-64e3-4f44-be97-1c12cf526d29]: [export_0]: 
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
script_6673257791059068984 in <module>
      sos convert {_input} {_output[1]} --template sos-report

----> """, stderr = False)

RuntimeError: Failed to execute commmand "/bin/bash /tmp/tmpef907zwx.sh" (ret=2, workdir=/Users/stephens/git/gtexresults, script now in /Users/stephens/git/gtexresults/.sos/export_0_0_506f2cf6.sh)

[stephens999]

this worked ok by the way:

Matthews-MacBook-Air-2:gtexresults stephens$ mash-docker uname -sn
Linux MASH

[gaow]

@stephens999 I cannot reproduce the error but I might have an idea of what is going on -- just to confirm, did you docker pull gaow/mash-paper a while ago or did you do it earlier today?

[stephens999]

A while ago

On Sun, May 27, 2018, 16:05 gaow notifications@github.com wrote:

@stephens999 https://github.com/stephens999 I cannot reproduce the error but I might have an idea of what is going on -- just to confirm, did you docker pull gaow/mash-paper a while ago or did you do it earlier today?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/stephenslab/gtexresults/issues/1#issuecomment-392369188, or mute the thread https://github.com/notifications/unsubscribe-auth/ABt4xQ0efDL1RKYjtmfTscwOLzT9ZZikks5t2xUjgaJpZM4UPasj .

[gaow]

Okay, I'm sorry for the inconvenience, but would you please run docker pull gaow/mash-paper again? We have updated the workflow since, which has resulted in additional software dependencies built into the new docker image.

[stephens999]

i now get new errors: (but i am doing --algorithm mashr)

 mash-docker sos run workflows/gtex6_mash_analysis.ipynb --algorithm mashr
INFO: Executing default_0: Run all analysis in this notebook
INFO: input:   []
INFO: Executing export_0: Export notebook to HTML file
INFO: input:   [file_target('/Users/stephens/git/gtexresults/workflows/gtex6_mash_analysis.ipynb')]
INFO: export_0 (index=0) is ignored due to saved signature
INFO: output:   [file_target('/Users/stephens/git/gtexresults/gtex6_workflow_output/gtex6_mash_analysis.full.html'), file_target('/Users/stephens/git/gtexresults/gtex6_workflow_output/gtex6_mash_analysis.lite.html')]
INFO: Workflow export (ID=00c4a04a85058d63) is ignored with 1 ignored step.
Installing package into '/home/stephens/Dropbox/Rpackages'
(as 'lib' is unspecified)
Bioconductor version 3.4 (BiocInstaller 1.24.0), ?biocLite for help
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.4 (BiocInstaller 1.24.0), R 3.4.4 (2018-03-15).
Installing package(s) 'ExtremeDeconvolution'
WARNING: R Library ExtremeDeconvolution is not available and cannot be installed.
Installing package into '/home/stephens/Dropbox/Rpackages'
(as 'lib' is unspecified)
trying URL 'http://cran.us.r-project.org/src/contrib/devtools_1.13.5.tar.gz'
Content type 'application/x-gzip' length 486555 bytes (475 KB)
==================================================
downloaded 475 KB

* installing *source* package 'devtools' ...
** package 'devtools' successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/home/stephens/Dropbox/Rpackages/digest/libs/digest.so':
  /home/stephens/Dropbox/Rpackages/digest/libs/digest.so: invalid ELF header
ERROR: lazy loading failed for package 'devtools'
* removing '/home/stephens/Dropbox/Rpackages/devtools'
* restoring previous '/home/stephens/Dropbox/Rpackages/devtools'

The downloaded source packages are in
    '/tmp/Rtmpffc6zr/downloaded_packages'
Bioconductor version 3.4 (BiocInstaller 1.24.0), ?biocLite for help
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.4 (BiocInstaller 1.24.0), R 3.4.4 (2018-03-15).
Installing package(s) 'devtools'
trying URL 'https://cran.rstudio.com/src/contrib/devtools_1.13.5.tar.gz'
Content type 'application/x-gzip' length 486555 bytes (475 KB)
==================================================
downloaded 475 KB

* installing *source* package 'devtools' ...
** package 'devtools' successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/home/stephens/Dropbox/Rpackages/digest/libs/digest.so':
  /home/stephens/Dropbox/Rpackages/digest/libs/digest.so: invalid ELF header
ERROR: lazy loading failed for package 'devtools'
* removing '/home/stephens/Dropbox/Rpackages/devtools'
* restoring previous '/home/stephens/Dropbox/Rpackages/devtools'

The downloaded source packages are in
    '/tmp/Rtmpffc6zr/downloaded_packages'
WARNING: R Library devtools is not available and cannot be installed.
Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/home/stephens/Dropbox/Rpackages/digest/libs/digest.so':
  /home/stephens/Dropbox/Rpackages/digest/libs/digest.so: invalid ELF header
Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted
WARNING: Failed to detect or install R library
Installing package into '/home/stephens/Dropbox/Rpackages'
(as 'lib' is unspecified)
trying URL 'http://cran.us.r-project.org/src/contrib/REBayes_1.3.tar.gz'
Content type 'application/x-gzip' length 1506793 bytes (1.4 MB)
==================================================
downloaded 1.4 MB

* installing *source* package 'REBayes' ...
** package 'REBayes' successfully unpacked and MD5 sums checked
** R
** data
*** moving datasets to lazyload DB
** demo
** inst
** preparing package for lazy loading
Error : package 'Matrix' was built for x86_64-apple-darwin13.4.0
ERROR: lazy loading failed for package 'REBayes'
* removing '/home/stephens/Dropbox/Rpackages/REBayes'
* restoring previous '/home/stephens/Dropbox/Rpackages/REBayes'

The downloaded source packages are in
    '/tmp/RtmpRiohDM/downloaded_packages'
Bioconductor version 3.4 (BiocInstaller 1.24.0), ?biocLite for help
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.4 (BiocInstaller 1.24.0), R 3.4.4 (2018-03-15).
Installing package(s) 'REBayes'
trying URL 'https://cran.rstudio.com/src/contrib/REBayes_1.3.tar.gz'
Content type 'application/x-gzip' length 1506793 bytes (1.4 MB)
==================================================
downloaded 1.4 MB

* installing *source* package 'REBayes' ...
** package 'REBayes' successfully unpacked and MD5 sums checked
** R
** data
*** moving datasets to lazyload DB
** demo
** inst
** preparing package for lazy loading
Error : package 'Matrix' was built for x86_64-apple-darwin13.4.0
ERROR: lazy loading failed for package 'REBayes'
* removing '/home/stephens/Dropbox/Rpackages/REBayes'
* restoring previous '/home/stephens/Dropbox/Rpackages/REBayes'

The downloaded source packages are in
    '/tmp/RtmpRiohDM/downloaded_packages'
WARNING: R Library REBayes is not available and cannot be installed.
ERROR: [e11162c0-8db0-4245-b8c8-2a6f4b57b340]: No step to generate target R_library("ExtremeDeconvolution") requested by mash-fast_1

[stephens999]

withouth the --algorithm i have a problem too:

INFO: Executing default_0: Run all analysis in this notebook
INFO: input:   []
INFO: Executing export_0: Export notebook to HTML file
INFO: input:   [file_target('/Users/stephens/git/gtexresults/workflows/gtex6_mash_analysis.ipynb')]
INFO: export_0 (index=0) is ignored due to saved signature
INFO: output:   [file_target('/Users/stephens/git/gtexresults/gtex6_workflow_output/gtex6_mash_analysis.full.html'), file_target('/Users/stephens/git/gtexresults/gtex6_workflow_output/gtex6_mash_analysis.lite.html')]
INFO: Workflow export (ID=bc995c01a4fb6465) is ignored with 1 ignored step.
Installing package into '/home/stephens/Dropbox/Rpackages'
(as 'lib' is unspecified)
Bioconductor version 3.4 (BiocInstaller 1.24.0), ?biocLite for help
BioC_mirror: https://bioconductor.org
Using Bioconductor 3.4 (BiocInstaller 1.24.0), R 3.4.4 (2018-03-15).
Installing package(s) 'ExtremeDeconvolution'
WARNING: R Library ExtremeDeconvolution is not available and cannot be installed.
ERROR: [b0c8c3de-a36c-4054-b97b-fa9d37d6a29d]: No step to generate target R_library("ExtremeDeconvolution") requested by mash-paper_1

[gaow]

Hmm ... you see apparently it complains about not finding ExtremeDeconvolution which should not happen considering we've built that into the docker image!

My theory now is that some of your own R environment configuration is interacting with the docker's environment in a bad way, when $HOME folder is mounted! Let me try to find a way to reproduce then fix it. I think I've got a solution that can surely fix it, but let me try something else (easier) first.

[gaow]

@stephens999 I cannot reproduce it for now, but I added these lines to the docker file:

ENV R_ENVIRON_USER ""
ENV R_PROFILE_USER ""

It should ensure that no configuration is loaded from your own ~/ folder. Would you try

docker pull gaow/mash-paper

and run again? The update should be quick. I certainly hope it will solve the problem. Because I cannot think of other reason why it does not work.

Also would you git pull the new repo to use the flashr based mashr routine? Thanks for trying it out. Very helpful!

[stephens999]

it's running!!

[gaow]

Oh good to know -- a pleasant surprise considering I never have seen that problem before nor could I reproduce on my end.