Missing required dependency [projected_paga]

FloWuenne commented 5 years ago

Hi dyno team,

After managing to run dyno on our HPC, I am running into a weird dependency issue when trying to run "projected_paga" on the test dataset. The same code works fine when using a different method, for example "angle". My colleague is able to run the exact same analysis on the same HPC, leading me to believe it must be related to some python packages I have previously installed and he has not. However, I don't fully understand why dyno is trying to use my local python instead of the python inside the singularity container.

Please find the code and error below:

library(dyno) library(tidyverse)

data("fibroblast_reprogramming_treutlein")

dataset <- wrap_expression( counts = fibroblast_reprogramming_treutlein$counts, expression = fibroblast_reprogramming_treutlein$expression )

Executing 'projected_paga' on '20190409_063127__data_wrapper__ygGdphzfxh'
With parameters: list(n_neighbors = 15L, n_comps = 50L, n_dcs = 15L, resolution = 1L,     embedding_type = "fa", connectivity_cutoff = 0.05),
inputs: counts, and
priors : 
Input saved to /tmp/RtmpuyJ9Xp/file2a3f56e8a7d33/ti
Running /opt/software/singularity-3.1/bin/singularity run --pwd /ti/workspace \
  -B \
  '/tmp/RtmpuyJ9Xp/file2a3f56e8a7d33/ti:/ti,/tmp/RtmpuyJ9Xp/file2a3f514c36879/tmp:/tmp2' \
  'docker://dynverse/ti_projected_paga:v0.9.9.01' --dataset /ti/input.h5 \
  --output /ti/output.h5
WARNING: Could not set container working directory /ti/workspace: chdir /ti/workspace: no such file or directory
Traceback (most recent call last):
  File "/code/run.py", line 3, in <module>
    import dynclipy
  File "/usr/local/lib/python3.7/site-packages/dynclipy/__init__.py", line 3, in <module>
    from rpy2.robjects import pandas2ri
  File "/usr/local/lib/python3.7/site-packages/rpy2/robjects/pandas2ri.py", line 13, in <module>
    from pandas.core.frame import DataFrame as PandasDataFrame
  File "/home/wueflo00/.local/lib/python3.7/site-packages/pandas/__init__.py", line 19, in <module>
    "Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['numpy']
Error: Error during trajectory inference 
WARNING: Could not set container working directory /ti/workspace: chdir /ti/workspace: no such file or directory
Traceback (most recent call last):
  File "/code/run.py", line 3, in <module>
    import dynclipy
  File "/usr/local/lib/python3.7/site-packages/dynclipy/__init__.py", line 3, in <module>
    from rpy2.robjects import pandas2ri
  File "/usr/local/lib/python3.7/site-packages/rpy2/robjects/pandas2ri.py", line 13, in <module>
    from pandas.core.frame import DataFrame as PandasDataFrame
  File "/home/wueflo00/.local/lib/python3.7/site-packages/pandas/__init__.py", line 19, in <module>
    "Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['numpy']

zouter commented 5 years ago

Hi Florian

Default behaviour of singularity is indeed to mount certain directories, such as the home directory. This may be useful when using singularity as a "virtual workspace", but isn't really desirable for our usecase. I pushed a hotfix disabling this behaviour to dynverse/babelwhale, could you try to reinstall it devtools::install_github("dynverse/babelwhale") and see whether it works?

FloWuenne commented 5 years ago

Hi @zouter

I reinstalled devtools::install_github("dynverse/babelwhale") today after the hotfix and this is working great without a problem now!

model <- infer_trajectory(dataset, c("projected_paga"), verbose = TRUE, debug = TRUE)

Executing 'projected_paga' on '20190410_065943__data_wrapper__CXl1D2dBub'
With parameters: list(n_neighbors = 15L, n_comps = 50L, n_dcs = 15L, resolution = 1L,     embedding_type = "fa", connectivity_cutoff = 0.05),
inputs: counts, and
priors : 
Input saved to /tmp/RtmptPGyiZ/file1457568fc8d0b/ti
Running /opt/software/singularity-3.1/bin/singularity run --containall --pwd \
  /ti/workspace -B \
  '/tmp/RtmptPGyiZ/file1457568fc8d0b/ti:/ti,/tmp/RtmptPGyiZ/file145755099071d/tmp:/tmp2' \
  'docker://dynverse/ti_projected_paga:v0.9.9.01' --dataset /ti/input.h5 \
  --output /ti/output.h5
/usr/local/lib/python3.7/site-packages/louvain/Optimiser.py:349: SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(issubclass(partition_type, LinearResolutionParameterVertexPartition),

Dyno test

It is noteworthy that when I initially ran it with debug == TRUE, I still get an error, so there might be some colliding with the debug feature and the hotfix?!

model <- infer_trajectory(dataset, c("projected_paga"), verbose = TRUE, debug = TRUE)

Executing 'projected_paga' on '20190410_065943__data_wrapper__CXl1D2dBub'
With parameters: list(n_neighbors = 15L, n_comps = 50L, n_dcs = 15L, resolution = 1L,     embedding_type = "fa", connectivity_cutoff = 0.05),
inputs: counts, and
priors : 
Input saved to /tmp/RtmptPGyiZ/file1457568f78ac9/ti
Error: Error during trajectory inference 
Use this command for debugging:
...

zouter commented 5 years ago

Hi @FloWuenne

Glad that it the method works for you! :clap:

As for the debug parameter: this is actually the intended functionality. Debug gives you one or more commands that you have to run inside a terminal so that you can enter the container. I agree that it can be confusing though with the error during trajectory inference, future versions of dynwrap won't have this error anymore!

Wouter

dynverse / dyno

Missing required dependency [projected_paga] #46