Running modality_alignment nextflow workflow fails

[zsteve]

Hi Robrecht, I'm trying to run the modality_alignment workflow as-is, but it is failing for me. The output is as follows:

[96/e3f220] process > get_scprep_csv_datasets:scprep_csv:scprep_csv_process (CBMC_8K_13AB_10x) [100%] 1 of 1, failed: 1 ✔                                                                                                                                                     
[-        ] process > mnn:mnn_process                                                          -
[-        ] process > scot:scot_process                                                        -
[-        ] process > harmonic_alignment:harmonic_alignment_process                            -
[-        ] process > knn_auc:knn_auc_process                                                  -
[-        ] process > mse:mse_process                                                          -
[30/372c83] process > extract_scores:extract_scores_process (combined)                         [100%] 1 of 1, failed: 1 ✔
[96/e3f220] NOTE: Process `get_scprep_csv_datasets:scprep_csv:scprep_csv_process (CBMC_8K_13AB_10x)` terminated with an error exit status (126) -- Error is ignored
[30/372c83] NOTE: Process `extract_scores:extract_scores_process (combined)` terminated with an error exit status (126) -- Error is ignored

I can look at the nextflow logs located in opsca-viash/work/96/e3f22007b9b1aa0e4fa0ecb3ef09f9:

(base) [syz@r8-geoff1 e3f22007b9b1aa0e4fa0ecb3ef09f9]$ cat .command.err
docker: Error response from daemon: error while creating mount source path '/zfs/users/syz/syz/opsca-viash': mkdir /zfs/users/syz/syz/opsca-viash: permission denied.                                                                                                         
time="2021-04-26T13:16:53-07:00" level=error msg="error waiting for container: context canceled"

I suspect there might be something wrong with my docker installation, but after some googling I am none the wiser. I'm able to run the script script.py directly from opsca-viash/src/modality_alignment/datasets/scprep_csv with no issues (it generates the output.h5ad file as expected).

Any help would be appreciated!

Stephen

[rcannood]

Hmm. Afaik Nextflow is trying to mount a directory /zfs/users/syz/syz/opsca-viash but is not succeeding.

Is the directory /zfs/users/syz/syz/opsca-viash the one you're launching the workflow from? Is it located on a ZFS partition?

[rcannood]

Using the latest commit, does src/modality_alignment/workflows/run_bash.sh work for you?

This is much slower and has several downsides, but it bypasses nextflow so could help debug the problem.

[tverbeiren]

@zsteve I was preparing a bunch of tests for you to perform, but then I found this:

https://docs.docker.com/storage/storagedriver/zfs-driver/

I was not aware ZFS can be this tricky to deal with in combination with Docker.

[rcannood]

Just as a test, can you put opsca-viash on a non-ZFS partition?

[zsteve]

Hi @tverbeiren and @rcannood, I see -- I'm trying to run on my work server at the moment (local machine is quite lightweight and prefer not to use docker on it). I'll see if it's possible to avoid ZFS.

[zsteve]

Using the latest commit, does src/modality_alignment/workflows/run_bash.sh work for you?

This is much slower and has several downsides, but it bypasses nextflow so could help debug the problem.

Also , for the record run_bash.sh has the same issue due to Docker. Seems like this is a Docker/ZFS issue rather than related to this repo. I'll close it for now. Time to nag the sysadmin... :( Thanks for your help though!!