Errors when running test dataset

[skudashev]

Hello, I have tried installing skywalker from the Dockerfile you've provided by converting it to a singularity definition file. I have then tried running the test dataset by running

singularity exec ../scywalker_docker.img scywalker -v 1 -d 8 \
    -refdir g17 \
    -sc_expectedcells 183 \
    -cellmarkerfile markers_chr17.tsv \
    -threads 6 \
    test10x

But it errored out and I am struggling to understand the error message. Could you please help me identify the issue here? Kind regards, Sofia process_project_test10x.2024-06-10_16-01-08-258.error.txt isoquant-isoquant_sc-sminimap2_splice-mix1-chr17-21969268-24840903.err.txt scywalker.def

Bootstrap: docker
From: centos:centos7.6.1810

%post
yum -y install wget tar git which fontconfig 
mkdir -p /opt/software
cd /opt/software
wget https://github.com/derijkp/scywalker/releases/download/0.109.0/scywalker-0.109.0-linux-x86_64.tar.gz
tar xvzf scywalker-0.109.0-linux-x86_64.tar.gz
rm scywalker-0.109.0-linux-x86_64.tar.gz
ln -s /opt/software/scywalker-0.109.0-linux-x86_64/scywalker /usr/local/bin/scywalker
ln -s /opt/software/scywalker-0.109.0-linux-x86_64/scywalker_makerefdir /usr/local/bin/scywalker_makerefdir
ln -s /opt/software/scywalker-0.109.0-linux-x86_64/cg /usr/local/bin/cg

%environment
    export PATH=/opt/software/scywalker-0.109.0-linux-x86_64:$PATH

%runscript
    exec /opt/software/scywalker-0.109.0-linux-x86_64/scywalker "$@"

[derijkp]

I tried reproducing the error by running the test data using singularity, first based a conversion of the docker image, then using an image build on your provided definition file, but both ran successfully without problems How did you prepare the reference directory? I used the "default" described in the docs:

singularity exec scywalker.sif scywalker_makerefdir -organelles '' g17 genome.fa genes.gtf

The error message does not give a direct indication where the issue lies: "child process exited abnormally" is given when an external program (in this case isoquant) fails without returning an error message. This can happen e.g. when the program is killed externally. It is unfortunately difficult in such case to say what exactly caused it, e.g. the OOM killer might have killed it for using too much memory, though with the small test data set, this is unlikely in this case.

This may by something temporal (other load): If you rerun the program, does it fail again? at the same point? It may also be local limitations in your singularity configuration. You might try running again using less cores (and thus less memory as well), e.g. using -d 4 Can you try running the application directory directly (default install method) to see if singularity is involved at all: The appdir comes with all dependencies (except some very basic ones such as which) needed and should work on all but very old Linux systems.

[skudashev]

Hello, I have worked out that the error was to do with the way that our HPC is set up so I get a write permission error. I do not get this error when I run isoquant separately.

2024-07-08 12:51:16,399 - CRITICAL - IsoQuant failed with the following error, please, submit this issue to
https://github.com/ablab/IsoQuant/issuesTraceback (most recent call last):
  File "/usr/local/bin/scywalker-0.109.0-linux-x86_64/extra/isoquant-3.3.0-linux-x86_64/bin/isoquant.py", line 698, in <module>
    main(sys.argv[1:])
  File "/usr/local/bin/scywalker-0.109.0-linux-x86_64/extra/isoquant-3.3.0-linux-x86_64/bin/isoquant.py", line 691, in main
    set_additional_params(args)
  File "/usr/local/bin/scywalker-0.109.0-linux-x86_64/extra/isoquant-3.3.0-linux-x86_64/bin/isoquant.py", line 598, in set_additional_params
    set_configs_directory(args)
  File "/usr/local/bin/scywalker-0.109.0-linux-x86_64/extra/isoquant-3.3.0-linux-x86_64/bin/isoquant.py", line 585, in set_configs_directory
    os.makedirs(config_dir, exist_ok=True)
  File "/usr/local/bin/scywalker-0.109.0-linux-x86_64/extra/isoquant-3.3.0-linux-x86_64/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/bin/scywalker-0.109.0-linux-x86_64/extra/isoquant-3.3.0-linux-x86_64/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/bin/scywalker-0.109.0-linux-x86_64/extra/isoquant-3.3.0-linux-x86_64/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/user'

[derijkp]

This allowed me to reproduce the problem: It is caused by having a read-only home directory (Read-only file system: '/home/user'): This goes against Unix/Linux standards, and a lot of applications expect (rightly so) to be able to write configuration and caching data there. In case, Isoquant tries to write to ~/.config (you probably did not get the error running isoquant separately because you did not do the (same) analysis, or used a different version). Because isoquant gets the homedir location from the environment variable HOME, you can use the following workaround to run scywalker regardless (replace /tmp with a directory where you do have write-access):

# make the .config dir first, as IsoQuant does not seem to make it if it does not exist yet
mkdir /tmp/.config 
# change HOME to /tmp for this command only
HOME=/tmp scywalker -stack 1 -v 1 -d 8 \
    -dmaxmem 32G \
    -refdir g17 \
    -sc_expectedcells 183 \
    -cellmarkerfile markers_chr17.tsv \
    -threads 6 \
    test10x

btw. the reason we got no informative error message earlier is caused by isoquant writing it's error message to stdout instead of stderr. I could see it in the individual log data:

less log_jobs/out/isoquant-isoquant_sc-sminimap2_splice-mix1-chr17-0-11039388.out

[skudashev]

Hello, Sorry for taking so long to respond. So I tried running the script as you'd suggested:

mkdir /tmp/.config 
HOME=/tmp singularity exec $scywalker/scywalker_docker.img scywalker -stack 1 -v 1 -d 8 \
    -refdir g17 \
    -sc_expectedcells 183 \
    -cellmarkerfile markers_chr17.tsv \
    -threads 6 \
    test10x

however this didn't change anything. I'm still getting the OSError: [Errno 30] Read-only file system: '/home/user' error, not sure why. I will try talking to our scientific computing team to try and resolve it. Thank you for the suggestion anyway.

Best, Sofia

[derijkp]

Sorry for taking this long to see your followup as well.

putting HOME=/tmp in front does not work if running via singularity. I gave the solution to changing HOME when running directly (but there you probably wouldn't need it). You should be able to change the home dir in singularity using the option -H (https://stackoverflow.com/questions/67893569/changing-the-home-folder-in-a-singularity-container), e.g. using the current wd

mkdir .config
singularity exec -H `pwd` scywalker.sif scywalker -stack 1 -v 1 -d 8 \
    -refdir g17 \
    -sc_expectedcells 183 \
    -cellmarkerfile markers_chr17.tsv \
    -threads 6 \
    test10x

remark: These are actually all singularity problems; you do not have to use singularity. The default binary distribution is portable, meaning it runs as is on most Linux distributions (except extremely old or minimal distributions), comes with all needed dependencies included and is isolated in one directory (does not clobber the installation). i.e. it provides most of the benefits of docker/singularity without some of the disdvantages (mounting/binding, permissions, overhead)

[skudashev]

Hi,

Thank you for getting back to me. I did try running

mkdir .config
singularity exec -H `pwd` scywalker.sif scywalker -stack 1 -v 1 -d 8 \
    -refdir g17 \
    -sc_expectedcells 183 \
    -cellmarkerfile markers_chr17.tsv \
    -threads 6 \
    test10x

which also did not work :/ . I think it might have something to do with Isoquant trying to create files within the container if you don't specify an output path. I will ask if it would be at all possible to install this tool as a binary rather than a container on our cluster.

Thanks a lot, Sofia

[derijkp]

Likely has to do with the singularity settings indeed, you can probably use the --bind option to bind directories, but using the portable binary should be easier: The portable binary does not need to be "installed" by admins: You can just download and unpack the directory (everything needed is contained in that one directory) anywhere you have write access, and run it from there. The description in the README.md describes such a local/user "install" in ~/bin (a subdir bin in your homedir, which you can make if it does not exist yet)