CIBERSORTx integration fails

[NoahHenrikKleinschmidt]

Hello there once more :-)

As I am aiming to become a regular customer, I have yet again a new issue 😄. Technically it's a CIBERSORTx issue not an EcoTyper issue, but perhaps the forum here can help solve it quicker.

I have finally received my token for CIBERSORTx 🥳, but alas the pipeline fails due to Segmentation Faults from the singularity image 🥺.

Here is some background:

I wish to perform EcoType discovery from bulk-RNASeq data using EcoTyper and CIBERSORTx.

On our cluster we only have Singularity. Hence, I converted the docker images to SIF using

 singularity build cibersortx_fractions.sif  docker://cibersortx/fractions

Next, I specified the appropriate path to the cibersortx_fractions.sif in the EcoTyper config file and submitted the job. I am using just the default Carcinoma setting for CIBERSORTx. However, it appears that some temporary files cannot be written, which causes the JobQueue() to fail repeatedly until the entire CIBERSORTx step crashes.

The repeated "error" is:

Warning message:
In fread(X_file, header = F, sep = "\t") :
  File '/src/outdir//temp.Fractions.coreSVR.X.tsv' has size 0. Returning a NULL data.table.
Warning message:
In fread(Y_file, header = F, sep = "\t") :
  File '/src/outdir//temp.Fractions.coreSVR.Y.tsv' has size 0. Returning a NULL data.table.
Error: $ operator is invalid for atomic vectors
In addition: Warning message:
In mclapply(1:length(nus), res, Y[, j], mc.cores = length(nus)) :
  all scheduled cores encountered errors in user code
Execution halted

which ultimately leads a segmentation fault error as:

sh: line 10: 62182 Segmentation fault      singularity exec -c -B /data/users/noahkleinschmidt/EcoTyper/scratchdir/:/tmp -B /data/users/noahkleinschmidt/EcoTyper/ecotyper/CIBERSORTx/fractions/discovery/TCGA_bulk_discovery/LM22/B_mode/:/src/data -B /data/users/noahkleinschmidt/EcoTyper/ecotyper/CIBERSORTx/fractions/discovery/TCGA_bulk_discovery/LM22/B_mode/:/src/outdir /data/users/noahkleinschmidt/EcoTyper/cibersortx/cibersortx_fractions.sif /src/CIBERSORTxFractions --username noah.kleinschmidt@students.unibe.ch --token 75d719fbb04eb7eae1705c75515c4db9 --mixture /src/data/mixture.txt --sigmatrix /src/data/sigmatrix.txt --sourceGEPs /src/data/sourceGEPs.txt --rmbatchBmode TRUE --verbose TRUE
Error: 
Execution halted
Error: 
Execution halted
Error in RunJobQueue() : 
  EcoTyper failed. Please check the error message above!
Execution halted

The full logs are available via pastecode.io.

You may notice from the command behind the segmentation fault, this is already not 100% the original command. Here are some things I have tried to fix this:

• (re-)setting SINGULARITY_TMPDIR / SINGULARITY_CACHEDIR / SINGULARITY_LOCALCACHEDIR (in any combination, both in the terminal before submitting the slurm job, and as part of the EcoTyper internal call to CIBERSORTx), all to no effect.
• manually binding some directory as /tmp/ (our cluster has a tmp quota, the log from above is from this one), but no effect either.
• raising the ulimit settings to unlimited - no effect.
• pulling the docker image on my local laptop (MacOS) and building the singularity image there using sudo - no effect.
  • while trying this, I also tried re-assigning my local SINGULARITY_{} variables during the build as I understand that _TMPDIR and _CACHEDIR are only used for building.
  • the uploaded SIF images did not behave any different from the ones built on the cluster directly.
  • in fact, they did raise the same segmentation fault when running on my local laptop
• building an entirely new image by first making a sandbox from the docker image (just creates a folder hierarchy), then converting that to SIF - no effect.
  • in this context I also tried writing a wrapper script in the sandbox to first increase ulimit settings before calling on the CIBERSORTxFractions executable. Convert sandbox to SIF, let EcoTyper call the wrapper instead. I just got the error that ulimit modifications were not allowed, so no effect too.
• building a docker sandbox on my laptop and running the CIBERSORTxFractions directly
  • this showed that the temp.{X|Y}.tsv files actually did appear in outdir but only for a brief moment until they disappeared again, followed by the same segmentation fault.

Something I have not yet tried is altering the EcoTyper CIBERSORTx cmd_line to directly call the CIBERSORTxFractions executable within /src/ in my sandbox instead of via singularity. I will try this and post an update.

I have seen that virtually the same issue was already described before but without solution. Thanks a lot already for any inputs 🌻

[diyang1354]

May I ask a question irrelevant to this issue？How long did it take to receive the token? I've been waiting for a month...

[NoahHenrikKleinschmidt]

It was around the same time for me 😅. I emailed the developers a couple of times, but things just take time, I assume. I'm sure they have not forgotten about you, though 🌻

[BALuca]

Hi Noah,

Apologize for this long overdue response. Unfortunately, it might be hard for us to work out why CIBERSORTx fails on your system. It can be either due to singularity configuration issues, or due to the input data. I have just run Tutorial 4 on our cluster, that also only allows for singularity, and it worked perfectly. I was wondering if you are able to run it on your system, rather than your input data. Also, if helpful, I can send you the .sif files that I have been using - they are basically the docker images from the CIBERSORTx website converted to .sif.

Best, The EcoTyper team

[NoahHenrikKleinschmidt]

Heyhow,

Thanks for the reply anyway. I haven't tried the tutorial on our cluster, I guess I should do that - but I'm busy with other stuff at the moment so it might take a while for me to get round to. Thanks for the .sif file offer. I'll first try with our files but if that should fail I'll let you know 👍

Cheers, Noah ☀️

[WangKang-Leo]

Hi Noah,

Apologize for this long overdue response. Unfortunately, it might be hard for us to work out why CIBERSORTx fails on your system. It can be either due to singularity configuration issues, or due to the input data. I have just run Tutorial 4 on our cluster, that also only allows for singularity, and it worked perfectly. I was wondering if you are able to run it on your system, rather than your input data. Also, if helpful, I can send you the .sif files that I have been using - they are basically the docker images from the CIBERSORTx website converted to .sif.

Best, The EcoTyper team

Very appreciated if you can send me (kang.wang@ki.se) two .sif files, I am going to run Tutorial4 in our cluster.

[123Whhhh]

Hello Noah,

I also met the same problem as you. Could you please share these two.sif files with me? This is my email 2022020490@hrbmu.edu.cn

Thank you! wh

[jpark27]

Dear @BALuca and Ecotyper team,

Hi, all! Thank you so much for sharing great tool with community. I come across with same error with @NoahHenrikKleinschmidt using self-built singularity image.

I wonder whether sharing Ecotyper team's sif images (Cibersortx fractions & HiRes) possible? jp27@sanger.ac.uk

best wishes, J