search

aertslab / SCENICprotocol

A scalable SCENIC workflow for single-cell gene regulatory network analysis
GNU General Public License v3.0
143 stars 63 forks source link

Nextflow broken pipe - GRNinference never starts #44

Open Theob0t opened 3 years ago

Theob0t commented 3 years ago

Hi,

First thanks for developing this great tool!

I am trying to run the full SCENIC pipeline using Nextflow and docker on our AWS linux instance (m5.12xlarge) but the pipeline seems to get stuck at the GRNinference step.

There is no error but after a while (~2h stuck at 0%), the instance disconnect and return "client_loop: send disconnect: Broken pipe".

On my local ubuntu, same thing happen, pipeline get stuck at GRNinference.

Do you have any idea what could be the problem?

image

Thanks in advance! I enclosed the CLI log file. Theo

scenic_aws_log.txt

cflerin commented 3 years ago

Hi @Theob0t ,

I've never seen this error before. It looks like it could possibly be a ssh disconnection issue though. Are you running this in a screen/tmux session? This would keep the process running if the ssh connection is broken.

Theob0t commented 3 years ago

Hi, Thanks for the quick answer. I am not running it on screen but it does the same when I run it on my local machine. It seems like the GRNinference step just do not start. I switched to our aws instance to get more threads and memory, but it didn't help. I'll try using screen.

cflerin commented 3 years ago

Hmm interesting. Could you also include the GRNinference logs? They will be in work/69/859bca[...] (the full path is truncated in the log). There will be a .command.log file in there that might have some clues as to what is happening. Is it also using cpu/memory for this step, or it just wont start at all?

Theob0t commented 3 years ago

So I have been running it through screen and could finish the GRNinference step. It crashes after that because I didn't add the motifs for the mouse (it was using the default ones for human). I ran it again with the good motifs yesterday and overnight (motifs-v9-nr.mgi-m0.001-o0.0.tbl), and this time it is stuck at the cisTarget step (0%). Do you know where can I access the log for the cisTarget step? Do you have any idea of how long your pipeline takes to run on a dataset of ~15k cells (I have been running it for ~24h)? Thanks!