"Completed" but no output files

[mhyleung]

Dear Sir/Madam

I am trying to use FindFungi on my dataset. I have downloaded all the Kraken and blast databases, and have specified my directory paths on the FindFungi-0.23.3.sh and LowestCommonAncestor_V4.sh script files. However, when I put in the command:

./FindFungi-0.23.3.sh joined_input_01.fastq 01_file

It only generates the following

Started at Thu Jan 3 13:35:29 HKT 2019 Finished at Thu Jan 3 13:35:29 HKT 2019

Given then the task finished at the same time, it seems like it actually did not perform the task. There are no output files nor log files for me to see what the issue is, or what it is that I am missing.

How can I go out finding out what exactly the error is when I cannot locate a log file? Thank you!

Regards

Marc

[GiantSpaceRobot]

Hi Marc,

Based on your message, it does seem like the pipeline did not execute any of the steps in the main if statement (lines 32 - 216). My best guess is that the pipeline already generated the results directory (known as $PreDir in the code) on a previous run, so the if statement is not executing any of the code within it. If I'm right, please delete the $PreDir directory and restart the analysis.

If I am wrong, please execute this command and send me the output: find . -type f | xargs ls -lrt > dirs-and-files.log This will tell us the files that the pipeline successfully generated, and might tell us where the problem occurred.

Best, Paul

[mhyleung]

Hi Paul

Thank for you your response. Greatly appreciate your help. I do not think the output was not generated because of a pre-existing directory. I ran it for the first time and it generated nothing.

Also, the command

find . -type f | xargs ls -lrt > dirs-and-files.log

also gave me absolutely nothing, as the output file "dirs-and-files.log" filesize is 0.

I notice that there is this before the script:

###Declare paths for FindFungi analysis, scripts, tools etc.
x=$1
z=$2

Am I supposed to customize this according to my directory paths?

Thanks

Marc

[GiantSpaceRobot]

Hi Marc,

No you do not need to customise that section.

The pipeline is failing to construct the directory substructure necessary to run the analysis. It is difficult for me to determine the issue without seeing it. One possible explanation is that the paths that you declared at the beginning of the script do not point to existing locations. For example, if I was setting the environment up in a directory called /home/user/fungi, I would copy the scripts and databases to subfolders in this directory and do the following:

##### USER INPUT REQUIRED:
ScriptPath=/home/user/fungi/scripts  #Location of downloaded python and shell scripts
PreDir=/home/user/fungi/$z  #Location you want FindFungi to build the analysis
KrakenDir=/home/user/fungi/databases  #Location of the 32 downloaded Kraken databases
FungTaxDir=/home/user/fungi  #Location of the Fungal taxids and PipelineSummary files from GitHub
BLAST_DB_Dir=/home/user/fungi/databases  #Location of the 949 downloaded BLAST databases
#####

Please retry to set these paths. Let me know if I there is anything I can do, thanks.

Best, Paul

[mhyleung]

Hi Paul

Thanks. I edited the script files a bit and it seems to work (at least it is running). Now I seems like I do not have the entire set of dependencies. I will sort that out and will get back to you if I encounter any problems! Thanks!

Marc

[mhyleung]

Hi Paul

So I have managed to get this to start running until the part "### Gather FASTA reads for each taxid predicted." It appears that to get the fasta reads a bsub command is necessary. I tried to remove bsub, but it still needs

-e [dir]/bsub_reports/ReadNames-to-FASTA.$Taxid.stderr

I am not entire sure when the files bsub_reports/ReadNames-to-FASTA.$Taxid.stderr are generated if I remove all the bsubs.

I will create a new issue thread with this specific issue. Thanks

Marcus

[GiantSpaceRobot]

The original issue was closed and a new issue was created for the previous comment.

[Lcornet]

Dear FindFungi creator,

I am trying to use find fungi to identify Fungal fasta bins in a lichen metagenome binning. I have a problem, i have no output with examples data.

./FindFungi-0.23.3.sh /media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3/test-git/ERR675624_both-pairs.fastq ERR675624 Started at Fri Jan 18 12:40:19 CET 2019 Finished at Fri Jan 18 12:40:19 CET 2019

FindFungi-0.23.3.sh

USER INPUT REQUIRED:

ScriptPath=/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3 #Location of downloaded python and shell scripts PreDir=/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3/test #Location you want FindFungi to build the analysis KrakenDir=/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/Kraken_32DB #Location of the 32 downloaded Kraken databases FungTaxDir=/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3/Fungal-Taxids.txt #Location of the Fungal taxids and PipelineSummary files from GitHub BLAST_DB_Dir=/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FungalGenomeDatabases_EqualContigs #Location of the 949 downloaded BLAST databases

LowestCommonAncestor_V4.sh

USER INPUT REQUIRED:

Edit this ScriptPath to point to the directory containing the downloaded scripts

ScriptPath=/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3 #Location of downloaded python and shell scripts

No dirs-and-files.log

Can you help me please ?

[GiantSpaceRobot]

Hi there,

I'm sorry to hear the pipeline is not working for you.

Can you run the following command in your test dir (/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3/) please? find . -type f | xargs ls -lrt > directory-substructure.log Please send me the output (pauldonovandonegal@gmail.com) or post it here and I'll be able to diagnose the problem better. Thank you.

Paul

[Lcornet]

Dear Paul,

Thanks for the help. This the result of : find . -type f | xargs ls -lrt > directory-substructure.log -rwxr-xr-x 1 lcornet phylogeno 2854 Jan 11 14:29 ./CSV-to-WordCloudFormat.py -rwxr-xr-x 1 lcornet phylogeno 2781 Jan 11 14:29 ./CSV-to-Tree.py -rwxr-xr-x 1 lcornet phylogeno 1792 Jan 11 14:29 ./Consensus-CrossRef-Skewness_V2.py -rwxr-xr-x 1 lcornet phylogeno 2395 Jan 11 14:29 ./Skewness-Calculator_V4.py -rwxr-xr-x 1 lcornet phylogeno 1094 Jan 11 14:29 ./ReadNames-to-FASTA_V8.py -rwxr-xr-x 1 lcornet phylogeno 4752 Jan 11 14:29 ./README.md -rwxr-xr-x 1 lcornet phylogeno 174 Jan 11 14:29 ./PipelineSummary.txt -rwxr-xr-x 1 lcornet phylogeno 2333 Jan 11 14:29 ./LowestCommonAncestor_V4.py -rwxr-xr-x 1 lcornet phylogeno 1069 Jan 11 14:29 ./LICENSE -rwxr-xr-x 1 lcornet phylogeno 4154 Jan 11 14:29 ./Kraken32-to-Consensus.py -rwxr-xr-x 1 lcornet phylogeno 6841 Jan 11 14:29 ./Fungal-Taxids.txt -rwxr-xr-x 1 lcornet phylogeno 557 Jan 18 12:23 ./LowestCommonAncestor_V4.sh -rw-r--r-- 1 lcornet phylogeno 1656440823 Jan 18 12:27 ./test-git/ERR675624_1.fastq -rw-r--r-- 1 lcornet phylogeno 1677098343 Jan 18 12:32 ./test-git/ERR675624_2.fastq -rw-r--r-- 1 lcornet phylogeno 3333539166 Jan 18 12:35 ./test-git/ERR675624_both-pairs.fastq -rw------- 1 lcornet phylogeno 81 Jan 18 12:37 ./nohup.out -rw-r--r-- 1 lcornet phylogeno 1426 Jan 18 12:44 ./dirs-and-files.log -rwxr-xr-x 1 lcornet phylogeno 12466 Jan 18 14:21 ./FindFungi-0.23.3.sh -rw-r--r-- 1 lcornet phylogeno 0 Jan 19 02:36 ./directory-substructure.log

Luc

[GiantSpaceRobot]

Hi Luc,

The directory substructure was not created at all by the looks of things. If that is true, then then main pipeline script (FindFungi-0.23.3.sh) could not find the directories you have provided.

Did you run that command in the directory named 'test'? If yes, there is a problem. The path you declared for the scripts was '/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3/', but it appears all of the scripts are in the '/media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/FindFungi/FindFungi-v0.23.3/test/' directory. My best advice I have for you right now is that you must make sure the paths you declared are correct.

You'll know that your paths are correct when a number of new directories are created within your 'test' directory after running the pipeline. Let me know if you are still having this problem.

Best, Paul

[Lcornet]

Dear Paul,

No it was run in the directory with all the script, downloaded from the git. In the test directory , the same command give me : -rw-r--r-- 1 lcornet phylogeno 0 Jan 19 15:08 ./directory-substructure.log This directory is empty I just check all directory (pwd) and it seems to be correct.

[Lcornet]

this is a look into kraken and blast directory, smtg wrong ? [lcornet@node01 FindFungi-v0.23.3]$ cd /media/vol2/scratch/lcornet/nico-project2/metagenome/Fungi/Kraken_32DB/ [lcornet@node01 Kraken_32DB]$ lt total 136 drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_2/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_3/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_6/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_8/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_11/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_12/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_15/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_19/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_20/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_21/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_22/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_23/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_26/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_27/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_28/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_29/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_32/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_1/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_14/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_31/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_17/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_10/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_5/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_16/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_25/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_18/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_7/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_13/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_30/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_4/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_24/ drwxr-xr-x 4 lcornet 4096 Jan 17 2018 Kraken_9/ drwxr-xr-x 34 lcornet 4096 Jan 16 20:58 ./ drwxr-xr-x 6 lcornet 4096 Jan 18 10:52 ../

Taxid-1229662.nhr Taxid-1417759.nin Taxid-226125.nsq Taxid-4829.nhr Taxid-67622.nin Taxid-98765.nsq ect ...

best, Luc

[Lcornet]

When i put some echo "PASS" in FindFungi-0.23.3.sh, it give me that :

it crash after if [ ! -d $PreDir ]; then .

Don't pass by ### Release the Kraken et other ### steps.

Hope it can help

[GiantSpaceRobot]

Hi again,

Okay, the pipeline crashes around the 'if [! -d $PreDir];' statement. It fails to even enter this if statement either because the $PreDir already exists (which does not appear to be the case), or it does not exist and the system has failed to create this directory ('mkdir $PreDir') or your system does not recognise the code.

Please check these questions out. Using 'echo "pass"' like before should be enough to help debug it. What operating system and version are you using?

Paul

[Lcornet]

Hi, I try with and without the PreDir created, it don't enter in the if [! -d $PreDir]. I use a centos : centos-release-6-6.el6.centos.12.2.x86_64 in a terminal. The version is FindFungi-0.23.3.sh.

Luc

[Lcornet]

Finally, i desactivated all my conda links in my bashrc and now it works, don't know why but it work.

[GiantSpaceRobot]

Conda seems to be a consistent issue with the pipeline. They are clashing in some way. Thank you for letting me know that you figured it out. Let me know if you have any more trouble.

Paul