search

faircloth-lab / phyluce

software for UCE (and general) phylogenomics
http://phyluce.readthedocs.org/
Other
76 stars 48 forks source link

ABySS/Velvet IndexError: list index out of range: ABySS cannot find abyss-pe.Makefile #194

Closed Lostinthecodingsauce closed 4 years ago

Lostinthecodingsauce commented 4 years ago

Hi Brant,

I am having a lot of problems getting abyss and velvet to run on a cluster. When I try and run either program I receive the error: return glob.glob(os.path.join(output, "*-contigs.fa"))[0] IndexError: list index out of range

The log files for both programs are empty except for abyss.err.log which states which: no abyss-pe.Makefile in (/rhome/bfscott1906/bigdata/.conda/envs/phyluce/bin)

The weird thing is that I was able to a) successful complete a single individual in ABySS before the error message appears, b) have found the abyss-pe.Make in my bin (and have the correct $PATH), and c) was able to successfully run the same individuals on a 12 core computer without any of the problems.

So far I have tried; 1) Running abyss-pe on a single species (on both the individual that worked and the individual that did not, both received the error no abyss-pe.Makefile) 2) Create conda environment to use the older version of phyluce to run Trinity, but permissions on the cluster prevent me from creating the environment. 3)Updating ABySS/Velvet

Do you have an suggestions on how I should approach this?

Here is my code: (phyluce) bash-4.2$ phyluce_assembly_assemblo_abyss \

--config assembly.b1.conf \
--output ABySS-assemblies2 \
--kmer 55 \
--subfolder split-adapter-quality-trimmed \
--cores 50 \
--clean \
--log-path log

Log: 2020-06-01 15:49:42,663 - phyluce_assembly_assemblo_abyss - INFO - ============ Starting phyluce_assembly_assemblo_abyss =========== 2020-06-01 15:49:42,664 - phyluce_assembly_assemblo_abyss - INFO - Version: git fatal: Not a git repository: '/rhome/bfscott1906/bigdata/.conda/envs/phyluce/lib/python2.7/site-packages/.git' 2020-06-01 15:49:42,664 - phyluce_assembly_assemblo_abyss - INFO - Argument --abyss_se: False 2020-06-01 15:49:42,664 - phyluce_assembly_assemblo_abyss - INFO - Argument --clean: True 2020-06-01 15:49:42,665 - phyluce_assembly_assemblo_abyss - INFO - Argument --config: /rhome/bfscott1906/assembly.b1.conf 2020-06-01 15:49:42,665 - phyluce_assembly_assemblo_abyss - INFO - Argument --cores: 50 2020-06-01 15:49:42,665 - phyluce_assembly_assemblo_abyss - INFO - Argument --dir: None 2020-06-01 15:49:42,665 - phyluce_assembly_assemblo_abyss - INFO - Argument --kmer: 55 2020-06-01 15:49:42,666 - phyluce_assembly_assemblo_abyss - INFO - Argument --log_path: /rhome/bfscott1906/log 2020-06-01 15:49:42,666 - phyluce_assembly_assemblo_abyss - INFO - Argument --output: /rhome/bfscott1906/ABySS-assemblies2 2020-06-01 15:49:42,666 - phyluce_assembly_assemblo_abyss - INFO - Argument --subfolder: split-adapter-quality-trimmed 2020-06-01 15:49:42,666 - phyluce_assembly_assemblo_abyss - INFO - Argument --verbosity: INFO 2020-06-01 15:49:42,667 - phyluce_assembly_assemblo_abyss - INFO - Getting input filenames and creating output directories 2020-06-01 15:49:42,672 - phyluce_assembly_assemblo_abyss - INFO - ------------------ Processing Amaur_carr_173546 ----------------- 2020-06-01 15:49:42,672 - phyluce_assembly_assemblo_abyss - INFO - Finding fastq/fasta files 2020-06-01 15:49:42,680 - phyluce_assembly_assemblo_abyss - INFO - File type is fastq 2020-06-01 15:49:42,682 - phyluce_assembly_assemblo_abyss - INFO - Running abyss-pe against data 2020-06-01 16:14:51,551 - phyluce_assembly_assemblo_abyss - INFO - Symlinking assembled contigs into /rhome/bfscott1906/ABySS-assemblies2/contigs 2020-06-01 16:14:51,552 - phyluce_assembly_assemblo_abyss - INFO - ----------------- Processing Conir_bicolor_38645 ---------------- 2020-06-01 16:14:51,552 - phyluce_assembly_assemblo_abyss - INFO - Finding fastq/fasta files 2020-06-01 16:14:51,556 - phyluce_assembly_assemblo_abyss - INFO - File type is fastq 2020-06-01 16:14:51,556 - phyluce_assembly_assemblo_abyss - INFO - Running abyss-pe against data Traceback (most recent call last): File "/rhome/bfscott1906/bigdata/.conda/envs/phyluce/bin/phyluce_assembly_assemblo_abyss", line 323, in main() File "/rhome/bfscott1906/bigdata/.conda/envs/phyluce/bin/phyluce_assembly_assemblo_abyss", line 311, in main contigs_file = get_contigs_file_from_output(output) File "/rhome/bfscott1906/bigdata/.conda/envs/phyluce/bin/phyluce_assembly_assemblo_abyss", line 218, in get_contigs_file_from_output return glob.glob(os.path.join(output, "*-contigs.fa"))[0] IndexError: list index out of range

brantfaircloth commented 4 years ago

If the program is running on the 12-core machine reasonably well, but not running on the cluster, then the problem likely lies in the way that the cluster is setup (or has something to do with how the $PATHs are set on the cluster. For the most part, that is out of my hands. ABySS runs successfully in the software test I use to make sure the code is sane.

That said, the way that ABySS works depends on using make. Phyluce comes with a version of make that will work, in most cases... but if your cluster or job script (or something else) are substituting some other version of make for the one expected, that could cause the problem.

I'd suggest foregoing ABySS and Trinity and Velvet, in favor of spades - it works mostly as well if not better; it's easier to run, consistently; and it doesn't have any troublesome dependencies.

I might even consider using metaspades, as implemented in mitofinder (https://github.com/RemiAllio/MitoFinder), because recent tests by Anna Hiller in my lab have showed that it somewhat outperforms regular spades (plus, you get the added bonus of mtDNA genome assemblies).

Lostinthecodingsauce commented 4 years ago

Hi Brant,

I figured it was likely a problem in how phyluce was installed on the cluster, I have had constant problems with it. Thank for you help and quick response. metaspades looks really interesting, thanks for the advise!

On Tue, Jun 2, 2020 at 7:41 AM Brant Faircloth notifications@github.com wrote:

If the program is running on the 12-core machine reasonably well, but not running on the cluster, then the problem likely lies in the way that the cluster is setup (or has something to do with how the $PATHs are set on the cluster. For the most part, that is out of my hands. ABySS runs successfully in the software test I use to make sure the code is sane.

That said, the way that ABySS works depends on using make. Phyluce comes with a version of make that will work, in most cases... but if your cluster or job script (or something else) are substituting some other version of make for the one expected, that could cause the problem.

I'd suggest foregoing ABySS and Trinity and Velvet, in favor of spades - it works mostly as well if not better; it's easier to run, consistently; and it doesn't have any troublesome dependencies.

I might even consider using metaspades, as implemented in mitofinder ( https://github.com/RemiAllio/MitoFinder), because recent tests by Anna Hiller in my lab have showed that it somewhat outperforms regular spades (plus, you get the added bonus of mtDNA genome assemblies).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/faircloth-lab/phyluce/issues/194#issuecomment-637587464, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALOMO2JRYPLNLIU3L2U2ETTRUUFRPANCNFSM4NQHZR7A .

--

Benjamin Scott Graduate Student/Teaching Assistant M.S. Evolutionary Biology student San Diego State University

brantfaircloth commented 4 years ago

One other option is to get a copy of abyss working on your cluster, then point phyluce to that copy using the ~/.phyluce.conf file (i.e., rather than depending on the phyluce copy). if your HPC uses modules, that would be one good way to go.