search

merenlab / anvio

An analysis and visualization platform for 'omics data
http://merenlab.org/software/anvio
GNU General Public License v3.0
442 stars 146 forks source link

anvi-self-test fails: input_file_path issue/bam files not found #383

Closed pug-ventures closed 8 years ago

pug-ventures commented 8 years ago

Dear developers,

I've installed anvio and all required dependencies (at least they work when called individually and they are all in $PATH), but when I run anvi-self-test, I get the following error:

[Fri Jul 22][16:46:10][jwippler@sunray-1][~]
$ anvi-self-test --debug

:: Output directory ...

/tmp/tmpXL1NU6/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2
Profile DB version ...........................: 16
Contigs DB version ...........................: 6
Samples information DB version ...............: 2
Auxiliary HDF5 DB version ....................: 1
Users DB version (for anvi-server) ...........: 1

:: Initializing raw BAM files ...

[22 Jul 16 16:47:35 SORT] Sorting BAM File... May take a while depending on the size.                                                                                open: No such file or directory
Sorted BAM File ..............................: /tmp/tmpXL1NU6/test-output/SAMPLE-01.bam                                                                              

Config Error: Sorry. Something went wrong. Samtools thinks it generated the sorted output, yet
              it is not there :(                                                              

Anvi'o's self-test is done, and the temporary output files are still here: /tmp/tmpXL1NU6

Seems that pysam can't find the bam files, because input_file_path isn't set properly:

[Fri Jul 22][17:00:19][jwippler@sunray-1][~/tools/anvio/bin]
$ cat anvi-init-bam 

[part not shown]

filesnpaths.is_file_exists(input_file_path)
    filesnpaths.is_output_file_writable(output_file_path)

    progress.new('SORT')
    progress.update('Sorting BAM File... May take a while depending on the size.')

    pysam.sort("-o", output_file_path, input_file_path)
    progress.end()

    run.info('Sorted BAM File', output_file_path)
    if not os.path.exists(output_file_path):
        raise ConfigError, "Sorry. Something went wrong. Samtools thinks it generated the sorted output, yet it is not there :("

[part not shown]

In any case, the bam files are present in the test directory of anvi'o:

[Fri Jul 22][16:58:54][jwippler@sunray-1][~/tools/anvio/tests/sandbox]
$ l
total 1.5M
drwxr-xr-x 2 jwippler symbiosis    7 2016-07-22 16:14 anvi_pangenome_files
drwxr-xr-x 3 jwippler symbiosis    3 2016-07-22 16:14 anvi_server_files
drwxr-xr-x 2 jwippler symbiosis    6 2016-07-22 16:14 BAMs_SF
drwxr-xr-x 2 jwippler symbiosis    4 2016-07-22 16:14 example_external_collections
drwxr-xr-x 2 jwippler symbiosis    4 2016-07-22 16:14 example_files_for_centrifuge_taxonomy
drwxr-xr-x 2 jwippler symbiosis    5 2016-07-22 16:14 example_files_for_external_binning_results
drwxr-xr-x 2 jwippler symbiosis    6 2016-07-22 16:14 external_hmm_profile
drwxr-xr-x 2 jwippler symbiosis   10 2016-07-22 16:14 files_for_manual_interactive
drwxr-xr-x 2 jwippler symbiosis    6 2016-07-22 16:14 myrast_cmdline
drwxr-xr-x 2 jwippler symbiosis    5 2016-07-22 16:14 myrast_gui
drwxr-xr-x 2 jwippler symbiosis    2 2016-07-22 16:40 test-output
-rw-r--r-- 1 jwippler symbiosis 1.7K 2016-07-22 16:14 additional_view_data.txt
-rw-r--r-- 1 jwippler symbiosis 3.6K 2016-07-22 16:14 additional_view.txt
-rw-r--r-- 1 jwippler symbiosis   58 2016-07-22 16:14 adjacent_positions_for_linkmers.txt
-rw-r--r-- 1 jwippler symbiosis 226K 2016-07-22 16:14 Brown_et_al-CPR-Campbell_et_al_BSCG.txt
-rw-r--r-- 1 jwippler symbiosis 3.2K 2016-07-22 16:14 concoct.txt
-rw-r--r-- 1 jwippler symbiosis  56K 2016-07-22 16:14 contigs.fa
-rw-r--r-- 1 jwippler symbiosis   61 2016-07-22 16:14 distant_positions_for_linkmers.txt
-rw-r--r-- 1 jwippler symbiosis  337 2016-07-22 16:14 example_clustering_configuration.ini
-rw-r--r-- 1 jwippler symbiosis 4.1K 2016-07-22 16:14 example_external_gene_calls.txt
-rw-r--r-- 1 jwippler symbiosis 5.4K 2016-07-22 16:14 example_gene_functions_input_matrix.txt
-rw-r--r-- 1 jwippler symbiosis   59 2016-07-22 16:14 example_genes_of_interest.txt
-rw-r--r-- 1 jwippler symbiosis  36K 2016-07-22 16:14 example_interpro_output.tsv
-rw-r--r-- 1 jwippler symbiosis  481 2016-07-22 16:14 example_interpro_output.tsv.README
-rw-r--r-- 1 jwippler symbiosis    6 2016-07-22 16:14 sample_CONCOCT_bin_id.txt
-rw-r--r-- 1 jwippler symbiosis   85 2016-07-22 16:14 sample_contig_ids.txt
-rw-r--r-- 1 jwippler symbiosis   12 2016-07-22 16:14 sample_gene_call_ids.txt
-rwxr-xr-x 1 jwippler symbiosis 536K 2016-07-22 16:14 SAMPLE-RAW-01.bam
-rwxr-xr-x 1 jwippler symbiosis 159K 2016-07-22 16:14 SAMPLE-RAW-02.bam
-rwxr-xr-x 1 jwippler symbiosis 614K 2016-07-22 16:14 SAMPLE-RAW-03.bam
-rw-r--r-- 1 jwippler symbiosis  152 2016-07-22 16:14 samples-information.txt
-rw-r--r-- 1 jwippler symbiosis  163 2016-07-22 16:14 samples-order.txt
-rw-r--r-- 1 jwippler symbiosis 1.7K 2016-07-22 16:14 short_reads_for_SNV_testing.ini
-rw-r--r-- 1 jwippler symbiosis 1.3K 2016-07-22 16:14 single_contig.fa
-rw-r--r-- 1 jwippler symbiosis  168 2016-07-22 16:14 smtp_config_sample.ini

I'm not good enough at reading python code to understand where the issue is coming from. Therefore any help from you wizards would be much appreciated (expecto patronum!) Many thanks in advance!

Juliane

meren commented 8 years ago

Hi Juliane!

Thank you for this report. This is rather odd. I am curious about something. Can you please send the output of this command:

ls /tmp/tmpXL1NU6

This is the directory that is mentioned at the very last line of the anvi-self-test --debug command.

Additionally, it would be very useful if you could run the test this way and kindly let me know if this works for you:

cd ~/tools/anvio/tests/
./run_mini_test.sh

Best wishes,

pug-ventures commented 8 years ago

Thanks for the fast reply!

The tmp directory was already gone, so I simply ran anvi-self-test again:

[Sat Jul 23][13:22:53][jwippler@sunray-2][~]
$ anvi-self-test --debug

:: Output directory ...

/tmp/tmpuzqLSE/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2
Profile DB version ...........................: 16
Contigs DB version ...........................: 6
Samples information DB version ...............: 2
Auxiliary HDF5 DB version ....................: 1
Users DB version (for anvi-server) ...........: 1                                                                                                                                           

:: Initializing raw BAM files ...                                                                                                                                                           

[23 Jul 16 13:23:21 SORT] Sorting BAM File... May take a while depending on the size.                                                                                                      
open: No such file or directory                                                                                                                                                              
Sorted BAM File ..............................: /tmp/tmpuzqLSE/test-output/SAMPLE-01.bam                                                                                                    

Config Error: Sorry. Something went wrong. Samtools thinks it generated the sorted output, yet                                                                                              
              it is not there :(                                                                                                                                                            

Anvi'o's self-test is done, and the temporary output files are still here: /tmp/tmpuzqLSE

[Sat Jul 23][13:23:21][jwippler@sunray-2][~]
$ ls /tmp/tmpuzqLSE
test-output                                                                                                                                                                                                                                                                                                              
[Sat Jul 23][13:26:00][jwippler@sunray-2][~]
$ l /tmp/tmpuzqLSE/test-output/
total 0

As you can see, empty.

Now for running the wrapper, I get the same result:

[Sat Jul 23][13:27:23][jwippler@sunray-2][~]
$ cd tools/anvio/tests/                                                                                                                                                                     
[Sat Jul 23][13:27:33][jwippler@sunray-2][~/tools/anvio/tests]
$ ./run_mini_test.sh                                                                                                                                                                        

:: Output directory ...

/home/jwippler/tools/anvio/tests/sandbox/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2
Profile DB version ...........................: 16
Contigs DB version ...........................: 6
Samples information DB version ...............: 2
Auxiliary HDF5 DB version ....................: 1
Users DB version (for anvi-server) ...........: 1

:: Initializing raw BAM files ...

[23 Jul 16 13:27:41 SORT] Sorting BAM File... May take a while depending on the size.                                                                                                      
open: No such file or directory
Sorted BAM File ..............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01.bam                                                                          

Config Error: Sorry. Something went wrong. Samtools thinks it generated the sorted output, yet
              it is not there :(

Again, no output is produced :

[Sat Jul 23][13:27:41][jwippler@sunray-2][~/tools/anvio/tests]
$ l /home/jwippler/tools/anvio/tests/sandbox/test-output                                                                                                                                    
total 0

Although the BAM files are present in anvio/test/:

[Sat Jul 23][13:33:50][jwippler@sunray-2][~/tools/anvio/tests/sandbox]
$ l *.bam
-rwxr-xr-x 1 jwippler symbiosis 536K 2016-07-22 16:14 SAMPLE-RAW-01.bam
-rwxr-xr-x 1 jwippler symbiosis 159K 2016-07-22 16:14 SAMPLE-RAW-02.bam
-rwxr-xr-x 1 jwippler symbiosis 614K 2016-07-22 16:14 SAMPLE-RAW-03.bam

In the meantime, I tried to follow the metagenomic workflow, just to see what happens and i failed already at the first step:


[Sat Jul 23][14:17:37][jwippler@sunray-2][~/tools/anvio/tests/sandbox]
$ anvi-gen-contigs-database -f contigs.fa -o contigs.db                                                                                                
Traceback (most recent call last):
  File "/home/jwippler/tools/anvio/bin/anvi-gen-contigs-database", line 7, in <module>
    import anvio.dbops as dbops
  File "/home/jwippler/tools/anvio/anvio/dbops.py", line 31, in <module>
    import anvio.auxiliarydataops as auxiliarydataops
  File "/home/jwippler/tools/anvio/anvio/auxiliarydataops.py", line 5, in <module>
    import h5py
ImportError: No module named h5py

Clearly, I am missing some dependencies, that weren't listed on the installation page, and maybe that's also what's causing trouble running the anvio test?

So I decided to be a smartass and updated my anvio installation as described on your install page:

git pull
git submodule update --init --recursive
python setup.py install --prefix=/path-to-my-python-installation

Only to possibly mess things up even more:


[Sat Jul 23][14:36:54][jwippler@sunray-2][~/tools/anvio/tests]
$ ./run_mini_test.sh 

:: Output directory ...

/home/jwippler/tools/anvio/tests/sandbox/test-output

:: Anvo'o version ...

Traceback (most recent call last):
  File "/home/jwippler/tools/Python-2.7.5/build/bin/anvi-profile", line 4, in <module>
    import pkg_resources
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg/pkg_resources.py", line 2807, in <module>
    parse_requirements(__requires__), Environment()
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg/pkg_resources.py", line 594, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: scipy>=0.13.3

Which told me my scipy installation is outdated:

[Sat Jul 23][14:37:01][jwippler@sunray-2][~/tools/anvio/tests]
$ python -c 'import scipy; print(scipy.__version__)'                                                                                                   
0.12.0

And then I noticed other modules aren't the correct version either, so I upgraded everything with pip.

So, now when I run run_mini_test.sh, I got rid of the old problem, and a lot more happens, however, there is still some issue at the end:

[Sat Jul 23][15:04:12][jwippler@sunray-2][~/tools/anvio/tests]
$ ./run_mini_test.sh                                                                                                                                   

:: Output directory ...

/home/jwippler/tools/anvio/tests/sandbox/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2
Profile DB version ...........................: 16
Contigs DB version ...........................: 6
Samples information DB version ...............: 2
Auxiliary HDF5 DB version ....................: 1
Users DB version (for anvi-server) ...........: 1

:: Initializing raw BAM files ...

Sorted BAM File ..............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01.bam                                     
BAM File Index ...............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01.bam.bai                                 

Sorted BAM File ..............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02.bam                                     
BAM File Index ...............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02.bam.bai                                 

Sorted BAM File ..............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03.bam                                     
BAM File Index ...............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03.bam.bai                                 

:: Generating an EMPTY contigs database ...

Finding ORFs in contigs                                                                                                                                
===============================================                                                                                                        
Genes ........................................: /tmp/tmpagXC58/contigs.genes                                                                           
Proteins .....................................: /tmp/tmpagXC58/contigs.proteins
Log file .....................................: /tmp/tmpagXC58/00_log.txt
Result .......................................: Prodigal (v2.60) has identified 51 genes.                                                              

Contigs with at least one gene call ..........: 6 of 6 (100.0%)                                                                                        
Contigs database .............................: A new database, /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db, has been created.
Number of contigs ............................: 6
Number of splits .............................: 38
Total number of nucleotides ..................: 57,030
Gene calling step skipped ....................: False
Splits broke genes (non-mindful mode) ........: False
Desired split length (what the user wanted) ..: 1,000
Average split length (wnat anvi'o gave back) .: 1,620

:: Populating taxonomy for splits table in the database using 'centrifuge' parser ...

Total num hits found .........................: 49
Removed due to low hit score of 250 ..........: 17
Removed due to too many competing hits .......: 0
Final num hits ...............................: 23
Num gene caller ids in the db ................: 51                                                                                                     
Num gene caller ids in the incoming data .....: 23
Taxon names table ............................: Updated with 1 unique taxon names
Genes taxonomy table .........................: Taxonomy stored for 23 gene calls
Splits taxonomy ..............................: Input data from "centrifuge" annotated 17 of 38 splits (44.7%) with taxonomy.

:: Populating search tables in the latest contigs database using default HMM profiles ...

HMM profiles .................................: 4 sources have been loaded: Dupont_et_al (111 genes), Alneberg_et_al (34 genes), Campbell_et_al (139 genes), Creevey_et_al (40 genes)                                                                                                                         
Sequences ....................................: 51 sequences reported.                                                                                 
FASTA ........................................: /tmp/tmpdyTAky/sequences.fa

HMM Profiling for Dupont_et_al                                                                                                                         
===============================================                                                                                                        
Reference ....................................: Dupont et al, http://www.nature.com/ismej/journal/v6/n6/full/ismej2011189a.html                        
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Dupont_et_al/genes.hmm.gz                                                                                                               
Number of genes ..............................: 111
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmp2FUt_y
HMM scan output ..............................: /tmp/tmp2FUt_y/hmm.output
HMM scan hits ................................: /tmp/tmp2FUt_y/hmm.hits
Log file .....................................: /tmp/tmp2FUt_y/00_log.txt
Number of raw hits ...........................: 2                                                                                                      

HMM Profiling for Alneberg_et_al                                                                                                                       
===============================================                                                                                                        
Reference ....................................: Alneberg et al, http://www.nature.com/nmeth/journal/v11/n11/full/nmeth.3103.html                       
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Alneberg_et_al/genes.hmm.gz                                                                                                             
Number of genes ..............................: 34
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmp6Ob01H
HMM scan output ..............................: /tmp/tmp6Ob01H/hmm.output
HMM scan hits ................................: /tmp/tmp6Ob01H/hmm.hits
Log file .....................................: /tmp/tmp6Ob01H/00_log.txt
Number of raw hits ...........................: 0                                                                                                      

HMM Profiling for Campbell_et_al                                                                                                                       
===============================================                                                                                                        
Reference ....................................: Campbell et al, http://www.pnas.org/content/110/14/5540.short                                          
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Campbell_et_al/genes.hmm.gz                                                                                                             
Number of genes ..............................: 139
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmpy_SVdo
HMM scan output ..............................: /tmp/tmpy_SVdo/hmm.output
HMM scan hits ................................: /tmp/tmpy_SVdo/hmm.hits
Log file .....................................: /tmp/tmpy_SVdo/00_log.txt
Number of raw hits ...........................: 2                                                                                                      

HMM Profiling for Creevey_et_al                                                                                                                        
===============================================                                                                                                        
Reference ....................................: Creevey et al, http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0022099                
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Creevey_et_al/genes.hmm.gz                                                                                                              
Number of genes ..............................: 40
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmpDpzXu1
HMM scan output ..............................: /tmp/tmpDpzXu1/hmm.output
HMM scan hits ................................: /tmp/tmpDpzXu1/hmm.hits
Log file .....................................: /tmp/tmpDpzXu1/00_log.txt
Number of raw hits ...........................: 0                                                                                                      

:: Importing gene function calls using 'interproscan' parser ...

Gene functions ...............................: 326 function calls from 11 sources for 48 unique gene calls has been added to the contigs database.

:: Populating HMM hits tables in the latest contigs database using a mock HMM collection from an external directory ...

HMM profiles .................................: 1 source been loaded: external_hmm_profile (2 genes)
Sequences ....................................: 51 sequences reported.                                                                                 
FASTA ........................................: /tmp/tmptR9ulo/sequences.fa

HMM Profiling for external_hmm_profile                                                                                                                 
===============================================                                                                                                        
Reference ....................................: Anvio et al, http://merenlab.org/projects/anvio                                                        
Pfam model ...................................: external_hmm_profile/genes.hmm.gz
Number of genes ..............................: 2
Number of CPUs will be used for search .......: 1
Temporary work dir ...........................: /tmp/tmpEZlWsV
HMM scan output ..............................: /tmp/tmpEZlWsV/hmm.output
HMM scan hits ................................: /tmp/tmpEZlWsV/hmm.hits
Log file .....................................: /tmp/tmpEZlWsV/00_log.txt
Number of raw hits ...........................: 4                                                                                                      

:: Contigs DB is ready; here are the tables in it: ...

collections_bins_info    gene_protein_sequences   hmm_hits_info          
collections_info         genes_in_contigs         kmer_contigs           
collections_of_contigs   genes_in_splits          kmer_splits            
collections_of_splits    genes_in_splits_summary  self                   
contig_sequences         genes_taxonomy           splits_basic_info      
contigs_basic_info       hmm_hits                 splits_taxonomy        
gene_functions           hmm_hits_in_splits       taxon_names            

:: Profiling sample SAMPLE-01 ...

Auxiliary Data .....................: Found: /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.h5 (v. 1)                                    
Contigs DB .........................: Initialized: /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db (v. 6)

WARNING                                                                                                                                                
=====================================                                                                                                                  
Amino acid linkmer frequencies will not be characterized for this profile.                                                                             

anvio ..............................: 2.0.2                                                                                                            
profiler_version ...................: 16
sample_id ..........................: SAMPLE_01
profile_db .........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01/PROFILE.db
contigs_db .........................: True
contigs_db_hash ....................: df8d05f5
cmd_line ...........................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-profile -i /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01.bam -o /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01 -c /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db     
merged .............................: False
blank ..............................: False
split_length .......................: 1,000
min_contig_length ..................: 2,500
min_mean_coverage ..................: 0
clustering_performed ...............: False
min_coverage_for_variability .......: 10
skip_SNV_profiling .................: False
profile_AA_frequencies .............: False
report_variability_full ............: False
gene_coverages_computed ............: 1
input_bam ..........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01.bam                                               
output_dir .........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01
total_reads_mapped .................: 23,783
num_contigs ........................: 3
num_contigs_after_M ................: 3
num_splits .........................: 35
total_length .......................: 56,709
split_coverage_values ..............: stored in /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01/AUXILIARY-DATA.h5                       
runinfo ............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01/RUNINFO.cp                                        

:: Profiling sample SAMPLE-02 ...

Auxiliary Data .....................: Found: /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.h5 (v. 1)                                    
Contigs DB .........................: Initialized: /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db (v. 6)

WARNING                                                                                                                                                
=====================================                                                                                                                  
Amino acid linkmer frequencies will not be characterized for this profile.                                                                             

anvio ..............................: 2.0.2                                                                                                            
profiler_version ...................: 16
sample_id ..........................: SAMPLE_02
profile_db .........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02/PROFILE.db
contigs_db .........................: True
contigs_db_hash ....................: df8d05f5
cmd_line ...........................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-profile -i /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02.bam -o /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02 -c /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db     
merged .............................: False
blank ..............................: False
split_length .......................: 1,000
min_contig_length ..................: 2,500
min_mean_coverage ..................: 0
clustering_performed ...............: False
min_coverage_for_variability .......: 10
skip_SNV_profiling .................: False
profile_AA_frequencies .............: False
report_variability_full ............: False
gene_coverages_computed ............: 1
input_bam ..........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02.bam                                               
output_dir .........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02
total_reads_mapped .................: 5,082
num_contigs ........................: 3
num_contigs_after_M ................: 3
num_splits .........................: 35
total_length .......................: 56,709
split_coverage_values ..............: stored in /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02/AUXILIARY-DATA.h5                       
runinfo ............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02/RUNINFO.cp                                        

:: Profiling sample SAMPLE-03 ...

Auxiliary Data .....................: Found: /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.h5 (v. 1)                                    
Contigs DB .........................: Initialized: /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db (v. 6)

WARNING                                                                                                                                                
=====================================                                                                                                                  
Amino acid linkmer frequencies will not be characterized for this profile.                                                                             

anvio ..............................: 2.0.2                                                                                                            
profiler_version ...................: 16
sample_id ..........................: SAMPLE_03
profile_db .........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03/PROFILE.db
contigs_db .........................: True
contigs_db_hash ....................: df8d05f5
cmd_line ...........................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-profile -i /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03.bam -o /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03 -c /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db     
merged .............................: False
blank ..............................: False
split_length .......................: 1,000
min_contig_length ..................: 2,500
min_mean_coverage ..................: 0
clustering_performed ...............: False
min_coverage_for_variability .......: 10
skip_SNV_profiling .................: False
profile_AA_frequencies .............: False
report_variability_full ............: False
gene_coverages_computed ............: 1
input_bam ..........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03.bam                                               
output_dir .........................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03
total_reads_mapped .................: 24,870
num_contigs ........................: 3
num_contigs_after_M ................: 3
num_splits .........................: 35
total_length .......................: 56,709
split_coverage_values ..............: stored in /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03/AUXILIARY-DATA.h5                       
runinfo ............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03/RUNINFO.cp                                        

:: Merging profiles ...

total_length .................................: 56,709
num_contigs ..................................: 3
num_splits ...................................: 35
split_length .................................: 1,000
min_contig_length ............................: 2,500
min_mean_coverage ............................: 0
min_coverage_for_variability .................: 10
report_variability_full ......................: False
profile_AA_frequencies .......................: False
skip_SNV_profiling ...........................: False
profiler_version .............................: 16
output_dir ...................................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLES-MERGED
sample_id ....................................: SAMPLES_MERGED
profile_db ...................................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLES-MERGED/PROFILE.db
merged .......................................: True
contigs_db_hash ..............................: df8d05f5
merged_sample_ids ............................: ['SAMPLE_01', 'SAMPLE_02', 'SAMPLE_03']
cmd_line .....................................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-merge /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01/RUNINFO.cp /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-02/RUNINFO.cp /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-03/RUNINFO.cp -o /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLES-MERGED -c /home/jwippler/tools/anvio/tests/sandbox/test-output/CONTIGS.db                                                                                                                                              
num_runs_processed ...........................: 3
clustering_performed .........................: True

WARNING                                                                                                                                                
===============================================                                                                                                        
anvio just set the normalization values for each sample based on how many mapped                                                                       
reads they contained. All normalized coverages will use this information:                                                                              
SAMPLE_01: 0.21, SAMPLE_02: 1.00, SAMPLE_03: 0.20                                                                                                      

WARNING                                                                                                                                                
===============================================                                                                                                        
AA frequencies were not profiled, these tables will be empty in the merged                                                                             
profile database.                                                                                                                                      

New hierarchical clusetring ..................: "tnf-cov:euclidean:average" has been added to the database...                                          
New hierarchical clusetring ..................: "tnf:euclidean:average" has been added to the database...                                              
New hierarchical clusetring ..................: "cov:euclidean:average" has been added to the database...                                              
runinfo ......................................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLES-MERGED/RUNINFO.mcp                        
CONCOCT INIT .................................: Complete for 35 splits                                                                                 
CONCOCT VGBMM ................................: Requesting 80 clusters
[23 Jul 16 15:05:02 VBGMM] Clustering ...                                                                                                             Traceback (most recent call last):
  File "/home/jwippler/tools/Python-2.7.5/build/bin/anvi-merge", line 5, in <module>
    pkg_resources.run_script('anvio==2.0.2', 'anvi-merge')
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg/pkg_resources.py", line 505, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg/pkg_resources.py", line 1245, in run_script
    execfile(script_filename, namespace, namespace)
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/EGG-INFO/scripts/anvi-merge", line 49, in <module>
    merger.MultipleRuns(args).merge()
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/merger.py", line 465, in merge
    self.bin_contigs_concoct()
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/merger.py", line 571, in bin_contigs_concoct
    c.cluster()
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/concoct.py", line 99, in cluster
    self.clusters = CONCOCT_INTERFACE(self.kmers, self.coverages, self.lengths, self.debug, NClusters=self.num_clusters_requested).cluster()
  File "/home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/concoct.py", line 223, in cluster
    vbgmm.fit(self.transformed_data, self.assign, self.NClusters, self.debug)
  File "vbgmm.pyx", line 19, in anvio.vbgmm.fit (./anvio/extensions/concoct/vbgmm.c:1338)
ValueError: ndarray is not C-contiguous

But I'm clueless what this error means. Again, thank you very much for your help!

Best regards, Juliane

meren commented 8 years ago

Hi Juliane,

Thank you very much for the detailed reporting of your journey, and I am glad you got to a point where you can almost finish all test runs :)

Before going back to the final error you got, I want to tell you that I am very surprised with the fact that even when you run this:

[Sat Jul 23][13:27:33][jwippler@sunray-2][~/tools/anvio/tests]
$ ./run_mini_test.sh

You got an error because BAM files were not found. There is no explanation to that I can think of, since the batch work is very simple, and it simply translates to this:

$ cd /home/jwippler/tools/anvio/tests/sandbox/
$ anvi-init-bam SAMPLE-RAW-01.bam --output-file test-output/SAMPLE-01.bam

The only reason for this to not work would be if SAMPLE-RAW-01.bam is missing from the directory /home/jwippler/tools/anvio/tests/sandbox/, in which case anvi-init-bam would have given an error. So I am very confused.

The final error you have is coming from CONCOCT, and I am not sure why it is upset. But I added a control, so things don't go crazy when this happens. Would you mind updating your local copy, reinstalling anvi'o, and running the test again?

Thank you very much again for your help!

pug-ventures commented 8 years ago

Hi again!

So, I reinstalled anvi'o, but unfortunately it now fails with a segmentation fault:


[Sun Jul 24][12:55:39][jwippler@sunray-2][~/tools/anvio/tests]
$ ./run_mini_test.sh

:: Output directory ...

/home/jwippler/tools/anvio/tests/sandbox/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2
Profile DB version ...........................: 16
Contigs DB version ...........................: 6
Samples information DB version ...............: 2
Auxiliary HDF5 DB version ....................: 1
Users DB version (for anvi-server) ...........: 1

:: Initializing raw BAM files ...

./run_mini_test.sh: line 29: 23555 Segmentation fault      anvi-init-bam SAMPLE-RAW-$f.bam --output-file $output_dir/SAMPLE-$f.bam

Just to make sure: I don't have to manually install CONCOCT right (since it's not listed as a dependency)?

Thanks and cheers, Juliane

meren commented 8 years ago

It is again very confusing. I thought we were done with this. Maybe your samtools is outdated? What do you get back when you type samtools --version?

Sorry for the frustration, and thank you. On Jul 24, 2016 05:58, "jwippler" notifications@github.com wrote:

Hi again!

So, I reinstalled anvi'o, but unfortunately it now fails with a segmentation fault:

[Sun Jul 24][12:55:39][jwippler@sunray-2][~/tools/anvio/tests] $ ./run_mini_test.sh

:: Output directory ...

/home/jwippler/tools/anvio/tests/sandbox/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2 Profile DB version ...........................: 16 Contigs DB version ...........................: 6 Samples information DB version ...............: 2 Auxiliary HDF5 DB version ....................: 1 Users DB version (for anvi-server) ...........: 1

:: Initializing raw BAM files ...

./run_mini_test.sh: line 29: 23555 Segmentation fault anvi-init-bam SAMPLE-RAW-$f.bam --output-file $output_dir/SAMPLE-$f.bam

Just to make sure: I don't have to manually install CONCOCT right (since it's not listed as a dependency)?

Thanks and cheers, Juliane

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/meren/anvio/issues/383#issuecomment-234770881, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMCu3GMhcmQ57ppDlZQnE3Y1FAqQb6hks5qY0VjgaJpZM4JS2op .

pug-ventures commented 8 years ago

I updated samtools to version 1.3.1 because it was kind of old and ran run_mini_test.sh again, but unfortunately this didn't help:

[Sun Jul 24][13:28:26][jwippler@sunray-2][~/tools/anvio/tests]
$ ./run_mini_test.sh                                                                                                                                                                        

:: Output directory ...

/home/jwippler/tools/anvio/tests/sandbox/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2
Profile DB version ...........................: 16
Contigs DB version ...........................: 6
Samples information DB version ...............: 2
Auxiliary HDF5 DB version ....................: 1
Users DB version (for anvi-server) ...........: 1

:: Initializing raw BAM files ...

Sorted BAM File ..............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01.bam                                                                          
BAM File Index ...............................: /home/jwippler/tools/anvio/tests/sandbox/test-output/SAMPLE-01.bam.bai                                                                      
./run_mini_test.sh: line 29:  6887 Segmentation fault      anvi-init-bam SAMPLE-RAW-$f.bam --output-file $output_dir/SAMPLE-$f.bam

No worries, I'm hopeful we'll get this running eventually! Thanks for your help and cheers, Juliane

pug-ventures commented 8 years ago

Hi again! So I ran anvi-self-test again, and I get different results depending on how I log into the machine (via ssh or via thinlinc - it's very weird, probably some sort of proxy issue I guess). So with the login method I used yesterday, I get the same segmentation fault error. But with the other login method that I used today, anvi-self-test runs, and I just don't get the visual results in the browser, maybe because we have chromium, not chrome installed, so I will have to fix that (or it is another proxy issue again, maybe you could have a look at the error and tell me what it means?).

UPDATE: Yes, there is definitely an issue with the proxy settings. I managed to fix them for firefox and run anvi-self-test successfully, but of course anvi'o doesn't run properly in firefox. I asked our local IT for help with the chrome proxy settings. Also the error messages below are not from anvio, but directly coming from the browser - I also get them when I just run the browser by itself.

UPDATE 2: With the help of our IT I managed to get the proxy settings for chrome right, and now its working! :D (see screen cap below)

Here is the new anvi-self-test output, and I've also attached a screen shot of the browser window that pops open when i run it. 

[Mon Jul 25][13:02:48][jwippler@sunray-1][~]
$ anvi-self-test 

:: Output directory ...

/tmp/tmp7zUpk7/test-output

:: Anvo'o version ...

Anvi'o version ...............................: 2.0.2
Profile DB version ...........................: 16
Contigs DB version ...........................: 6
Samples information DB version ...............: 2
Auxiliary HDF5 DB version ....................: 1
Users DB version (for anvi-server) ...........: 1

:: Initializing raw BAM files ...

Sorted BAM File ..............................: /tmp/tmp7zUpk7/test-output/SAMPLE-01.bam                                                                              
BAM File Index ...............................: /tmp/tmp7zUpk7/test-output/SAMPLE-01.bam.bai                                                                          

Sorted BAM File ..............................: /tmp/tmp7zUpk7/test-output/SAMPLE-02.bam                                                                              
BAM File Index ...............................: /tmp/tmp7zUpk7/test-output/SAMPLE-02.bam.bai                                                                          

Sorted BAM File ..............................: /tmp/tmp7zUpk7/test-output/SAMPLE-03.bam                                                                              
BAM File Index ...............................: /tmp/tmp7zUpk7/test-output/SAMPLE-03.bam.bai                                                                          

:: Generating an EMPTY contigs database ...

Finding ORFs in contigs                                                                                                                                               
===============================================                                                                                                                       
Genes ........................................: /tmp/tmpF52RDP/contigs.genes
Proteins .....................................: /tmp/tmpF52RDP/contigs.proteins
Log file .....................................: /tmp/tmpF52RDP/00_log.txt
Result .......................................: Prodigal (v2.60) has identified 51 genes.                                                                             

Contigs with at least one gene call ..........: 6 of 6 (100.0%)                                                                                                       
Contigs database .............................: A new database, /tmp/tmp7zUpk7/test-output/CONTIGS.db, has been created.
Number of contigs ............................: 6
Number of splits .............................: 38
Total number of nucleotides ..................: 57,030
Gene calling step skipped ....................: False
Splits broke genes (non-mindful mode) ........: False
Desired split length (what the user wanted) ..: 1,000
Average split length (wnat anvi'o gave back) .: 1,620

:: Populating taxonomy for splits table in the database using 'centrifuge' parser ...

Total num hits found .........................: 49
Removed due to low hit score of 250 ..........: 17
Removed due to too many competing hits .......: 0
Final num hits ...............................: 23
Num gene caller ids in the db ................: 51                                                                                                                    
Num gene caller ids in the incoming data .....: 23
Taxon names table ............................: Updated with 1 unique taxon names
Genes taxonomy table .........................: Taxonomy stored for 23 gene calls
Splits taxonomy ..............................: Input data from "centrifuge" annotated 17 of 38 splits (44.7%) with taxonomy.

:: Populating search tables in the latest contigs database using default HMM profiles ...

HMM profiles .................................: 4 sources have been loaded: Dupont_et_al (111 genes), Alneberg_et_al (34 genes), Campbell_et_al (139 genes), Creevey_et_al (40 genes)                                                                                                                                                       
Sequences ....................................: 51 sequences reported.                                                                                                
FASTA ........................................: /tmp/tmpjy8vEa/sequences.fa

HMM Profiling for Dupont_et_al                                                                                                                                        
===============================================                                                                                                                       
Reference ....................................: Dupont et al, http://www.nature.com/ismej/journal/v6/n6/full/ismej2011189a.html
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Dupont_et_al/genes.hmm.gz                                                                                                                                             
Number of genes ..............................: 111
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmpSVsWqq
HMM scan output ..............................: /tmp/tmpSVsWqq/hmm.output
HMM scan hits ................................: /tmp/tmpSVsWqq/hmm.hits
Log file .....................................: /tmp/tmpSVsWqq/00_log.txt
Number of raw hits ...........................: 2                                                                                                                     

HMM Profiling for Alneberg_et_al                                                                                                                                      
===============================================                                                                                                                       
Reference ....................................: Alneberg et al, http://www.nature.com/nmeth/journal/v11/n11/full/nmeth.3103.html
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Alneberg_et_al/genes.hmm.gz                                                                                                                                           
Number of genes ..............................: 34
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmpK9XcUv
HMM scan output ..............................: /tmp/tmpK9XcUv/hmm.output
HMM scan hits ................................: /tmp/tmpK9XcUv/hmm.hits
Log file .....................................: /tmp/tmpK9XcUv/00_log.txt
Number of raw hits ...........................: 0                                                                                                                     

HMM Profiling for Campbell_et_al                                                                                                                                      
===============================================                                                                                                                       
Reference ....................................: Campbell et al, http://www.pnas.org/content/110/14/5540.short
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Campbell_et_al/genes.hmm.gz                                                                                                                                           
Number of genes ..............................: 139
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmpY1nbPR
HMM scan output ..............................: /tmp/tmpY1nbPR/hmm.output
HMM scan hits ................................: /tmp/tmpY1nbPR/hmm.hits
Log file .....................................: /tmp/tmpY1nbPR/00_log.txt
Number of raw hits ...........................: 2                                                                                                                     

HMM Profiling for Creevey_et_al                                                                                                                                       
===============================================                                                                                                                       
Reference ....................................: Creevey et al, http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0022099
Pfam model ...................................: /home/jwippler/tools/Python-2.7.5/build/lib/python2.7/site-packages/anvio-2.0.2-py2.7-linux-x86_64.egg/anvio/data/hmm/Creevey_et_al/genes.hmm.gz                                                                                                                                            
Number of genes ..............................: 40
Number of CPUs will be used for search .......: 2
Temporary work dir ...........................: /tmp/tmpTxr5Mx
HMM scan output ..............................: /tmp/tmpTxr5Mx/hmm.output
HMM scan hits ................................: /tmp/tmpTxr5Mx/hmm.hits
Log file .....................................: /tmp/tmpTxr5Mx/00_log.txt
Number of raw hits ...........................: 0                                                                                                                     

:: Importing gene function calls using 'interproscan' parser ...

Gene functions ...............................: 326 function calls from 11 sources for 48 unique gene calls has been added to the contigs database.

:: Populating HMM hits tables in the latest contigs database using a mock HMM collection from an external directory ...

HMM profiles .................................: 1 source been loaded: external_hmm_profile (2 genes)
Sequences ....................................: 51 sequences reported.                                                                                                
FASTA ........................................: /tmp/tmp8FSX_X/sequences.fa

HMM Profiling for external_hmm_profile                                                                                                                                
===============================================                                                                                                                       
Reference ....................................: Anvio et al, http://merenlab.org/projects/anvio
Pfam model ...................................: external_hmm_profile/genes.hmm.gz
Number of genes ..............................: 2
Number of CPUs will be used for search .......: 1
Temporary work dir ...........................: /tmp/tmp0ueQzQ
HMM scan output ..............................: /tmp/tmp0ueQzQ/hmm.output
HMM scan hits ................................: /tmp/tmp0ueQzQ/hmm.hits
Log file .....................................: /tmp/tmp0ueQzQ/00_log.txt
Number of raw hits ...........................: 4                                                                                                                     

:: Contigs DB is ready; here are the tables in it: ...

collections_bins_info    gene_protein_sequences   hmm_hits_info          
collections_info         genes_in_contigs         kmer_contigs           
collections_of_contigs   genes_in_splits          kmer_splits            
collections_of_splits    genes_in_splits_summary  self                   
contig_sequences         genes_taxonomy           splits_basic_info      
contigs_basic_info       hmm_hits                 splits_taxonomy        
gene_functions           hmm_hits_in_splits       taxon_names            

:: Profiling sample SAMPLE-01 ...

Auxiliary Data .....................: Found: /tmp/tmp7zUpk7/test-output/CONTIGS.h5 (v. 1)                                                                             
Contigs DB .........................: Initialized: /tmp/tmp7zUpk7/test-output/CONTIGS.db (v. 6)

WARNING                                                                                                                                                               
=====================================                                                                                                                                 
Amino acid linkmer frequencies will not be characterized for this profile.                                                                                            

anvio ..............................: 2.0.2
profiler_version ...................: 16
sample_id ..........................: SAMPLE_01
profile_db .........................: /tmp/tmp7zUpk7/test-output/SAMPLE-01/PROFILE.db
contigs_db .........................: True
contigs_db_hash ....................: 7a3b4fcc
cmd_line ...........................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-profile -i /tmp/tmp7zUpk7/test-output/SAMPLE-01.bam -o /tmp/tmp7zUpk7/test-output/SAMPLE-01 -c /tmp/tmp7zUpk7/test-output/CONTIGS.db                                                                                                                 
merged .............................: False
blank ..............................: False
split_length .......................: 1,000
min_contig_length ..................: 2,500
min_mean_coverage ..................: 0
clustering_performed ...............: False
min_coverage_for_variability .......: 10
skip_SNV_profiling .................: False
profile_AA_frequencies .............: False
report_variability_full ............: False
gene_coverages_computed ............: 1
input_bam ..........................: /tmp/tmp7zUpk7/test-output/SAMPLE-01.bam                                                                                        
output_dir .........................: /tmp/tmp7zUpk7/test-output/SAMPLE-01
total_reads_mapped .................: 23,783
num_contigs ........................: 3
num_contigs_after_M ................: 3
num_splits .........................: 35
total_length .......................: 56,709
split_coverage_values ..............: stored in /tmp/tmp7zUpk7/test-output/SAMPLE-01/AUXILIARY-DATA.h5                                                                
runinfo ............................: /tmp/tmp7zUpk7/test-output/SAMPLE-01/RUNINFO.cp                                                                                 

:: Profiling sample SAMPLE-02 ...

Auxiliary Data .....................: Found: /tmp/tmp7zUpk7/test-output/CONTIGS.h5 (v. 1)                                                                             
Contigs DB .........................: Initialized: /tmp/tmp7zUpk7/test-output/CONTIGS.db (v. 6)

WARNING                                                                                                                                                               
=====================================                                                                                                                                 
Amino acid linkmer frequencies will not be characterized for this profile.                                                                                            

anvio ..............................: 2.0.2
profiler_version ...................: 16
sample_id ..........................: SAMPLE_02
profile_db .........................: /tmp/tmp7zUpk7/test-output/SAMPLE-02/PROFILE.db
contigs_db .........................: True
contigs_db_hash ....................: 7a3b4fcc
cmd_line ...........................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-profile -i /tmp/tmp7zUpk7/test-output/SAMPLE-02.bam -o /tmp/tmp7zUpk7/test-output/SAMPLE-02 -c /tmp/tmp7zUpk7/test-output/CONTIGS.db                                                                                                                 
merged .............................: False
blank ..............................: False
split_length .......................: 1,000
min_contig_length ..................: 2,500
min_mean_coverage ..................: 0
clustering_performed ...............: False
min_coverage_for_variability .......: 10
skip_SNV_profiling .................: False
profile_AA_frequencies .............: False
report_variability_full ............: False
gene_coverages_computed ............: 1
input_bam ..........................: /tmp/tmp7zUpk7/test-output/SAMPLE-02.bam                                                                                        
output_dir .........................: /tmp/tmp7zUpk7/test-output/SAMPLE-02
total_reads_mapped .................: 5,082
num_contigs ........................: 3
num_contigs_after_M ................: 3
num_splits .........................: 35
total_length .......................: 56,709
split_coverage_values ..............: stored in /tmp/tmp7zUpk7/test-output/SAMPLE-02/AUXILIARY-DATA.h5                                                                
runinfo ............................: /tmp/tmp7zUpk7/test-output/SAMPLE-02/RUNINFO.cp                                                                                 

:: Profiling sample SAMPLE-03 ...

Auxiliary Data .....................: Found: /tmp/tmp7zUpk7/test-output/CONTIGS.h5 (v. 1)                                                                             
Contigs DB .........................: Initialized: /tmp/tmp7zUpk7/test-output/CONTIGS.db (v. 6)

WARNING                                                                                                                                                               
=====================================                                                                                                                                 
Amino acid linkmer frequencies will not be characterized for this profile.                                                                                            

anvio ..............................: 2.0.2
profiler_version ...................: 16
sample_id ..........................: SAMPLE_03
profile_db .........................: /tmp/tmp7zUpk7/test-output/SAMPLE-03/PROFILE.db
contigs_db .........................: True
contigs_db_hash ....................: 7a3b4fcc
cmd_line ...........................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-profile -i /tmp/tmp7zUpk7/test-output/SAMPLE-03.bam -o /tmp/tmp7zUpk7/test-output/SAMPLE-03 -c /tmp/tmp7zUpk7/test-output/CONTIGS.db                                                                                                                 
merged .............................: False
blank ..............................: False
split_length .......................: 1,000
min_contig_length ..................: 2,500
min_mean_coverage ..................: 0
clustering_performed ...............: False
min_coverage_for_variability .......: 10
skip_SNV_profiling .................: False
profile_AA_frequencies .............: False
report_variability_full ............: False
gene_coverages_computed ............: 1
input_bam ..........................: /tmp/tmp7zUpk7/test-output/SAMPLE-03.bam                                                                                        
output_dir .........................: /tmp/tmp7zUpk7/test-output/SAMPLE-03
total_reads_mapped .................: 24,870
num_contigs ........................: 3
num_contigs_after_M ................: 3
num_splits .........................: 35
total_length .......................: 56,709
split_coverage_values ..............: stored in /tmp/tmp7zUpk7/test-output/SAMPLE-03/AUXILIARY-DATA.h5                                                                
runinfo ............................: /tmp/tmp7zUpk7/test-output/SAMPLE-03/RUNINFO.cp                                                                                 

:: Merging profiles ...

total_length .................................: 56,709
num_contigs ..................................: 3
num_splits ...................................: 35
split_length .................................: 1,000
min_contig_length ............................: 2,500
min_mean_coverage ............................: 0
min_coverage_for_variability .................: 10
report_variability_full ......................: False
profile_AA_frequencies .......................: False
skip_SNV_profiling ...........................: False
profiler_version .............................: 16
output_dir ...................................: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED
sample_id ....................................: SAMPLES_MERGED
profile_db ...................................: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/PROFILE.db
merged .......................................: True
contigs_db_hash ..............................: 7a3b4fcc
merged_sample_ids ............................: ['SAMPLE_01', 'SAMPLE_02', 'SAMPLE_03']
cmd_line .....................................: /home/jwippler/tools/Python-2.7.5/build/bin/anvi-merge /tmp/tmp7zUpk7/test-output/SAMPLE-01/RUNINFO.cp /tmp/tmp7zUpk7/test-output/SAMPLE-02/RUNINFO.cp /tmp/tmp7zUpk7/test-output/SAMPLE-03/RUNINFO.cp -o /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED -c /tmp/tmp7zUpk7/test-output/CONTIGS.db
num_runs_processed ...........................: 3
clustering_performed .........................: True

WARNING                                                                                                                                                               
===============================================                                                                                                                       
anvio just set the normalization values for each sample based on how many mapped                                                                                      
reads they contained. All normalized coverages will use this information:                                                                                             
SAMPLE_01: 0.21, SAMPLE_02: 1.00, SAMPLE_03: 0.20                                                                                                                     

WARNING                                                                                                                                                               
===============================================                                                                                                                       
AA frequencies were not profiled, these tables will be empty in the merged                                                                                            
profile database.                                                                                                                                                     

New hierarchical clusetring ..................: "tnf-cov:euclidean:average" has been added to the database...                                                         
New hierarchical clusetring ..................: "tnf:euclidean:average" has been added to the database...                                                             
New hierarchical clusetring ..................: "cov:euclidean:average" has been added to the database...                                                             
runinfo ......................................: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/RUNINFO.mcp                                                                 
CONCOCT INIT .................................: Complete for 35 splits                                                                                                
CONCOCT VGBMM ................................: Requesting 80 clusters
CONCOCT VGBMM ................................: Returning 3 final clusters                                                                                            
Collections ..................................: CONCOCT annotations for 35 splits have been successfully added to the database at "/tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/PROFILE.db".                                                                                                                                                   
CONCOCT results in db ........................: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/PROFILE.db

:: Generating a samples information database with samples information and samples order ...

Samples information ..........................: Loaded for 3 samples
Samples order ................................: Loaded for 2 attributes
Samples information database .................: A new samples information database, /tmp/tmp7zUpk7/test-output/SAMPLES.db, has been created.
Number of samples ............................: 3
Number of organizations ......................: 8

:: Firing up the interactive interface ...

Auxiliary Data ...............................: Found: /tmp/tmp7zUpk7/test-output/CONTIGS.h5 (v. 1)                                                                   
Contigs DB ...................................: Initialized: /tmp/tmp7zUpk7/test-output/CONTIGS.db (v. 6)
Auxiliary Data ...............................: Found: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/AUXILIARY-DATA.h5 (v. 1)                                             
Profile DB ...................................: Initialized: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/PROFILE.db (v. 16)

* The server is now listening the port number "8080". When you are finished, press
CTRL+C to terminate the server.

[25870:25893:0725/130424:ERROR:object_proxy.cc(608)] Failed to get name owner. Got org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.freedesktop.NetworkManager': no such name
[25870:25901:0725/130425:ERROR:object_proxy.cc(608)] Failed to get name owner. Got org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.chromium.Mtpd': no such name
[25870:25901:0725/130425:ERROR:object_proxy.cc(608)] Failed to get name owner. Got org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.chromium.Mtpd': no such name
[25870:25870:0725/130426:ERROR:object_proxy.cc(513)] Failed to call method: org.chromium.Mtpd.EnumerateStorage: object_path= /org/chromium/Mtpd: org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.Mtpd was not provided by any .service files
^C
Please wait...
===============================================
The server is being terminated.

:: Summarizing CONCOCT results ...

Auxiliary Data ...............................: Found: /tmp/tmp7zUpk7/test-output/CONTIGS.h5 (v. 1)                                                                   
Contigs DB ...................................: Initialized: /tmp/tmp7zUpk7/test-output/CONTIGS.db (v. 6)
Auxiliary Data ...............................: Found: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/AUXILIARY-DATA.h5 (v. 1)                                             
Profile DB ...................................: Initialized: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED/PROFILE.db (v. 16)
HTML Output ..................................: /tmp/tmp7zUpk7/test-output/SAMPLES-MERGED-SUMMARY/index.html                                                          

Anvi'o's self-test is done, and the temporary output files are cleaned.

capture

capture2

meren commented 8 years ago

Brilliant! Thank you very much for all your effort to get it up and running! Please let me know if you run into other issues.

Can you also please run anvi-self-test --full just to see whether the expanded list of operations also work as expected?

Best,

pug-ventures commented 8 years ago

I did and it also works fine (not putting the cli output here, because it's a lot, and you know how it looks), but see screen cap. BTW I love that the cli output is so colorful! It's a nice visual aid.

Thanks for your help and quick replies! Best wishes, Juliane

capture3

meren commented 8 years ago

Yay! Thank you, Juliane! :)