search

bxlab / metaWRAP

MetaWRAP - a flexible pipeline for genome-resolved metagenomic data analysis
MIT License
396 stars 190 forks source link

Quant_bins Seaborn ValueError: condensed distance matrix must contain only finite values #209

Open raissafedora opened 5 years ago

raissafedora commented 5 years ago

Hi ursky,

Thanks for the cool pipeline! I bumped into an error while running quant_bins:

metawrap quant_bins -b BIN_REFINEMENT_95/metawrap_95_5_bins -o QUANT_BINS -a ASSEMBLY/final_assembly.fasta COMBINED_1.fastq COMBINED_2.fastq

------------------------------------------------------------------------------------------------------------------------
-----                                  1 forward and 1 reverse read files detected                                 -----
------------------------------------------------------------------------------------------------------------------------

########################################################################################################################
#####                                    SETTING UP OUTPUT AND INDEXING ASSEMBLY                                   #####
########################################################################################################################

mkdir: cannot create directory `QUANT_BINS': File exists

------------------------------------------------------------------------------------------------------------------------
-----                            Indexing assembly file with salmon. Ignore any warnings                           -----
------------------------------------------------------------------------------------------------------------------------

[2019-07-19 18:50:33.498] [jLog] [info] building index
[2019-07-19 18:50:33.501] [jointLog] [info] [Step 1 of 4] : counting k-mers
[2019-07-19 18:50:33.953] [jointLog] [warning] Entry with header [NODE_1_length_574306_cov_23.897390] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:33.964] [jointLog] [warning] Entry with header [NODE_2_length_533400_cov_201.728071] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:33.973] [jointLog] [warning] Entry with header [NODE_3_length_402796_cov_43.592266] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:33.981] [jointLog] [warning] Entry with header [NODE_4_length_390025_cov_36.254420] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:33.988] [jointLog] [warning] Entry with header [NODE_5_length_347876_cov_16.955851] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:33.996] [jointLog] [warning] Entry with header [NODE_6_length_344232_cov_61.656165] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.003] [jointLog] [warning] Entry with header [NODE_7_length_333012_cov_57.000324] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.011] [jointLog] [warning] Entry with header [NODE_8_length_320664_cov_125.863488] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.017] [jointLog] [warning] Entry with header [NODE_9_length_308860_cov_247.868723] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.023] [jointLog] [warning] Entry with header [NODE_10_length_285154_cov_352.485445] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.028] [jointLog] [warning] Entry with header [NODE_11_length_256618_cov_55.647607] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.033] [jointLog] [warning] Entry with header [NODE_12_length_245986_cov_41.473088] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.038] [jointLog] [warning] Entry with header [NODE_13_length_243377_cov_37.319474] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.045] [jointLog] [warning] Entry with header [NODE_14_length_240124_cov_53.378395] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.050] [jointLog] [warning] Entry with header [NODE_15_length_239095_cov_105.478104] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.054] [jointLog] [warning] Entry with header [NODE_16_length_237882_cov_45.823708] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.059] [jointLog] [warning] Entry with header [NODE_17_length_234228_cov_47.282278] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.064] [jointLog] [warning] Entry with header [NODE_18_length_232220_cov_15.235414] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.069] [jointLog] [warning] Entry with header [NODE_19_length_230559_cov_19.311044] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.073] [jointLog] [warning] Entry with header [NODE_20_length_229165_cov_253.169473] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.078] [jointLog] [warning] Entry with header [NODE_21_length_225179_cov_72.759741] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.083] [jointLog] [warning] Entry with header [NODE_22_length_225031_cov_46.020780] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.090] [jointLog] [warning] Entry with header [NODE_23_length_223309_cov_17.821096] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.094] [jointLog] [warning] Entry with header [NODE_24_length_222972_cov_25.942929] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.099] [jointLog] [warning] Entry with header [NODE_25_length_221792_cov_384.989393] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.103] [jointLog] [warning] Entry with header [NODE_26_length_218523_cov_25.401972] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.108] [jointLog] [warning] Entry with header [NODE_27_length_217557_cov_66.241092] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.112] [jointLog] [warning] Entry with header [NODE_28_length_215888_cov_56.265942] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.116] [jointLog] [warning] Entry with header [NODE_29_length_214067_cov_70.172757] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.120] [jointLog] [warning] Entry with header [NODE_30_length_206300_cov_117.036369] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.125] [jointLog] [warning] Entry with header [NODE_31_length_204716_cov_114.141312] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.129] [jointLog] [warning] Entry with header [NODE_32_length_204602_cov_18.249566] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.133] [jointLog] [warning] Entry with header [NODE_33_length_203151_cov_300.282615] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
[2019-07-19 18:50:34.137] [jointLog] [warning] Entry with header [NODE_34_length_201539_cov_268.924838] was longer than 200000 nucleotides.  Are you certain that we are indexing a transcriptome and not a genome?
Version Info: ### PLEASE UPGRADE SALMON ###
### A newer version of salmon with important bug fixes and improvements is available. ####
###
The newest version, available at https://github.com/COMBINE-lab/salmon/releases
contains new features, improvements, and bug fixes; please upgrade at your
earliest convenience.
###
Sign up for the salmon mailing list to hear about new versions, features and updates at:
https://oceangenomics.com/subscribe
###

counted k-mers for 10,000 transcripts

counted k-mers for 20,000 transcripts

counted k-mers for 30,000 transcripts

counted k-mers for 40,000 transcripts

counted k-mers for 50,000 transcripts

counted k-mers for 60,000 transcripts

counted k-mers for 70,000 transcripts

counted k-mers for 80,000 transcripts

counted k-mers for 90,000 transcriptsElapsed time: 8.55057s

[2019-07-19 18:50:42.203] [jointLog] [info] Replaced 106,717 non-ATCG nucleotides
[2019-07-19 18:50:42.203] [jointLog] [info] Clipped poly-A tails from 11 transcripts
[2019-07-19 18:50:42.274] [jointLog] [info] Building rank-select dictionary and saving to disk
[2019-07-19 18:50:42.327] [jointLog] [info] done
Elapsed time: 0.0526109s
[2019-07-19 18:50:42.433] [jointLog] [info] Writing sequence data to file . . . 
[2019-07-19 18:50:42.891] [jointLog] [info] done
Elapsed time: 0.458057s
[2019-07-19 18:50:42.893] [jointLog] [info] Building 32-bit suffix array (length of generalized text is 373,366,920)
[2019-07-19 18:50:43.921] [jointLog] [info] Building suffix array . . . 
success
saving to disk . . . done
Elapsed time: 1.95459s
done
Elapsed time: 52.3371s

processed 0 positions

processed 1,000,000 positions

processed 2,000,000 positions

processed 3,000,000 positions

processed 4,000,000 positions

processed 5,000,000 positions

processed 6,000,000 positions

processed 7,000,000 positions

processed 8,000,000 positions

processed 9,000,000 positions

processed 10,000,000 positions

processed 11,000,000 positions

processed 12,000,000 positions

processed 13,000,000 positions

processed 14,000,000 positions

processed 15,000,000 positions

processed 16,000,000 positions

processed 17,000,000 positions

processed 18,000,000 positions

processed 19,000,000 positions

processed 20,000,000 positions

processed 21,000,000 positions

processed 22,000,000 positions

processed 23,000,000 positions

processed 24,000,000 positions

processed 25,000,000 positions

processed 26,000,000 positions

processed 27,000,000 positions

processed 28,000,000 positions

processed 29,000,000 positions

processed 30,000,000 positions

processed 31,000,000 positions

processed 32,000,000 positions

processed 33,000,000 positions

processed 34,000,000 positions

processed 35,000,000 positions

processed 36,000,000 positions

processed 37,000,000 positions

processed 38,000,000 positions

processed 39,000,000 positions

processed 40,000,000 positions

processed 41,000,000 positions

processed 42,000,000 positions

processed 43,000,000 positions

processed 44,000,000 positions

processed 45,000,000 positions

processed 46,000,000 positions

processed 47,000,000 positions

processed 48,000,000 positions

processed 49,000,000 positions

processed 50,000,000 positions

processed 51,000,000 positions

processed 52,000,000 positions

processed 53,000,000 positions

processed 54,000,000 positions

processed 55,000,000 positions

processed 56,000,000 positions

processed 57,000,000 positions

processed 58,000,000 positions

processed 59,000,000 positions

processed 60,000,000 positions

processed 61,000,000 positions

processed 62,000,000 positions

processed 63,000,000 positions

processed 64,000,000 positions

processed 65,000,000 positions

processed 66,000,000 positions

processed 67,000,000 positions

processed 68,000,000 positions

processed 69,000,000 positions

processed 70,000,000 positions

processed 71,000,000 positions

processed 72,000,000 positions

processed 73,000,000 positions

processed 74,000,000 positions

processed 75,000,000 positions

processed 76,000,000 positions

processed 77,000,000 positions

processed 78,000,000 positions

processed 79,000,000 positions

processed 80,000,000 positions

processed 81,000,000 positions

processed 82,000,000 positions

processed 83,000,000 positions

processed 84,000,000 positions

processed 85,000,000 positions

processed 86,000,000 positions

processed 87,000,000 positions

processed 88,000,000 positions

processed 89,000,000 positions

processed 90,000,000 positions

processed 91,000,000 positions

processed 92,000,000 positions

processed 93,000,000 positions

processed 94,000,000 positions

processed 95,000,000 positions

processed 96,000,000 positions

processed 97,000,000 positions

processed 98,000,000 positions

processed 99,000,000 positions

processed 100,000,000 positions

processed 101,000,000 positions

processed 102,000,000 positions

processed 103,000,000 positions

processed 104,000,000 positions

processed 105,000,000 positions

processed 106,000,000 positions

processed 107,000,000 positions

processed 108,000,000 positions

processed 109,000,000 positions

processed 110,000,000 positions

processed 111,000,000 positions

processed 112,000,000 positions

processed 113,000,000 positions

processed 114,000,000 positions

processed 115,000,000 positions

processed 116,000,000 positions

processed 117,000,000 positions

processed 118,000,000 positions

processed 119,000,000 positions

processed 120,000,000 positions

processed 121,000,000 positions

processed 122,000,000 positions

processed 123,000,000 positions

processed 124,000,000 positions

processed 125,000,000 positions

processed 126,000,000 positions

processed 127,000,000 positions

processed 128,000,000 positions

processed 129,000,000 positions

processed 130,000,000 positions

processed 131,000,000 positions

processed 132,000,000 positions

processed 133,000,000 positions

processed 134,000,000 positions

processed 135,000,000 positions

processed 136,000,000 positions

processed 137,000,000 positions

processed 138,000,000 positions

processed 139,000,000 positions

processed 140,000,000 positions

processed 141,000,000 positions

processed 142,000,000 positions

processed 143,000,000 positions

processed 144,000,000 positions

processed 145,000,000 positions

processed 146,000,000 positions

processed 147,000,000 positions

processed 148,000,000 positions

processed 149,000,000 positions

processed 150,000,000 positions

processed 151,000,000 positions

processed 152,000,000 positions

processed 153,000,000 positions

processed 154,000,000 positions

processed 155,000,000 positions

processed 156,000,000 positions

processed 157,000,000 positions

processed 158,000,000 positions

processed 159,000,000 positions

processed 160,000,000 positions

processed 161,000,000 positions

processed 162,000,000 positions

processed 163,000,000 positions

processed 164,000,000 positions

processed 165,000,000 positions

processed 166,000,000 positions

processed 167,000,000 positions

processed 168,000,000 positions

processed 169,000,000 positions

processed 170,000,000 positions

processed 171,000,000 positions

processed 172,000,000 positions

processed 173,000,000 positions

processed 174,000,000 positions

processed 175,000,000 positions

processed 176,000,000 positions

processed 177,000,000 positions

processed 178,000,000 positions

processed 179,000,000 positions

processed 180,000,000 positions

processed 181,000,000 positions

processed 182,000,000 positions

processed 183,000,000 positions

processed 184,000,000 positions

processed 185,000,000 positions

processed 186,000,000 positions

processed 187,000,000 positions

processed 188,000,000 positions

processed 189,000,000 positions

processed 190,000,000 positions

processed 191,000,000 positions

processed 192,000,000 positions

processed 193,000,000 positions

processed 194,000,000 positions

processed 195,000,000 positions

processed 196,000,000 positions

processed 197,000,000 positions

processed 198,000,000 positions

processed 199,000,000 positions

processed 200,000,000 positions

processed 201,000,000 positions

processed 202,000,000 positions

processed 203,000,000 positions

processed 204,000,000 positions

processed 205,000,000 positions

processed 206,000,000 positions

processed 207,000,000 positions

processed 208,000,000 positions

processed 209,000,000 positions

processed 210,000,000 positions

processed 211,000,000 positions

processed 212,000,000 positions

processed 213,000,000 positions

processed 214,000,000 positions

processed 215,000,000 positions

processed 216,000,000 positions

processed 217,000,000 positions

processed 218,000,000 positions

processed 219,000,000 positions

processed 220,000,000 positions

processed 221,000,000 positions

processed 222,000,000 positions

processed 223,000,000 positions

processed 224,000,000 positions

processed 225,000,000 positions

processed 226,000,000 positions

processed 227,000,000 positions

processed 228,000,000 positions

processed 229,000,000 positions

processed 230,000,000 positions

processed 231,000,000 positions

processed 232,000,000 positions

processed 233,000,000 positions

processed 234,000,000 positions

processed 235,000,000 positions

processed 236,000,000 positions

processed 237,000,000 positions

processed 238,000,000 positions

processed 239,000,000 positions

processed 240,000,000 positions

processed 241,000,000 positions

processed 242,000,000 positions

processed 243,000,000 positions

processed 244,000,000 positions

processed 245,000,000 positions

processed 246,000,000 positions

processed 247,000,000 positions

processed 248,000,000 positions

processed 249,000,000 positions

processed 250,000,000 positions

processed 251,000,000 positions

processed 252,000,000 positions

processed 253,000,000 positions

processed 254,000,000 positions

processed 255,000,000 positions

processed 256,000,000 positions

processed 257,000,000 positions

processed 258,000,000 positions

processed 259,000,000 positions

processed 260,000,000 positions

processed 261,000,000 positions

processed 262,000,000 positions

processed 263,000,000 positions

processed 264,000,000 positions

processed 265,000,000 positions

processed 266,000,000 positions

processed 267,000,000 positions

processed 268,000,000 positions

processed 269,000,000 positions

processed 270,000,000 positions

processed 271,000,000 positions

processed 272,000,000 positions

processed 273,000,000 positions

processed 274,000,000 positions

processed 275,000,000 positions

processed 276,000,000 positions

processed 277,000,000 positions

processed 278,000,000 positions

processed 279,000,000 positions

processed 280,000,000 positions

processed 281,000,000 positions

processed 282,000,000 positions

processed 283,000,000 positions

processed 284,000,000 positions

processed 285,000,000 positions

processed 286,000,000 positions

processed 287,000,000 positions

processed 288,000,000 positions

processed 289,000,000 positions

processed 290,000,000 positions

processed 291,000,000 positions

processed 292,000,000 positions

processed 293,000,000 positions

processed 294,000,000 positions

processed 295,000,000 positions

processed 296,000,000 positions

processed 297,000,000 positions

processed 298,000,000 positions

processed 299,000,000 positions

processed 300,000,000 positions

processed 301,000,000 positions

processed 302,000,000 positions

processed 303,000,000 positions

processed 304,000,000 positions

processed 305,000,000 positions

processed 306,000,000 positions

processed 307,000,000 positions

processed 308,000,000 positions

processed 309,000,000 positions

processed 310,000,000 positions

processed 311,000,000 positions

processed 312,000,000 positions

processed 313,000,000 positions

processed 314,000,000 positions

processed 315,000,000 positions

processed 316,000,000 positions

processed 317,000,000 positions

processed 318,000,000 positions

processed 319,000,000 positions

processed 320,000,000 positions

processed 321,000,000 positions

processed 322,000,000 positions

processed 323,000,000 positions

processed 324,000,000 positions

processed 325,000,000 positions

processed 326,000,000 positions

processed 327,000,000 positions

processed 328,000,000 positions

processed 329,000,000 positions

processed 330,000,000 positions

processed 331,000,000 positions

processed 332,000,000 positions

processed 333,000,000 positions

processed 334,000,000 positions

processed 335,000,000 positions

processed 336,000,000 positions

processed 337,000,000 positions

processed 338,000,000 positions

processed 339,000,000 positions

processed 340,000,000 positions

processed 341,000,000 positions

processed 342,000,000 positions

processed 343,000,000 positions

processed 344,000,000 positions

processed 345,000,000 positions

processed 346,000,000 positions

processed 347,000,000 positions

processed 348,000,000 positions

processed 349,000,000 positions

processed 350,000,000 positions

processed 351,000,000 positions

processed 352,000,000 positions

processed 353,000,000 positions

processed 354,000,000 positions

processed 355,000,000 positions

processed 356,000,000 positions

processed 357,000,000 positions

processed 358,000,000 positions

processed 359,000,000 positions

processed 360,000,000 positions

processed 361,000,000 positions

processed 362,000,000 positions

processed 363,000,000 positions

processed 364,000,000 positions

processed 365,000,000 positions

processed 366,000,000 positions

processed 367,000,000 positions

processed 368,000,000 positions

processed 369,000,000 positions

processed 370,000,000 positions

processed 371,000,000 positions

processed 372,000,000 positions

processed 373,000,000 positions[2019-07-19 18:56:18.370] [jointLog] [info] khash had 368,374,780 keys
[2019-07-19 18:56:19.058] [jointLog] [info] saving hash to disk . . . 
[2019-07-19 18:57:04.785] [jointLog] [info] done
Elapsed time: 45.7269s
[2019-07-19 18:57:08.751] [jLog] [info] done building index

########################################################################################################################
#####                           ALIGNING READS FROM ALL SAMPLES BACK TO BINS WITH SALMON                           #####
########################################################################################################################

mkdir: cannot create directory `QUANT_BINS/alignment_files': File exists

------------------------------------------------------------------------------------------------------------------------
-----                processing sample COMBINED with reads COMBINED_1.fastq and COMBINED_2.fastq...                -----
------------------------------------------------------------------------------------------------------------------------

### salmon (mapping-based) v0.13.1
### [ program ] => salmon 
### [ command ] => quant 
### [ index ] => { QUANT_BINS/assembly_index }
### [ libType ] => { IU }
### [ mates1 ] => { COMBINED_1.fastq }
### [ mates2 ] => { COMBINED_2.fastq }
### [ output ] => { QUANT_BINS/alignment_files/COMBINED.quant }
### [ meta ] => { }
### [ threads ] => { 1 }
Version Info: ### PLEASE UPGRADE SALMON ###
### A newer version of salmon with important bug fixes and improvements is available. ####
###
The newest version, available at https://github.com/COMBINE-lab/salmon/releases
contains new features, improvements, and bug fixes; please upgrade at your
earliest convenience.
###
Sign up for the salmon mailing list to hear about new versions, features and updates at:
https://oceangenomics.com/subscribe
###
Logs will be written to QUANT_BINS/alignment_files/COMBINED.quant/logs
[2019-07-19 18:57:09.351] [jointLog] [info] Fragment incompatibility prior below threshold.  Incompatible fragments will be ignored.
[2019-07-19 18:57:09.351] [jointLog] [warning] 

NOTE: It appears you are running salmon without the `--validateMappings` option.
Mapping validation can generally improve both the sensitivity and specificity of mapping,
with only a moderate increase in use of computational resources. 
Mapping validation is planned to become a default option (i.e. turned on by default) in
the next release of salmon.
Unless there is a specific reason to do this (e.g. testing on clean simulated data),
`--validateMappings` is generally recommended.

[2019-07-19 18:57:09.351] [jointLog] [info] parsing read library format
[2019-07-19 18:57:09.351] [jointLog] [info] There is 1 library.
[2019-07-19 18:57:09.398] [jointLog] [info] Loading Quasi index
[2019-07-19 18:57:09.398] [jointLog] [info] Loading 32-bit quasi index
[2019-07-19 18:57:09.399] [stderrLog] [info] Loading Suffix Array 
[2019-07-19 18:57:10.292] [stderrLog] [info] Loading Transcript Info 
[2019-07-19 18:57:10.510] [stderrLog] [info] Loading Rank-Select Bit Array
[2019-07-19 18:57:10.547] [stderrLog] [info] There were 95,095 set bits in the bit array
[2019-07-19 18:57:10.567] [stderrLog] [info] Computing transcript lengths
[2019-07-19 18:57:10.567] [stderrLog] [info] Waiting to finish loading hash

[2019-07-19 18:57:38.858] [jointLog] [info] done
[2019-07-19 18:57:38.858] [jointLog] [info] Index contained 95,095 targets
[2019-07-19 18:57:38.858] [stderrLog] [info] Done loading index


processed 500,000 fragments
hits: 411,681, hits per frag:  0.831679

processed 1,000,000 fragments
hits: 823,679, hits per frag:  0.827818

processed 1,500,000 fragments
hits: 1,234,877, hits per frag:  0.826005

processed 2,000,000 fragments
hits: 1,646,568, hits per frag:  0.825347

processed 2,500,000 fragments
hits: 2,057,886, hits per frag:  0.824804

processed 3,000,000 fragments
hits: 2,468,839, hits per frag:  0.82432

processed 3,500,000 fragments
hits: 2,880,457, hits per frag:  0.824165

processed 4,000,000 fragments
hits: 3,292,383, hits per frag:  0.824126

processed 4,500,000 fragments
hits: 3,703,745, hits per frag:  0.82397

processed 5,000,000 fragments
hits: 4,115,337, hits per frag:  0.823891

processed 5,500,000 fragments
hits: 4,526,782, hits per frag:  0.8238

processed 6,000,000 fragments
hits: 4,938,283, hits per frag:  0.823734

processed 6,500,000 fragments
hits: 5,349,737, hits per frag:  0.82367

processed 7,000,000 fragments
hits: 5,761,362, hits per frag:  0.82364

processed 7,500,000 fragments
hits: 6,172,579, hits per frag:  0.82356

processed 8,000,000 fragments
hits: 6,583,427, hits per frag:  0.823443

processed 8,500,000 fragments
hits: 6,994,736, hits per frag:  0.823394

processed 9,000,000 fragments
hits: 7,406,373, hits per frag:  0.823388

processed 9,500,000 fragments
hits: 7,817,668, hits per frag:  0.823346

processed 10,000,000 fragments
hits: 8,229,220, hits per frag:  0.823334

processed 10,500,000 fragments
hits: 8,639,949, hits per frag:  0.823244

processed 11,000,000 fragments
hits: 9,050,973, hits per frag:  0.82319

processed 11,500,000 fragments
hits: 9,462,224, hits per frag:  0.82316

processed 12,000,000 fragments
hits: 9,873,705, hits per frag:  0.823152

processed 12,500,000 fragments
hits: 10,284,788, hits per frag:  0.823112

processed 13,000,000 fragments
hits: 10,696,046, hits per frag:  0.823089

processed 13,500,000 fragments
hits: 11,107,227, hits per frag:  0.823062

processed 14,000,000 fragments
hits: 11,518,588, hits per frag:  0.82305

processed 14,500,000 fragments
hits: 11,930,363, hits per frag:  0.823067

processed 15,000,000 fragments
hits: 12,341,470, hits per frag:  0.823039

processed 15,500,000 fragments
hits: 12,753,026, hits per frag:  0.823041

processed 16,000,000 fragments
hits: 13,165,186, hits per frag:  0.823081

processed 16,500,000 fragments
hits: 13,576,824, hits per frag:  0.823087

processed 17,000,000 fragments
hits: 13,987,984, hits per frag:  0.823065

processed 17,500,000 fragments
hits: 14,358,668, hits per frag:  0.82073

processed 18,000,000 fragments
hits: 14,361,535, hits per frag:  0.798085

processed 18,500,000 fragments
hits: 14,364,347, hits per frag:  0.776661

processed 19,000,000 fragments
hits: 14,367,126, hits per frag:  0.756364

processed 19,500,000 fragments
hits: 14,369,988, hits per frag:  0.737111

processed 20,000,000 fragments
hits: 14,372,880, hits per frag:  0.718824

processed 20,500,000 fragments
hits: 14,375,802, hits per frag:  0.70143

processed 21,000,000 fragments
hits: 14,378,622, hits per frag:  0.684859

processed 21,500,000 fragments
hits: 14,381,547, hits per frag:  0.669065

processed 22,000,000 fragments
hits: 14,384,383, hits per frag:  0.653984

processed 22,500,000 fragments
hits: 14,387,201, hits per frag:  0.639573

processed 23,000,000 fragments
hits: 14,389,984, hits per frag:  0.625787

processed 23,500,000 fragments
hits: 14,392,924, hits per frag:  0.612595

processed 24,000,000 fragments
hits: 14,395,881, hits per frag:  0.599953

processed 24,500,000 fragments
hits: 14,398,762, hits per frag:  0.587825

processed 25,000,000 fragments
hits: 14,401,615, hits per frag:  0.57618

processed 25,500,000 fragments
hits: 14,404,417, hits per frag:  0.56499

processed 26,000,000 fragments
hits: 14,407,318, hits per frag:  0.554234

processed 26,500,000 fragments
hits: 14,410,165, hits per frag:  0.543882

processed 27,000,000 fragments
hits: 14,412,977, hits per frag:  0.533913

processed 27,500,000 fragments
hits: 14,415,833, hits per frag:  0.524307

processed 28,000,000 fragments
hits: 14,418,634, hits per frag:  0.515043

processed 28,500,000 fragments
hits: 14,421,535, hits per frag:  0.506108

processed 29,000,000 fragments
hits: 14,424,356, hits per frag:  0.497477

processed 29,500,000 fragments
hits: 14,427,259, hits per frag:  0.489143

processed 30,000,000 fragments
hits: 14,430,216, hits per frag:  0.481087

processed 30,500,000 fragments
hits: 14,433,136, hits per frag:  0.473295

processed 31,000,000 fragments
hits: 14,435,939, hits per frag:  0.465751

processed 31,500,000 fragments
hits: 14,438,797, hits per frag:  0.458447

processed 32,000,000 fragments
hits: 14,441,623, hits per frag:  0.451371

processed 32,500,000 fragments
hits: 14,444,460, hits per frag:  0.444513

processed 33,000,000 fragments
hits: 14,447,297, hits per frag:  0.437863

processed 33,500,000 fragments
hits: 14,450,187, hits per frag:  0.431413

processed 34,000,000 fragments
hits: 14,453,081, hits per frag:  0.425153

processed 34,500,000 fragments
hits: 14,456,048, hits per frag:  0.419077

processed 35,000,000 fragments
hits: 14,769,524, hits per frag:  0.422047

processed 35,500,000 fragments
hits: 15,202,770, hits per frag:  0.428307

processed 36,000,000 fragments
hits: 15,635,643, hits per frag:  0.434384

processed 36,500,000 fragments
hits: 16,069,108, hits per frag:  0.44031

processed 37,000,000 fragments
hits: 16,502,008, hits per frag:  0.446061

processed 37,500,000 fragments
hits: 16,935,048, hits per frag:  0.451661

processed 38,000,000 fragments
hits: 17,368,119, hits per frag:  0.457116

processed 38,500,000 fragments
hits: 17,801,648, hits per frag:  0.462441

processed 39,000,000 fragments
hits: 18,234,445, hits per frag:  0.46761

processed 39,500,000 fragments
hits: 18,668,353, hits per frag:  0.472676

processed 40,000,000 fragments
hits: 19,101,585, hits per frag:  0.477599

processed 40,500,000 fragments
hits: 19,535,258, hits per frag:  0.482412

processed 41,000,000 fragments
hits: 19,968,873, hits per frag:  0.487105

processed 41,500,000 fragments
hits: 20,402,095, hits per frag:  0.491676

processed 42,000,000 fragments
hits: 20,835,166, hits per frag:  0.496134

processed 42,500,000 fragments
hits: 21,268,332, hits per frag:  0.50049

processed 43,000,000 fragments
hits: 21,701,646, hits per frag:  0.504748

processed 43,500,000 fragments
hits: 22,135,090, hits per frag:  0.508911

processed 44,000,000 fragments
hits: 22,568,600, hits per frag:  0.512981

processed 44,500,000 fragments
hits: 23,001,807, hits per frag:  0.516953

processed 45,000,000 fragments
hits: 23,435,211, hits per frag:  0.52084

processed 45,500,000 fragments
hits: 23,868,553, hits per frag:  0.524641

processed 46,000,000 fragments
hits: 24,301,401, hits per frag:  0.528349

processed 46,500,000 fragments
hits: 24,734,549, hits per frag:  0.531983

processed 47,000,000 fragments
hits: 25,167,162, hits per frag:  0.535528

processed 47,500,000 fragments
hits: 25,599,944, hits per frag:  0.539003

processed 48,000,000 fragments
hits: 26,032,497, hits per frag:  0.5424

processed 48,500,000 fragments
hits: 26,466,061, hits per frag:  0.545748

processed 49,000,000 fragments
hits: 26,899,001, hits per frag:  0.549015

processed 49,500,000 fragments
hits: 27,332,361, hits per frag:  0.552225

processed 50,000,000 fragments
hits: 27,765,620, hits per frag:  0.555368

processed 50,500,000 fragments
hits: 28,013,894, hits per frag:  0.554785

processed 51,000,000 fragments
hits: 28,015,739, hits per frag:  0.549382

processed 51,500,000 fragments
hits: 28,017,533, hits per frag:  0.544083

processed 52,000,000 fragments
hits: 28,019,378, hits per frag:  0.538886

processed 52,500,000 fragments
hits: 28,021,257, hits per frag:  0.533789

processed 53,000,000 fragments
hits: 28,023,163, hits per frag:  0.528789

processed 53,500,000 fragments
hits: 28,024,992, hits per frag:  0.523881

processed 54,000,000 fragments
hits: 28,026,897, hits per frag:  0.519065

processed 54,500,000 fragments
hits: 28,028,707, hits per frag:  0.514335

processed 55,000,000 fragments
hits: 28,030,606, hits per frag:  0.509694

processed 55,500,000 fragments
hits: 28,032,517, hits per frag:  0.505136

processed 56,000,000 fragments
hits: 28,034,320, hits per frag:  0.500658

processed 56,500,000 fragments
hits: 28,036,159, hits per frag:  0.496259

processed 57,000,000 fragments
hits: 28,037,966, hits per frag:  0.491937

processed 57,500,000 fragments
hits: 28,039,862, hits per frag:  0.487692

processed 58,000,000 fragments
hits: 28,041,732, hits per frag:  0.48352

processed 58,500,000 fragments
hits: 28,043,649, hits per frag:  0.47942

processed 59,000,000 fragments
hits: 28,045,478, hits per frag:  0.475387

processed 59,500,000 fragments
hits: 28,047,277, hits per frag:  0.471422

processed 60,000,000 fragments
hits: 28,049,174, hits per frag:  0.467525

processed 60,500,000 fragments
hits: 28,050,976, hits per frag:  0.463691

processed 61,000,000 fragments
hits: 28,052,810, hits per frag:  0.45992

processed 61,500,000 fragments
hits: 28,054,701, hits per frag:  0.456211

processed 62,000,000 fragments
hits: 28,056,562, hits per frag:  0.452562

processed 62,500,000 fragments
hits: 28,058,375, hits per frag:  0.44897

processed 63,000,000 fragments
hits: 28,060,127, hits per frag:  0.445434

processed 63,500,000 fragments
hits: 28,061,978, hits per frag:  0.441956

processed 64,000,000 fragments
hits: 28,063,854, hits per frag:  0.438532

processed 64,500,000 fragments
hits: 28,065,691, hits per frag:  0.435161

processed 65,000,000 fragments
hits: 28,067,536, hits per frag:  0.431841

processed 65,500,000 fragments
hits: 28,069,436, hits per frag:  0.428574

processed 66,000,000 fragments
hits: 28,071,299, hits per frag:  0.425355

[2019-07-19 19:21:23.685] [jointLog] [info] Computed 106,862 rich equivalence classes for further processing
[2019-07-19 19:21:23.693] [jointLog] [info] Counted 27,761,265 total reads in the equivalence classes 
[2019-07-19 19:21:23.694] [jointLog] [info] Number of fragments discarded because they have only dovetail (discordant) mappings : 285,945
[2019-07-19 19:21:23.694] [jointLog] [info] Mapping rate = 41.8724%

[2019-07-19 19:21:23.695] [jointLog] [info] finished quantifyLibrary()
[2019-07-19 19:21:24.296] [jointLog] [info] Starting optimizer
[2019-07-19 19:21:24.423] [jointLog] [info] Marked 0 weighted equivalence classes as degenerate
[2019-07-19 19:21:24.426] [jointLog] [info] iteration = 0 | max rel diff. = 572.866
[2019-07-19 19:21:24.637] [jointLog] [info] iteration = 100 | max rel diff. = 0.00324883
[2019-07-19 19:21:24.639] [jointLog] [info] Finished optimizer
[2019-07-19 19:21:24.639] [jointLog] [info] writing output 

------------------------------------------------------------------------------------------------------------------------
-----                                           summarize salmon files...                                          -----
------------------------------------------------------------------------------------------------------------------------

Starting in: /scratch/users/nus/bchref/COMBINED/QUANT_BINS/alignment_files
Loading counts from: ./M26.quant quant.sf
Loading counts from: ./COMBINED.quant quant.sf
Loading counts from: ./M22.quant quant.sf
Loading counts from: ./M5.quant quant.sf
Loading counts from: ./M25.quant quant.sf
"COMBINED.quant.counts",
"M22.quant.counts",
"M25.quant.counts",
"M26.quant.counts",
"M5.quant.counts"
mkdir: cannot create directory `QUANT_BINS/quant_files': File exists

########################################################################################################################
#####                                   EXTRACTING AVERAGE ABUNDANCE OF EACH BIN                                   #####
########################################################################################################################

------------------------------------------------------------------------------------------------------------------------
-----                            There were 5 samples detected. Making abundance table!                            -----
------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------
-----                     Average bin abundance table stored in QUANT_BINS/abundance_table.tab                     -----
------------------------------------------------------------------------------------------------------------------------

########################################################################################################################
#####                                 MAKING GENOME ABUNDANCE HEATMAP WITH SEABORN                                 #####
########################################################################################################################

------------------------------------------------------------------------------------------------------------------------
-----                                          making heatmap with Seaborn                                         -----
------------------------------------------------------------------------------------------------------------------------

/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/pandas/core/computation/check.py:19: UserWarning: The installed version of numexpr 2.6.0 is not supported in pandas and will be not be used
The minimum supported version is 2.6.1

  ver=ver, min_ver=_MIN_NUMEXPR_VERSION), UserWarning)
loading libs...
loading abundance data...
drawing clustermap...
Traceback (most recent call last):
  File "/scratch/users/nus/bchref/.conda/metawrap/bin/metawrap-scripts/make_heatmap.py", line 75, in <module>
    draw_clustermap(df, lut)
  File "/scratch/users/nus/bchref/.conda/metawrap/bin/metawrap-scripts/make_heatmap.py", line 58, in draw_clustermap
    g = sns.clustermap(df, figsize=(14,8), col_cluster=True, yticklabels=True, cmap="magma")
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/seaborn/matrix.py", line 1301, in clustermap
    **kwargs)
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/seaborn/matrix.py", line 1128, in plot
    row_linkage=row_linkage, col_linkage=col_linkage)
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/seaborn/matrix.py", line 1021, in plot_dendrograms
    ax=self.ax_row_dendrogram, rotate=True, linkage=row_linkage)
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/seaborn/matrix.py", line 747, in dendrogram
    label=label, rotate=rotate)
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/seaborn/matrix.py", line 564, in __init__
    self.linkage = self.calculated_linkage
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/seaborn/matrix.py", line 628, in calculated_linkage
    return self._calculate_linkage_scipy()
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/seaborn/matrix.py", line 603, in _calculate_linkage_scipy
    metric=self.metric)
  File "/scratch/users/nus/bchref/.conda/metawrap-env/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 1109, in linkage
    raise ValueError("The condensed distance matrix must contain only "
ValueError: The condensed distance matrix must contain only finite values.

************************************************************************************************************************
*****                           something went wrong with making the heatmap. Exiting...                           *****
************************************************************************************************************************

What does this mean? I'm not sure what happened. Thanks!

ursky commented 5 years ago

Can you have a look at abundance_table.tab and see if there is anything strange about it? Could you post it here?

raissafedora commented 5 years ago

Hi Ursky

Genomic bins    M5  M25 M26 COMBINED    M22
bin.9   0.0 0.0 60.397918   25.213254   0.002815
bin.20  0.0 0.0 160.459377  133.743908  117.441425
bin.4   0.0 0.0 2.296145    5.013589    6.889901
bin.25  0.0 0.0 129.174375  79.186493   45.406463
bin.22  0.0 0.0 21.071283   12.500952   6.111791
bin.10  0.0 0.0 51.188237   28.934029   13.071642
bin.27  0.0 0.0 1.11595 8.743348    13.618427
bin.15  0.0 0.0 16.537576   16.235004   15.926159
bin.17  0.0 0.0 121.04374   50.553649   0.01002
bin.3   0.0 0.0 4.712288    5.321462    5.706704
bin.2   0.0 0.0 134.549137  101.038521  76.485008
bin.5   0.0 0.0 33.424272   25.822484   20.044773
bin.11  0.0 0.0 13.210436   12.687355   12.03634
bin.7   0.0 0.0 12.746236   7.747318    4.24715
bin.14  0.0 0.0 15.078543   6.714839    0.686083
bin.13  0.0 0.0 0.268219    2.852411    4.642532
bin.18  0.0 0.0 136.091313  105.743339  82.574149
bin.21  0.0 0.0 2.503962    22.360288   35.957433
bin.16  0.0 0.0 71.384164   49.288246   33.532732
bin.8   0.0 0.0 32.463425   21.15258    13.438886
bin.6   0.0 0.0 67.698038   44.107235   27.131392
bin.12  0.0 0.0 47.894945   21.382816   2.450009
bin.1   0.0 0.0 9.884207    10.757614   11.250775
bin.24  0.0 0.0 172.14749   107.025751  60.037034
bin.26  0.0 0.0 3.507508    1.764115    0.47934
bin.19  0.0 0.0 3.596521    14.547957   22.254873
bin.23  0.0 0.0 10.178119   37.82982    57.238947

would the misalignment be the problem? I am not sure whether this happens randomly or not, though.

ursky commented 5 years ago

Looks like two of your columns are only 0s, so they come up as NaN after standardization. I'll change the script to allow for this, but you might want to check why this is the case.

ursky commented 5 years ago

Maybe you have some old .quant files in the directory?

raissafedora commented 5 years ago

No I don't think so. I might be too stringent for the refinement (95% completion 5% contamination). I thought that coming from the same source (different time points) allows me to set a stringent threshold for binning.