search

statisticalbiotechnology / quandenser

QUANtification by Distillation for ENhanced Signals with Error Regulation
Apache License 2.0
9 stars 1 forks source link

Percolator error #12

Closed andrewjmc closed 4 years ago

andrewjmc commented 4 years ago

Hello,

In a larger run of quandenser with files from three runs in two labs (alignments previous shared in issue #5) I hit a snag with a percolator run:

Percolator version 3.02.1, Build Date May 27 2019 09:50:48
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms quandenser_output\percolator/search_and_link_59_to_72.psms.pout --decoy-results-psms quandenser_output\percolator/search_and_link_59_to_72.psms.pout.decoys
Started Wed Dec 18 19:19:17 2019
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 12955 positives and 1312 negatives, size ratio=9.87424 and pi0=1
Selecting Cpos by cross-validation.
Selecting Cneg by cross-validation.
Found 10771 test set positives with q<0.02 in initial direction
Reading in data and feature calculation took 0.052 cpu seconds or 0 seconds wall clock time.
---Training with Cpos selected by cross validation, Cneg selected by cross validation, initial_fdr=0.02, fdr=0.02
Iteration 1:    Estimated 12103 PSMs with q<0.02
Iteration 2:    Estimated 12092 PSMs with q<0.02
Iteration 3:    Estimated 12093 PSMs with q<0.02
Iteration 4:    Estimated 12093 PSMs with q<0.02
Iteration 5:    Estimated 12093 PSMs with q<0.02
Iteration 6:    Estimated 12093 PSMs with q<0.02
Iteration 7:    Estimated 12093 PSMs with q<0.02
Iteration 8:    Estimated 12093 PSMs with q<0.02
Iteration 9:    Estimated 12093 PSMs with q<0.02
Iteration 10:   Estimated 12093 PSMs with q<0.02
Learned normalized SVM weights for the 3 cross-validation splits:
 Split1  Split2  Split3 FeatureName
-2.4057 -3.0399 -3.2933 ppmDiff
 0.0151  0.0357  0.0614 rtDiff
-0.1766 -0.0463 -0.1890 precMz
-0.0511 -0.1114 -0.0613 rTime
 0.0000  0.0000  0.0000 queryIsPlaceHolder
 0.0000  0.0000  0.0000 targetIsPlaceHolder
 0.0891  0.0889  0.1302 charge1
-0.0154 -0.0244 -0.0618 charge2
-0.1260 -0.1273 -0.1379 charge3
-0.0253 -0.0100 -0.0169 charge4plus
 0.0303 -0.1723 -0.4573 m0
Found 12073 test set PSMs with q<0.02.
Selected best-scoring PSM per scan+expMass (target-decoy competition): 12879 target PSMs and 851 decoy PSMs.
Calculating q values.
Final list yields 12193 target PSMs with q<0.02.
Calculating posterior error probabilities (PEPs).
Processing took 6.869 cpu seconds or 7 seconds wall clock time.
Links before 22833
Links after 24635
Added 50968 placeholders.
Matching features 60->8
Features:
ppmDiff rtDiff precMz rTime queryIsPlaceHolder targetIsPlaceHolder charge1 charge2 charge3 charge4plus
Percolator version 3.02.1, Build Date May 27 2019 09:50:48
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms quandenser_output\percolator/link_60_to_8.psms --decoy-results-psms quandenser_output\percolator/link_60_to_8.psms.decoys
Started Wed Dec 18 19:19:25 2019
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 0 positives and 0 negatives, size ratio=-nan(ind) and pi0=1
Exception caught: Error: no target PSMs were provided.

I presume this might indicate a low number of shared features? Is there any way to make the process work in spite of this?

My first guess is perhaps I need to increase the size of the alignment window.

Best wishes,

Andrew

MatthewThe commented 4 years ago

That's a bit odd, could you perhaps share the entire log file. There might be some information about the alignment earlier in the log file.

There is a flag to set the retention time window even larger: --align-rtime-tol, but this by default is already set to 10*stdev, where stdev is the standard deviation of aligned MS2 spectra for the particular alignment, so stdev should already be rather large if the alignment between these runs was not so good.

andrewjmc commented 4 years ago

I can't find a log file, but here are the relevant percolator files if they will help, 59_to_72.zip

andrewjmc commented 4 years ago

Scrolling back through the available log in the console, I see this which looks suspicious:

Inserting link 0 to 52 with rmse nan
Inserting link 0 to 55 with rmse nan
Inserting link 0 to 63 with rmse nan
Inserting link 0 to 65 with rmse nan
Inserting link 0 to 66 with rmse nan
Inserting link 0 to 82 with rmse nan
Inserting link 0 to 86 with rmse nan
Inserting link 0 to 98 with rmse nan
Inserting link 0 to 102 with rmse nan
Inserting link 0 to 119 with rmse nan
Inserting link 0 to 128 with rmse nan
Inserting link 0 to 134 with rmse nan
Inserting link 0 to 141 with rmse nan
Inserting link 0 to 145 with rmse nan
Inserting link 1 to 55 with rmse nan
Inserting link 1 to 61 with rmse nan
Inserting link 1 to 67 with rmse nan
Inserting link 1 to 75 with rmse nan
Inserting link 1 to 80 with rmse nan
Inserting link 1 to 81 with rmse nan
Inserting link 1 to 84 with rmse nan
Inserting link 1 to 85 with rmse nan
Inserting link 1 to 89 with rmse nan
Inserting link 1 to 95 with rmse nan
Inserting link 1 to 100 with rmse nan
Inserting link 1 to 104 with rmse nan
Inserting link 1 to 111 with rmse nan
Inserting link 1 to 114 with rmse nan
Inserting link 1 to 142 with rmse nan
Inserting link 1 to 143 with rmse nan
Inserting link 2 to 65 with rmse nan
Inserting link 2 to 54 with rmse nan
Inserting link 2 to 56 with rmse nan
Inserting link 2 to 74 with rmse nan
Inserting link 2 to 78 with rmse nan
Inserting link 2 to 96 with rmse nan
Inserting link 2 to 107 with rmse nan
Inserting link 2 to 144 with rmse nan
Inserting link 3 to 52 with rmse nan
Inserting link 3 to 83 with rmse nan
Inserting link 3 to 90 with rmse nan
Inserting link 3 to 97 with rmse nan
Inserting link 3 to 101 with rmse nan
Inserting link 3 to 129 with rmse nan
Inserting link 4 to 54 with rmse nan
Inserting link 4 to 62 with rmse nan
Inserting link 4 to 64 with rmse nan
Inserting link 4 to 73 with rmse nan
Inserting link 4 to 123 with rmse nan
Inserting link 4 to 133 with rmse nan
Inserting link 5 to 54 with rmse nan
Inserting link 6 to 54 with rmse nan
Inserting link 6 to 87 with rmse nan
Inserting link 7 to 52 with rmse nan
Inserting link 7 to 57 with rmse nan
Inserting link 7 to 126 with rmse nan
Inserting link 8 to 54 with rmse nan
Inserting link 8 to 60 with rmse nan
Inserting link 9 to 52 with rmse nan
Inserting link 54 to 72 with rmse 0.0125221
Inserting link 72 to 88 with rmse 0.0135621
Inserting link 53 to 72 with rmse 0.0163098
Inserting link 68 to 72 with rmse 0.0179118
Inserting link 59 to 72 with rmse 0.0226068
Inserting link 72 to 94 with rmse 0.02522
Inserting link 72 to 105 with rmse 0.0272367
Inserting link 72 to 91 with rmse 0.0276863
Inserting link 72 to 79 with rmse 0.028457
Inserting link 72 to 93 with rmse 0.0287593
Inserting link 72 to 92 with rmse 0.0299834
Inserting link 99 to 114 with rmse 0.0306238
Inserting link 72 to 76 with rmse 0.0337567
Inserting link 102 to 117 with rmse 0.0340731
Inserting link 52 to 106 with rmse 0.0369443
Inserting link 71 to 72 with rmse 0.0420174
Inserting link 71 to 77 with rmse 0.0453543
Inserting link 70 to 79 with rmse 0.0469063
Inserting link 69 to 94 with rmse 0.0610856
Inserting link 126 to 131 with rmse 0.0728114
Inserting link 125 to 126 with rmse 0.0767181
Inserting link 120 to 125 with rmse 0.0736985
Inserting link 116 to 120 with rmse 0.0696426
Inserting link 115 to 120 with rmse 0.0771026
Inserting link 122 to 125 with rmse 0.0788148
Inserting link 108 to 122 with rmse 0.0765662
Inserting link 108 to 110 with rmse 0.0755593
Inserting link 116 to 121 with rmse 0.0817587
Inserting link 113 to 121 with rmse 0.0805299
Inserting link 112 to 113 with rmse 0.0739824
Inserting link 122 to 127 with rmse 0.0823003
Inserting link 109 to 120 with rmse 0.083077
Inserting link 112 to 118 with rmse 0.0862196
Inserting link 116 to 130 with rmse 0.0888808
Inserting link 130 to 135 with rmse 0.0791985
Inserting link 135 to 137 with rmse 0.0681626
Inserting link 122 to 124 with rmse 0.0926709
Inserting link 112 to 140 with rmse 0.0947413
Inserting link 139 to 140 with rmse 0.0930341
Inserting link 136 to 139 with rmse 0.0787865
Inserting link 136 to 138 with rmse 0.0935104
Inserting link 108 to 132 with rmse 0.100333
Inserting link 72 to 103 with rmse 0.105192
Inserting link 57 to 58 with rmse 0.200056
Inserting link 3 to 35 with rmse 0.895922
Inserting link 5 to 23 with rmse 0.931392
Inserting link 35 to 50 with rmse 0.95337
Inserting link 7 to 12 with rmse 0.958579
Inserting link 5 to 29 with rmse 0.983814
Inserting link 28 to 29 with rmse 0.92142
Inserting link 28 to 30 with rmse 0.941314
Inserting link 20 to 29 with rmse 0.965001
Inserting link 25 to 30 with rmse 1.01299
Inserting link 16 to 25 with rmse 0.952276
Inserting link 50 to 51 with rmse 1.01658
Inserting link 49 to 51 with rmse 0.889802
Inserting link 48 to 51 with rmse 1.0241
Inserting link 26 to 30 with rmse 1.02472
Inserting link 26 to 27 with rmse 1.02314
Inserting link 47 to 50 with rmse 1.0396
Inserting link 17 to 26 with rmse 1.09697
Inserting link 17 to 19 with rmse 1.06998
Inserting link 19 to 24 with rmse 0.783992
Inserting link 14 to 24 with rmse 0.956417
Inserting link 15 to 19 with rmse 1.0254
Inserting link 3 to 38 with rmse 1.09978
Inserting link 21 to 29 with rmse 1.10925
Inserting link 21 to 32 with rmse 1.08957
Inserting link 31 to 32 with rmse 0.901579
Inserting link 29 to 41 with rmse 1.11901
Inserting link 11 to 30 with rmse 1.22141
Inserting link 18 to 114 with rmse 1.23279
Inserting link 28 to 46 with rmse 1.23693
Inserting link 12 to 13 with rmse 1.24026
Inserting link 37 to 142 with rmse 1.24845
Inserting link 22 to 114 with rmse 1.27794
Inserting link 14 to 42 with rmse 1.30242
Inserting link 36 to 37 with rmse 1.34994
Inserting link 31 to 45 with rmse 1.35885
Inserting link 17 to 39 with rmse 1.36557
Inserting link 43 to 85 with rmse 1.41355
Inserting link 26 to 40 with rmse 1.46204
Inserting link 10 to 129 with rmse 1.7054
Inserting link 9 to 44 with rmse 1.72417
Inserting link 3 to 33 with rmse 1.75469
Inserting link 6 to 34 with rmse 1.96057
andrewjmc commented 4 years ago

Though 59 to 72 has a rmse: Inserting link 59 to 72 with rmse 0.0226068

andrewjmc commented 4 years ago

Here is a more extensive section of the log:

Matching features 59->72
Features:
ppmDiff rtDiff precMz rTime queryIsPlaceHolder targetIsPlaceHolder charge1 charge2 charge3 charge4plus
Percolator version 3.02.1, Build Date May 27 2019 09:50:48
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms quandenser_output\percolator/link_59_to_72.psms --decoy-results-psms quandenser_output\percolator/link_59_to_72.psms.decoys
Started Wed Dec 18 19:18:04 2019
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 33430 positives and 1975 negatives, size ratio=16.9266 and pi0=1
Selecting Cpos by cross-validation.
Selecting Cneg by cross-validation.
Found 32437 test set positives with q<0.02 in initial direction
Reading in data and feature calculation took 0.091 cpu seconds or 0 seconds wall clock time.
---Training with Cpos selected by cross validation, Cneg selected by cross validation, initial_fdr=0.02, fdr=0.02
Iteration 1:    Estimated 32415 PSMs with q<0.02
Iteration 2:    Estimated 32415 PSMs with q<0.02
Iteration 3:    Estimated 32415 PSMs with q<0.02
Iteration 4:    Estimated 32415 PSMs with q<0.02
Iteration 5:    Estimated 32415 PSMs with q<0.02
Iteration 6:    Estimated 32415 PSMs with q<0.02
Iteration 7:    Estimated 32415 PSMs with q<0.02
Iteration 8:    Estimated 32415 PSMs with q<0.02
Iteration 9:    Estimated 32415 PSMs with q<0.02
Iteration 10:   Estimated 32415 PSMs with q<0.02
Learned normalized SVM weights for the 3 cross-validation splits:
 Split1  Split2  Split3 FeatureName
-1.4632 -1.5799 -1.5884 ppmDiff
 0.0045  0.0048  0.0071 rtDiff
-0.0553 -0.0547 -0.0626 precMz
-0.0496 -0.0656 -0.1353 rTime
 0.0000  0.0000  0.0000 queryIsPlaceHolder
-0.1126 -0.0929 -0.0987 targetIsPlaceHolder
 0.0956  0.1051  0.1235 charge1
-0.0719 -0.0832 -0.0908 charge2
-0.1331 -0.1253 -0.1221 charge3
 0.0362  0.0294  0.0075 charge4plus
 0.5352  0.4937  0.5233 m0
Found 32415 test set PSMs with q<0.02.
Less identifications (32415 vs 32437) after percolator processing than before processing
Resetting score vector, using default vector. Use --override flag to prevent this.
Selected best-scoring PSM per scan+expMass (target-decoy competition): 23678 target PSMs and 1070 decoy PSMs.
Calculating q values.
Final list yields 23090 target PSMs with q<0.02.
Calculating posterior error probabilities (PEPs).
Processing took 8.712 cpu seconds or 9 seconds wall clock time.
Links before 0
Links after 22833
java -Xmx56G -jar "C:\Program Files\quandenser-v0-02\share\java/Dinosaur-1.1.3.free.jar" --force --concurrency=11 --profiling=true --nReport=0 --outDir=quandenser_output\percolator/search_and_link_59_to_72.psms.pout_dinosaur --advParams="C:\Program Files\quandenser-v0-02\share\java/advParams_dinosaur_targeted.txt" --mode=target --targets=quandenser_output\percolator/search_and_link_59_to_72.psms.pout.dinosaur_targets.tsv E:\RAW\Oxford\FL603_MSQ1101_Honglei_Kawasaki_28.mzML
Dinosaur 1.1.3    built:${maven.build.timestamp}
  mzML file: E:\RAW\Oxford\FL603_MSQ1101_Honglei_Kawasaki_28.mzML
    out dir: quandenser_output\percolator/search_and_link_59_to_72.psms.pout_dinosaur
   out name: FL603_MSQ1101_Honglei_Kawasaki_28

.                              .
[==============================]all hills, n=836786
hill checkSum = 178344285691146
peaky hills, n=812304
peaky hill checkSum = 169101891663884
  nScans    nHills
         2        0
         3   165040
         4    99667
      5-10   265768
     10-20   188855
     20-50    86347
    50-100     5507
   100-200     1120
   200-500        0
  500-1000        0
 1000-2000        0
 2000-5000        0
5000-10000        0
    >10000        0
writing hill reports...
hill reports written
=== IN TARGETED MODE - BEWARE!!! ===
deisotoping complete
isotopes, n=29453

=== OUTPUT ===
  writing targeted isotope pattern file quandenser_output\percolator\search_and_link_59_to_72.psms.pout_dinosaur\FL603_MSQ1101_Honglei_Kawasaki_28.targets.csv

=== PROFILING ===
         mzml parse time: _ days __:__:21.247
           centroid time: _ days __:__:__.___
        hill report time: _ days __:__:33.350
          deisotope time: _ days __:__:_4.031
 iso pattern report time: _ days __:__:__.___
         mass calib time: _ days __:__:__.687
              write time: _ days __:__:__.485

         deiso edge time: _ days __:__:__.___
          deiso rip time: _ days __:__:__.___
       deiso deconv time: _ days __:__:__.___

total time: _ days __:_1:00.756
Features:
ppmDiff rtDiff precMz rTime queryIsPlaceHolder targetIsPlaceHolder charge1 charge2 charge3 charge4plus
Percolator version 3.02.1, Build Date May 27 2019 09:50:48
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms quandenser_output\percolator/search_and_link_59_to_72.psms.pout --decoy-results-psms quandenser_output\percolator/search_and_link_59_to_72.psms.pout.decoys
Started Wed Dec 18 19:19:17 2019
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 12955 positives and 1312 negatives, size ratio=9.87424 and pi0=1
Selecting Cpos by cross-validation.
Selecting Cneg by cross-validation.
Found 10771 test set positives with q<0.02 in initial direction
Reading in data and feature calculation took 0.052 cpu seconds or 0 seconds wall clock time.
---Training with Cpos selected by cross validation, Cneg selected by cross validation, initial_fdr=0.02, fdr=0.02
Iteration 1:    Estimated 12103 PSMs with q<0.02
Iteration 2:    Estimated 12092 PSMs with q<0.02
Iteration 3:    Estimated 12093 PSMs with q<0.02
Iteration 4:    Estimated 12093 PSMs with q<0.02
Iteration 5:    Estimated 12093 PSMs with q<0.02
Iteration 6:    Estimated 12093 PSMs with q<0.02
Iteration 7:    Estimated 12093 PSMs with q<0.02
Iteration 8:    Estimated 12093 PSMs with q<0.02
Iteration 9:    Estimated 12093 PSMs with q<0.02
Iteration 10:   Estimated 12093 PSMs with q<0.02
Learned normalized SVM weights for the 3 cross-validation splits:
 Split1  Split2  Split3 FeatureName
-2.4057 -3.0399 -3.2933 ppmDiff
 0.0151  0.0357  0.0614 rtDiff
-0.1766 -0.0463 -0.1890 precMz
-0.0511 -0.1114 -0.0613 rTime
 0.0000  0.0000  0.0000 queryIsPlaceHolder
 0.0000  0.0000  0.0000 targetIsPlaceHolder
 0.0891  0.0889  0.1302 charge1
-0.0154 -0.0244 -0.0618 charge2
-0.1260 -0.1273 -0.1379 charge3
-0.0253 -0.0100 -0.0169 charge4plus
 0.0303 -0.1723 -0.4573 m0
Found 12073 test set PSMs with q<0.02.
Selected best-scoring PSM per scan+expMass (target-decoy competition): 12879 target PSMs and 851 decoy PSMs.
Calculating q values.
Final list yields 12193 target PSMs with q<0.02.
Calculating posterior error probabilities (PEPs).
Processing took 6.869 cpu seconds or 7 seconds wall clock time.
Links before 22833
Links after 24635
Added 50968 placeholders.
Matching features 60->8
Features:
ppmDiff rtDiff precMz rTime queryIsPlaceHolder targetIsPlaceHolder charge1 charge2 charge3 charge4plus
Percolator version 3.02.1, Build Date May 27 2019 09:50:48
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms quandenser_output\percolator/link_60_to_8.psms --decoy-results-psms quandenser_output\percolator/link_60_to_8.psms.decoys
Started Wed Dec 18 19:19:25 2019
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 0 positives and 0 negatives, size ratio=-nan(ind) and pi0=1
Exception caught: Error: no target PSMs were provided.
MatthewThe commented 4 years ago

The problem seems to be the 60->8 alignment, which indeed has a rmse of nan. I suspect that the problem is that the nans are sorted as the lowest values, so we could probably still obtain a good tree if we simply ignore the nan values. I'll see if I can fix this.

Are you using the standalone quandenser now, or the quandenser-pipeline?

andrewjmc commented 4 years ago

Thanks. Currently standalone, on the best machine we have!

Andrew

On Thu, 19 Dec 2019, 14:30 MatthewThe, notifications@github.com wrote:

The problem seems to be the 60->8 alignment, which indeed has a rmse of nan. I suspect that the problem is that the nans are sorted as the lowest values, so we could probably still obtain a good tree if we simply ignore the nan values. I'll see if I can fix this.

Are you using the standalone quandenser now, or the quandenser-pipeline?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/statisticalbiotechnology/quandenser/issues/12?email_source=notifications&email_token=ACYVC7IQEDWQXFLA62SH5OLQZOAQTA5CNFSM4J5GWLSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHJY2YA#issuecomment-567512416, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYVC7KORIKP2XM4Y4JFBVLQZOAQTANCNFSM4J5GWLSA .

MatthewThe commented 4 years ago

I created a fix. There is still a possibility that there are certain runs with too little overlap (i.e. too few MS2 spectra that cluster with MS2 spectra from other runs), but at least I added an error message that should tell you which file that is in that case.

Should I create a binary installer for you to test it out? If so, please tell me which platform you are currently using.

andrewjmc commented 4 years ago

Many thanks. I'm running it on windows at the moment.

Best wishes,

Andrew

On Thu, 19 Dec 2019, 15:42 MatthewThe, notifications@github.com wrote:

I created a fix. There is still a possibility that there are certain runs with too little overlap (i.e. too few MS2 spectra that cluster with MS2 spectra from other runs), but at least I added an error message that should tell you which file that is in that case.

Should I create a binary installer for you to test it out? If so, please tell me which platform you are currently using.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/statisticalbiotechnology/quandenser/issues/12?email_source=notifications&email_token=ACYVC7L5GTKI65YOU6H3F5TQZOI6ZA5CNFSM4J5GWLSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHKAHOY#issuecomment-567542715, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYVC7KXPHCRBPCKNC4RONDQZOI6ZANCNFSM4J5GWLSA .

andrewjmc commented 4 years ago

Hi @MatthewThe

I'm back at full speed now. Did you have a chance to make a Windows Installer for the updated version?

If not, I'll see if I can compile here.

Best wishes,

Andrew

MatthewThe commented 4 years ago

Hi Andrew,

Sorry, it completely slipped my mind, I'll generate one now! If everything goes well I should be able to send you the binary tomorrow.

andrewjmc commented 4 years ago

Great, thanks. Tried to compile on a Windows machine, but had errors in the Boost and ProteoWizard compilation steps.

MatthewThe commented 4 years ago

quandenser-v0-02-win64-msvc-x64.zip

Here you go, let me know if it works!

Do you happen to have some log files for the failed compilation of Boost/ProteoWizard, it would be good to try and find out why it fails.

andrewjmc commented 4 years ago

Thanks. Running the new version now. This is the output from the compilation (quickbuild64.bat):

The system cannot find the path specified.
Downloading and installing Maven
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (404) Not Found."
At line:1 char:68
+ ... ls11, tls'; (new-object System.Net.WebClient).DownloadFile('http://ft ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException

cmake quandenser.....
CMake Error: Could not create named generator Visual Studio  Win64

Generators
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 12 2013 [arch] = Generates Visual Studio 2013 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 11 2012 [arch] = Generates Visual Studio 2012 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 10 2010 [arch] = Generates Visual Studio 2010 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Visual Studio 9 2008 [arch]  = Generates Visual Studio 2008 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Borland Makefiles            = Generates Borland makefiles.
  NMake Makefiles              = Generates NMake makefiles.
  NMake Makefiles JOM          = Generates JOM makefiles.
  Green Hills MULTI            = Generates Green Hills MULTI files
                                 (experimental, work-in-progress).
  MSYS Makefiles               = Generates MSYS makefiles.
  MinGW Makefiles              = Generates a make file for use with
                                 mingw32-make.
  Unix Makefiles               = Generates standard UNIX makefiles.
  Ninja                        = Generates build.ninja files.
  Watcom WMake                 = Generates Watcom WMake makefiles.
  CodeBlocks - MinGW Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles JOM
                               = Generates CodeBlocks project files.
  CodeBlocks - Ninja           = Generates CodeBlocks project files.
  CodeBlocks - Unix Makefiles  = Generates CodeBlocks project files.
  CodeLite - MinGW Makefiles   = Generates CodeLite project files.
  CodeLite - NMake Makefiles   = Generates CodeLite project files.
  CodeLite - Ninja             = Generates CodeLite project files.
  CodeLite - Unix Makefiles    = Generates CodeLite project files.
  Sublime Text 2 - MinGW Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - NMake Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - Ninja       = Generates Sublime Text 2 project files.
  Sublime Text 2 - Unix Makefiles
                               = Generates Sublime Text 2 project files.
  Kate - MinGW Makefiles       = Generates Kate project files.
  Kate - NMake Makefiles       = Generates Kate project files.
  Kate - Ninja                 = Generates Kate project files.
  Kate - Unix Makefiles        = Generates Kate project files.
  Eclipse CDT4 - NMake Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - MinGW Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Ninja         = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Unix Makefiles= Generates Eclipse CDT 4.0 project files.

build quandenser (this will take a few minutes).....
'msbuild' is not recognized as an internal or external command,
operable program or batch file.
cmake quandenser with vendor support.....
CMake Error: Could not create named generator Visual Studio  Win64

Generators
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 12 2013 [arch] = Generates Visual Studio 2013 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 11 2012 [arch] = Generates Visual Studio 2012 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 10 2010 [arch] = Generates Visual Studio 2010 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Visual Studio 9 2008 [arch]  = Generates Visual Studio 2008 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Borland Makefiles            = Generates Borland makefiles.
  NMake Makefiles              = Generates NMake makefiles.
  NMake Makefiles JOM          = Generates JOM makefiles.
  Green Hills MULTI            = Generates Green Hills MULTI files
                                 (experimental, work-in-progress).
  MSYS Makefiles               = Generates MSYS makefiles.
  MinGW Makefiles              = Generates a make file for use with
                                 mingw32-make.
  Unix Makefiles               = Generates standard UNIX makefiles.
  Ninja                        = Generates build.ninja files.
  Watcom WMake                 = Generates Watcom WMake makefiles.
  CodeBlocks - MinGW Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles JOM
                               = Generates CodeBlocks project files.
  CodeBlocks - Ninja           = Generates CodeBlocks project files.
  CodeBlocks - Unix Makefiles  = Generates CodeBlocks project files.
  CodeLite - MinGW Makefiles   = Generates CodeLite project files.
  CodeLite - NMake Makefiles   = Generates CodeLite project files.
  CodeLite - Ninja             = Generates CodeLite project files.
  CodeLite - Unix Makefiles    = Generates CodeLite project files.
  Sublime Text 2 - MinGW Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - NMake Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - Ninja       = Generates Sublime Text 2 project files.
  Sublime Text 2 - Unix Makefiles
                               = Generates Sublime Text 2 project files.
  Kate - MinGW Makefiles       = Generates Kate project files.
  Kate - NMake Makefiles       = Generates Kate project files.
  Kate - Ninja                 = Generates Kate project files.
  Kate - Unix Makefiles        = Generates Kate project files.
  Eclipse CDT4 - NMake Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - MinGW Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Ninja         = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Unix Makefiles= Generates Eclipse CDT 4.0 project files.

build quandenser with vendor support (this will take a few minutes).....
'msbuild' is not recognized as an internal or external command,
operable program or batch file.
Copying installers to C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\release\win64
C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\build\win64\quandenser\quan*.exe
The system cannot find the file specified.
        0 file(s) copied.
C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\build\win64\quandenser-vendor-support\quan*.exe
The system cannot find the file specified.
        0 file(s) copied.
Finished buildscript execution in build directory C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\build\win64

C:\Users\levinpeaks\Desktop\quandenser-master>quickbuild64.bat

C:\Users\levinpeaks\Desktop\quandenser-master>call ".\admin\builders\nativew64_build.bat" -s C:\Users\levinpeaks\Desktop\quandenser-master\\..
The system cannot find the path specified.
Downloading and installing 7-Zip
Downloading and installing CMake
Downloading and installing ProteoWizard
Starting Proteowizard and Boost build, this can take a while...
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the file specified.
'dumpbin' is not recognized as an internal or external command,
operable program or batch file.
'lib' is not recognized as an internal or external command,
operable program or batch file.
Downloading and installing Maven
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (404) Not Found."
At line:1 char:68
+ ... ls11, tls'; (new-object System.Net.WebClient).DownloadFile('http://ft ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException

Downloading and installing Java JDK
Downloading and installing NSIS installer
cmake quandenser.....
CMake Error: Could not create named generator Visual Studio  Win64

Generators
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 12 2013 [arch] = Generates Visual Studio 2013 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 11 2012 [arch] = Generates Visual Studio 2012 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 10 2010 [arch] = Generates Visual Studio 2010 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Visual Studio 9 2008 [arch]  = Generates Visual Studio 2008 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Borland Makefiles            = Generates Borland makefiles.
  NMake Makefiles              = Generates NMake makefiles.
  NMake Makefiles JOM          = Generates JOM makefiles.
  Green Hills MULTI            = Generates Green Hills MULTI files
                                 (experimental, work-in-progress).
  MSYS Makefiles               = Generates MSYS makefiles.
  MinGW Makefiles              = Generates a make file for use with
                                 mingw32-make.
  Unix Makefiles               = Generates standard UNIX makefiles.
  Ninja                        = Generates build.ninja files.
  Watcom WMake                 = Generates Watcom WMake makefiles.
  CodeBlocks - MinGW Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles JOM
                               = Generates CodeBlocks project files.
  CodeBlocks - Ninja           = Generates CodeBlocks project files.
  CodeBlocks - Unix Makefiles  = Generates CodeBlocks project files.
  CodeLite - MinGW Makefiles   = Generates CodeLite project files.
  CodeLite - NMake Makefiles   = Generates CodeLite project files.
  CodeLite - Ninja             = Generates CodeLite project files.
  CodeLite - Unix Makefiles    = Generates CodeLite project files.
  Sublime Text 2 - MinGW Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - NMake Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - Ninja       = Generates Sublime Text 2 project files.
  Sublime Text 2 - Unix Makefiles
                               = Generates Sublime Text 2 project files.
  Kate - MinGW Makefiles       = Generates Kate project files.
  Kate - NMake Makefiles       = Generates Kate project files.
  Kate - Ninja                 = Generates Kate project files.
  Kate - Unix Makefiles        = Generates Kate project files.
  Eclipse CDT4 - NMake Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - MinGW Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Ninja         = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Unix Makefiles= Generates Eclipse CDT 4.0 project files.

build quandenser (this will take a few minutes).....
'msbuild' is not recognized as an internal or external command,
operable program or batch file.
cmake quandenser with vendor support.....
CMake Error: Could not create named generator Visual Studio  Win64

Generators
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 12 2013 [arch] = Generates Visual Studio 2013 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 11 2012 [arch] = Generates Visual Studio 2012 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 10 2010 [arch] = Generates Visual Studio 2010 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Visual Studio 9 2008 [arch]  = Generates Visual Studio 2008 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Borland Makefiles            = Generates Borland makefiles.
  NMake Makefiles              = Generates NMake makefiles.
  NMake Makefiles JOM          = Generates JOM makefiles.
  Green Hills MULTI            = Generates Green Hills MULTI files
                                 (experimental, work-in-progress).
  MSYS Makefiles               = Generates MSYS makefiles.
  MinGW Makefiles              = Generates a make file for use with
                                 mingw32-make.
  Unix Makefiles               = Generates standard UNIX makefiles.
  Ninja                        = Generates build.ninja files.
  Watcom WMake                 = Generates Watcom WMake makefiles.
  CodeBlocks - MinGW Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles JOM
                               = Generates CodeBlocks project files.
  CodeBlocks - Ninja           = Generates CodeBlocks project files.
  CodeBlocks - Unix Makefiles  = Generates CodeBlocks project files.
  CodeLite - MinGW Makefiles   = Generates CodeLite project files.
  CodeLite - NMake Makefiles   = Generates CodeLite project files.
  CodeLite - Ninja             = Generates CodeLite project files.
  CodeLite - Unix Makefiles    = Generates CodeLite project files.
  Sublime Text 2 - MinGW Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - NMake Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - Ninja       = Generates Sublime Text 2 project files.
  Sublime Text 2 - Unix Makefiles
                               = Generates Sublime Text 2 project files.
  Kate - MinGW Makefiles       = Generates Kate project files.
  Kate - NMake Makefiles       = Generates Kate project files.
  Kate - Ninja                 = Generates Kate project files.
  Kate - Unix Makefiles        = Generates Kate project files.
  Eclipse CDT4 - NMake Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - MinGW Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Ninja         = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Unix Makefiles= Generates Eclipse CDT 4.0 project files.

build quandenser with vendor support (this will take a few minutes).....
'msbuild' is not recognized as an internal or external command,
operable program or batch file.
Copying installers to C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\release\win64
C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\build\win64\quandenser\quan*.exe
The system cannot find the file specified.
        0 file(s) copied.
C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\build\win64\quandenser-vendor-support\quan*.exe
The system cannot find the file specified.
        0 file(s) copied.
Finished buildscript execution in build directory C:\Users\levinpeaks\Desktop\quandenser-master\admin\builders\..\..\..\\build\win64

I presume it's something really obvious like needing Visual Studio?!

Thanks,

Andrew

MatthewThe commented 4 years ago

Yes, you indeed need Visual Studio to compile on windows :)

andrewjmc commented 4 years ago

Still getting an error:

Matching features 111->103 (106/290)
Links before 0
Links after 48
java -Xmx24000M -jar "C:\Program Files\quandenser-v0-02\share\java/Dinosaur-1.1.3.free.jar" --force  --profiling=true --nReport=0  --concurrency=11 --seed=1 --outDir=quandenser_output\percolator/search_and_link_111_to_103.psms.pout_dinosaur --advParams="C:\Program Files\quandenser-v0-02\share\java/advParams_dinosaur_targeted.txt" --mode=target --targets=quandenser_output\percolator/search_and_link_111_to_103.psms.pout.dinosaur_targets.tsv E:\FL603_MSQ1101_7.mzML
Dinosaur 1.1.3    built:${maven.build.timestamp}
  mzML file: E:\RAW\Oxford\FL603_MSQ1101_7.mzML
    out dir: quandenser_output\percolator/search_and_link_111_to_103.psms.pout_dinosaur
   out name: FL603_MSQ1101_7

java.lang.NumberFormatException: For input string: "-nan(ind)"
        at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
        at sun.misc.FloatingDecimal.parseDouble(Unknown Source)
        at java.lang.Double.parseDouble(Unknown Source)
        at scala.collection.immutable.StringLike$class.toDouble(StringLike.scala:232)
        at scala.collection.immutable.StringOps.toDouble(StringOps.scala:31)
        at se.lth.immun.TargetFile$$anonfun$parse$1.apply(TargetFile.scala:89)
        at se.lth.immun.TargetFile$$anonfun$parse$1.apply(TargetFile.scala:71)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at se.lth.immun.TargetFile.parse(TargetFile.scala:71)
        at se.lth.immun.TargetFile.read(TargetFile.scala:50)
        at se.lth.immun.TargetFile.read(TargetFile.scala:34)
        at se.lth.immun.Dinosaur$.main(Dinosaur.scala:55)
        at se.lth.immun.Dinosaur.main(Dinosaur.scala)
.                              .
[

There are now no NaNs in the minimum spanning tree:

Inserting link 54 to 72 with rmse 0.0125221
Inserting link 72 to 88 with rmse 0.0135621
Inserting link 56 to 72 with rmse 0.0144762
Inserting link 53 to 72 with rmse 0.0163098
Inserting link 72 to 90 with rmse 0.0167795
Inserting link 68 to 72 with rmse 0.0179118
Inserting link 72 to 101 with rmse 0.0201064
Inserting link 59 to 72 with rmse 0.0226068
Inserting link 72 to 96 with rmse 0.0233861
Inserting link 72 to 94 with rmse 0.02522
Inserting link 72 to 105 with rmse 0.0272367
Inserting link 72 to 91 with rmse 0.0276863
Inserting link 66 to 72 with rmse 0.0280463
Inserting link 72 to 79 with rmse 0.028457
Inserting link 72 to 93 with rmse 0.0287593
Inserting link 72 to 92 with rmse 0.0299834
Inserting link 72 to 74 with rmse 0.0329041
Inserting link 72 to 76 with rmse 0.0337567
Inserting link 72 to 82 with rmse 0.0377252
Inserting link 71 to 72 with rmse 0.0420174
Inserting link 71 to 77 with rmse 0.0453543
Inserting link 72 to 73 with rmse 0.0464842
Inserting link 73 to 123 with rmse 0.0351741
Inserting link 73 to 126 with rmse 0.0396354
Inserting link 70 to 79 with rmse 0.0469063
Inserting link 102 to 123 with rmse 0.054099
Inserting link 102 to 117 with rmse 0.0340731
Inserting link 77 to 89 with rmse 0.059959
Inserting link 77 to 87 with rmse 0.0606947
Inserting link 69 to 94 with rmse 0.0610856
Inserting link 102 to 119 with rmse 0.0625573
Inserting link 77 to 86 with rmse 0.0642264
Inserting link 55 to 123 with rmse 0.0661563
Inserting link 126 to 131 with rmse 0.0728114
Inserting link 67 to 76 with rmse 0.0743959
Inserting link 61 to 77 with rmse 0.0758508
Inserting link 125 to 126 with rmse 0.0767181
Inserting link 120 to 125 with rmse 0.0736985
Inserting link 116 to 120 with rmse 0.0696426
Inserting link 115 to 120 with rmse 0.0771026
Inserting link 122 to 125 with rmse 0.0788148
Inserting link 108 to 122 with rmse 0.0765662
Inserting link 108 to 110 with rmse 0.0755593
Inserting link 77 to 84 with rmse 0.0794927
Inserting link 116 to 121 with rmse 0.0817587
Inserting link 113 to 121 with rmse 0.0805299
Inserting link 112 to 113 with rmse 0.0739824
Inserting link 122 to 127 with rmse 0.0823003
Inserting link 109 to 120 with rmse 0.083077
Inserting link 112 to 118 with rmse 0.0862196
Inserting link 116 to 130 with rmse 0.0888808
Inserting link 130 to 135 with rmse 0.0791985
Inserting link 135 to 137 with rmse 0.0681626
Inserting link 106 to 118 with rmse 0.0891418
Inserting link 52 to 106 with rmse 0.0369443
Inserting link 102 to 114 with rmse 0.0910787
Inserting link 99 to 114 with rmse 0.0306238
Inserting link 122 to 124 with rmse 0.0926709
Inserting link 112 to 140 with rmse 0.0947413
Inserting link 139 to 140 with rmse 0.0930341
Inserting link 136 to 139 with rmse 0.0787865
Inserting link 136 to 138 with rmse 0.0935104
Inserting link 116 to 143 with rmse 0.0987949
Inserting link 143 to 144 with rmse 0.0771408
Inserting link 144 to 145 with rmse 0.0791569
Inserting link 142 to 144 with rmse 0.0870449
Inserting link 68 to 83 with rmse 0.100001
Inserting link 108 to 132 with rmse 0.100333
Inserting link 75 to 77 with rmse 0.103668
Inserting link 69 to 98 with rmse 0.103971
Inserting link 102 to 133 with rmse 0.104351
Inserting link 85 to 133 with rmse 0.047317
Inserting link 72 to 103 with rmse 0.105192
Inserting link 55 to 134 with rmse 0.10646
Inserting link 81 to 134 with rmse 0.0373476
Inserting link 110 to 129 with rmse 0.109543
Inserting link 52 to 63 with rmse 0.11113
Inserting link 94 to 97 with rmse 0.117369
Inserting link 62 to 69 with rmse 0.118162
Inserting link 65 to 77 with rmse 0.125864
Inserting link 77 to 78 with rmse 0.133116
Inserting link 82 to 141 with rmse 0.138012
Inserting link 128 to 136 with rmse 0.138457
Inserting link 78 to 80 with rmse 0.149409
Inserting link 64 to 117 with rmse 0.149751
Inserting link 77 to 95 with rmse 0.157326
Inserting link 99 to 107 with rmse 0.167112
Inserting link 77 to 104 with rmse 0.172099
Inserting link 94 to 100 with rmse 0.185955
Inserting link 57 to 123 with rmse 0.200649
Inserting link 57 to 58 with rmse 0.200056
Inserting link 103 to 111 with rmse 0.278724
Inserting link 54 to 60 with rmse 0.280711
Inserting link 20 to 73 with rmse 1.15674
Inserting link 20 to 29 with rmse 0.965001
Inserting link 28 to 29 with rmse 0.92142
Inserting link 28 to 30 with rmse 0.941314
Inserting link 5 to 29 with rmse 0.983814
Inserting link 5 to 9 with rmse 0.605266
Inserting link 5 to 23 with rmse 0.931392
Inserting link 5 to 35 with rmse 0.99353
Inserting link 3 to 35 with rmse 0.895922
Inserting link 35 to 50 with rmse 0.95337
Inserting link 25 to 30 with rmse 1.01299
Inserting link 16 to 25 with rmse 0.952276
Inserting link 50 to 51 with rmse 1.01658
Inserting link 49 to 51 with rmse 0.889802
Inserting link 48 to 51 with rmse 1.0241
Inserting link 26 to 30 with rmse 1.02472
Inserting link 26 to 27 with rmse 1.02314
Inserting link 47 to 50 with rmse 1.0396
Inserting link 17 to 26 with rmse 1.09697
Inserting link 17 to 19 with rmse 1.06998
Inserting link 19 to 24 with rmse 0.783992
Inserting link 14 to 24 with rmse 0.956417
Inserting link 15 to 19 with rmse 1.0254
Inserting link 3 to 38 with rmse 1.09978
Inserting link 21 to 29 with rmse 1.10925
Inserting link 21 to 32 with rmse 1.08957
Inserting link 31 to 32 with rmse 0.901579
Inserting link 29 to 41 with rmse 1.11901
Inserting link 4 to 17 with rmse 1.16368
Inserting link 4 to 6 with rmse 1.14845
Inserting link 11 to 30 with rmse 1.22141
Inserting link 18 to 114 with rmse 1.23279
Inserting link 28 to 46 with rmse 1.23693
Inserting link 37 to 142 with rmse 1.24845
Inserting link 6 to 8 with rmse 1.25496
Inserting link 7 to 8 with rmse 1.15142
Inserting link 7 to 12 with rmse 0.958579
Inserting link 12 to 13 with rmse 1.24026
Inserting link 22 to 114 with rmse 1.27794
Inserting link 2 to 9 with rmse 1.29819
Inserting link 14 to 42 with rmse 1.30242
Inserting link 0 to 54 with rmse 1.33496
Inserting link 36 to 37 with rmse 1.34994
Inserting link 31 to 45 with rmse 1.35885
Inserting link 17 to 39 with rmse 1.36557
Inserting link 43 to 85 with rmse 1.41355
Inserting link 1 to 54 with rmse 1.46123
Inserting link 26 to 40 with rmse 1.46204
Inserting link 10 to 129 with rmse 1.7054
Inserting link 9 to 44 with rmse 1.72417
Inserting link 3 to 33 with rmse 1.75469
Inserting link 6 to 34 with rmse 1.96057
Inserting feature alignment 0->54 to the queue for round 9.
Inserting feature alignment 1->54 to the queue for round 9.
Inserting feature alignment 2->9 to the queue for round 7.
Inserting feature alignment 3->35 to the queue for round 7.
Inserting feature alignment 3->33 to the queue for round 17.
Inserting feature alignment 3->38 to the queue for round 17.
Inserting feature alignment 4->17 to the queue for round 5.
Inserting feature alignment 4->6 to the queue for round 19.
Inserting feature alignment 5->29 to the queue for round 9.
Inserting feature alignment 5->9 to the queue for round 15.
Inserting feature alignment 5->23 to the queue for round 15.
Inserting feature alignment 5->35 to the queue for round 15.
Inserting feature alignment 6->4 to the queue for round 4.
Inserting feature alignment 6->8 to the queue for round 20.
Inserting feature alignment 6->34 to the queue for round 20.
Inserting feature alignment 7->8 to the queue for round 2.
Inserting feature alignment 7->12 to the queue for round 22.
Inserting feature alignment 8->6 to the queue for round 3.
Inserting feature alignment 8->7 to the queue for round 21.
Inserting feature alignment 9->5 to the queue for round 8.
Inserting feature alignment 9->2 to the queue for round 16.
Inserting feature alignment 9->44 to the queue for round 16.
Inserting feature alignment 10->129 to the queue for round 5.
Inserting feature alignment 11->30 to the queue for round 7.
Inserting feature alignment 12->7 to the queue for round 1.
Inserting feature alignment 12->13 to the queue for round 23.
Inserting feature alignment 13->12 to the queue for round 0.
Inserting feature alignment 14->24 to the queue for round 3.
Inserting feature alignment 14->42 to the queue for round 21.
Inserting feature alignment 15->19 to the queue for round 4.
Inserting feature alignment 16->25 to the queue for round 6.
Inserting feature alignment 17->26 to the queue for round 6.
Inserting feature alignment 17->4 to the queue for round 18.
Inserting feature alignment 17->19 to the queue for round 18.
Inserting feature alignment 17->39 to the queue for round 18.
Inserting feature alignment 18->114 to the queue for round 8.
Inserting feature alignment 19->17 to the queue for round 5.
Inserting feature alignment 19->15 to the queue for round 19.
Inserting feature alignment 19->24 to the queue for round 19.
Inserting feature alignment 20->73 to the queue for round 11.
Inserting feature alignment 20->29 to the queue for round 13.
Inserting feature alignment 21->29 to the queue for round 9.
Inserting feature alignment 21->32 to the queue for round 15.
Inserting feature alignment 22->114 to the queue for round 8.
Inserting feature alignment 23->5 to the queue for round 8.
Inserting feature alignment 24->19 to the queue for round 4.
Inserting feature alignment 24->14 to the queue for round 20.
Inserting feature alignment 25->30 to the queue for round 7.
Inserting feature alignment 25->16 to the queue for round 17.
Inserting feature alignment 26->30 to the queue for round 7.
Inserting feature alignment 26->17 to the queue for round 17.
Inserting feature alignment 26->27 to the queue for round 17.
Inserting feature alignment 26->40 to the queue for round 17.
Inserting feature alignment 27->26 to the queue for round 6.
Inserting feature alignment 28->29 to the queue for round 9.
Inserting feature alignment 28->30 to the queue for round 15.
Inserting feature alignment 28->46 to the queue for round 15.
Inserting feature alignment 29->20 to the queue for round 10.
Inserting feature alignment 29->5 to the queue for round 14.
Inserting feature alignment 29->21 to the queue for round 14.
Inserting feature alignment 29->28 to the queue for round 14.
Inserting feature alignment 29->41 to the queue for round 14.
Inserting feature alignment 30->28 to the queue for round 8.
Inserting feature alignment 30->11 to the queue for round 16.
Inserting feature alignment 30->25 to the queue for round 16.
Inserting feature alignment 30->26 to the queue for round 16.
Inserting feature alignment 31->32 to the queue for round 7.
Inserting feature alignment 31->45 to the queue for round 17.
Inserting feature alignment 32->21 to the queue for round 8.
Inserting feature alignment 32->31 to the queue for round 16.
Inserting feature alignment 33->3 to the queue for round 6.
Inserting feature alignment 34->6 to the queue for round 3.
Inserting feature alignment 35->5 to the queue for round 8.
Inserting feature alignment 35->3 to the queue for round 16.
Inserting feature alignment 35->50 to the queue for round 16.
Inserting feature alignment 36->37 to the queue for round 3.
Inserting feature alignment 37->142 to the queue for round 4.
Inserting feature alignment 37->36 to the queue for round 20.
Inserting feature alignment 38->3 to the queue for round 6.
Inserting feature alignment 39->17 to the queue for round 5.
Inserting feature alignment 40->26 to the queue for round 6.
Inserting feature alignment 41->29 to the queue for round 9.
Inserting feature alignment 42->14 to the queue for round 2.
Inserting feature alignment 43->85 to the queue for round 7.
Inserting feature alignment 44->9 to the queue for round 7.
Inserting feature alignment 45->31 to the queue for round 6.
Inserting feature alignment 46->28 to the queue for round 8.
Inserting feature alignment 47->50 to the queue for round 6.
Inserting feature alignment 48->51 to the queue for round 5.
Inserting feature alignment 49->51 to the queue for round 5.
Inserting feature alignment 50->35 to the queue for round 7.
Inserting feature alignment 50->47 to the queue for round 17.
Inserting feature alignment 50->51 to the queue for round 17.
Inserting feature alignment 51->50 to the queue for round 6.
Inserting feature alignment 51->48 to the queue for round 18.
Inserting feature alignment 51->49 to the queue for round 18.
Inserting feature alignment 52->106 to the queue for round 2.
Inserting feature alignment 52->63 to the queue for round 22.
Inserting feature alignment 53->72 to the queue for round 10.
Inserting feature alignment 54->72 to the queue for round 10.
Inserting feature alignment 54->0 to the queue for round 14.
Inserting feature alignment 54->1 to the queue for round 14.
Inserting feature alignment 54->60 to the queue for round 14.
Inserting feature alignment 55->123 to the queue for round 10.
Inserting feature alignment 55->134 to the queue for round 14.
Inserting feature alignment 56->72 to the queue for round 10.
Inserting feature alignment 57->123 to the queue for round 10.
Inserting feature alignment 57->58 to the queue for round 14.
Inserting feature alignment 58->57 to the queue for round 9.
Inserting feature alignment 59->72 to the queue for round 10.
Inserting feature alignment 60->54 to the queue for round 9.
Inserting feature alignment 61->77 to the queue for round 8.
Inserting feature alignment 62->69 to the queue for round 8.
Inserting feature alignment 63->52 to the queue for round 1.
Inserting feature alignment 64->117 to the queue for round 8.
Inserting feature alignment 65->77 to the queue for round 8.
Inserting feature alignment 66->72 to the queue for round 10.
Inserting feature alignment 67->76 to the queue for round 9.
Inserting feature alignment 68->72 to the queue for round 10.
Inserting feature alignment 68->83 to the queue for round 14.
Inserting feature alignment 69->94 to the queue for round 9.
Inserting feature alignment 69->62 to the queue for round 15.
Inserting feature alignment 69->98 to the queue for round 15.
Inserting feature alignment 70->79 to the queue for round 9.
Inserting feature alignment 71->72 to the queue for round 10.
Inserting feature alignment 71->77 to the queue for round 14.
Inserting feature alignment 72->73 to the queue for round 11.
Inserting feature alignment 72->53 to the queue for round 13.
Inserting feature alignment 72->54 to the queue for round 13.
Inserting feature alignment 72->56 to the queue for round 13.
Inserting feature alignment 72->59 to the queue for round 13.
Inserting feature alignment 72->66 to the queue for round 13.
Inserting feature alignment 72->68 to the queue for round 13.
Inserting feature alignment 72->71 to the queue for round 13.
Inserting feature alignment 72->74 to the queue for round 13.
Inserting feature alignment 72->76 to the queue for round 13.
Inserting feature alignment 72->79 to the queue for round 13.
Inserting feature alignment 72->82 to the queue for round 13.
Inserting feature alignment 72->88 to the queue for round 13.
Inserting feature alignment 72->90 to the queue for round 13.
Inserting feature alignment 72->91 to the queue for round 13.
Inserting feature alignment 72->92 to the queue for round 13.
Inserting feature alignment 72->93 to the queue for round 13.
Inserting feature alignment 72->94 to the queue for round 13.
Inserting feature alignment 72->96 to the queue for round 13.
Inserting feature alignment 72->101 to the queue for round 13.
Inserting feature alignment 72->103 to the queue for round 13.
Inserting feature alignment 72->105 to the queue for round 13.
Inserting feature alignment 73->20 to the queue for round 12.
Inserting feature alignment 73->72 to the queue for round 12.
Inserting feature alignment 73->123 to the queue for round 12.
Inserting feature alignment 73->126 to the queue for round 12.
Inserting feature alignment 74->72 to the queue for round 10.
Inserting feature alignment 75->77 to the queue for round 8.
Inserting feature alignment 76->72 to the queue for round 10.
Inserting feature alignment 76->67 to the queue for round 14.
Inserting feature alignment 77->71 to the queue for round 9.
Inserting feature alignment 77->61 to the queue for round 15.
Inserting feature alignment 77->65 to the queue for round 15.
Inserting feature alignment 77->75 to the queue for round 15.
Inserting feature alignment 77->78 to the queue for round 15.
Inserting feature alignment 77->84 to the queue for round 15.
Inserting feature alignment 77->86 to the queue for round 15.
Inserting feature alignment 77->87 to the queue for round 15.
Inserting feature alignment 77->89 to the queue for round 15.
Inserting feature alignment 77->95 to the queue for round 15.
Inserting feature alignment 77->104 to the queue for round 15.
Inserting feature alignment 78->77 to the queue for round 8.
Inserting feature alignment 78->80 to the queue for round 16.
Inserting feature alignment 79->72 to the queue for round 10.
Inserting feature alignment 79->70 to the queue for round 14.
Inserting feature alignment 80->78 to the queue for round 7.
Inserting feature alignment 81->134 to the queue for round 8.
Inserting feature alignment 82->72 to the queue for round 10.
Inserting feature alignment 82->141 to the queue for round 14.
Inserting feature alignment 83->68 to the queue for round 9.
Inserting feature alignment 84->77 to the queue for round 8.
Inserting feature alignment 85->133 to the queue for round 8.
Inserting feature alignment 85->43 to the queue for round 16.
Inserting feature alignment 86->77 to the queue for round 8.
Inserting feature alignment 87->77 to the queue for round 8.
Inserting feature alignment 88->72 to the queue for round 10.
Inserting feature alignment 89->77 to the queue for round 8.
Inserting feature alignment 90->72 to the queue for round 10.
Inserting feature alignment 91->72 to the queue for round 10.
Inserting feature alignment 92->72 to the queue for round 10.
Inserting feature alignment 93->72 to the queue for round 10.
Inserting feature alignment 94->72 to the queue for round 10.
Inserting feature alignment 94->69 to the queue for round 14.
Inserting feature alignment 94->97 to the queue for round 14.
Inserting feature alignment 94->100 to the queue for round 14.
Inserting feature alignment 95->77 to the queue for round 8.
Inserting feature alignment 96->72 to the queue for round 10.
Inserting feature alignment 97->94 to the queue for round 9.
Inserting feature alignment 98->69 to the queue for round 8.
Inserting feature alignment 99->114 to the queue for round 8.
Inserting feature alignment 99->107 to the queue for round 16.
Inserting feature alignment 100->94 to the queue for round 9.
Inserting feature alignment 101->72 to the queue for round 10.
Inserting feature alignment 102->123 to the queue for round 10.
Inserting feature alignment 102->114 to the queue for round 14.
Inserting feature alignment 102->117 to the queue for round 14.
Inserting feature alignment 102->119 to the queue for round 14.
Inserting feature alignment 102->133 to the queue for round 14.
Inserting feature alignment 103->72 to the queue for round 10.
Inserting feature alignment 103->111 to the queue for round 14.
Inserting feature alignment 104->77 to the queue for round 8.
Inserting feature alignment 105->72 to the queue for round 10.
Inserting feature alignment 106->118 to the queue for round 3.
Inserting feature alignment 106->52 to the queue for round 21.
Inserting feature alignment 107->99 to the queue for round 7.
Inserting feature alignment 108->122 to the queue for round 8.
Inserting feature alignment 108->110 to the queue for round 16.
Inserting feature alignment 108->132 to the queue for round 16.
Inserting feature alignment 109->120 to the queue for round 8.
Inserting feature alignment 110->108 to the queue for round 7.
Inserting feature alignment 110->129 to the queue for round 17.
Inserting feature alignment 111->103 to the queue for round 9.
Inserting feature alignment 112->113 to the queue for round 5.
Inserting feature alignment 112->118 to the queue for round 19.
Inserting feature alignment 112->140 to the queue for round 19.
Inserting feature alignment 113->121 to the queue for round 6.
Inserting feature alignment 113->112 to the queue for round 18.
Inserting feature alignment 114->102 to the queue for round 9.
Inserting feature alignment 114->18 to the queue for round 15.
Inserting feature alignment 114->22 to the queue for round 15.
Inserting feature alignment 114->99 to the queue for round 15.
Inserting feature alignment 115->120 to the queue for round 8.
Inserting feature alignment 116->120 to the queue for round 8.
Inserting feature alignment 116->121 to the queue for round 16.
Inserting feature alignment 116->130 to the queue for round 16.
Inserting feature alignment 116->143 to the queue for round 16.
Inserting feature alignment 117->102 to the queue for round 9.
Inserting feature alignment 117->64 to the queue for round 15.
Inserting feature alignment 118->112 to the queue for round 4.
Inserting feature alignment 118->106 to the queue for round 20.
Inserting feature alignment 119->102 to the queue for round 9.
Inserting feature alignment 120->125 to the queue for round 9.
Inserting feature alignment 120->109 to the queue for round 15.
Inserting feature alignment 120->115 to the queue for round 15.
Inserting feature alignment 120->116 to the queue for round 15.
Inserting feature alignment 121->116 to the queue for round 7.
Inserting feature alignment 121->113 to the queue for round 17.
Inserting feature alignment 122->125 to the queue for round 9.
Inserting feature alignment 122->108 to the queue for round 15.
Inserting feature alignment 122->124 to the queue for round 15.
Inserting feature alignment 122->127 to the queue for round 15.
Inserting feature alignment 123->73 to the queue for round 11.
Inserting feature alignment 123->55 to the queue for round 13.
Inserting feature alignment 123->57 to the queue for round 13.
Inserting feature alignment 123->102 to the queue for round 13.
Inserting feature alignment 124->122 to the queue for round 8.
Inserting feature alignment 125->126 to the queue for round 10.
Inserting feature alignment 125->120 to the queue for round 14.
Inserting feature alignment 125->122 to the queue for round 14.
Inserting feature alignment 126->73 to the queue for round 11.
Inserting feature alignment 126->125 to the queue for round 13.
Inserting feature alignment 126->131 to the queue for round 13.
Inserting feature alignment 127->122 to the queue for round 8.
Inserting feature alignment 128->136 to the queue for round 1.
Inserting feature alignment 129->110 to the queue for round 6.
Inserting feature alignment 129->10 to the queue for round 18.
Inserting feature alignment 130->116 to the queue for round 7.
Inserting feature alignment 130->135 to the queue for round 17.
Inserting feature alignment 131->126 to the queue for round 10.
Inserting feature alignment 132->108 to the queue for round 7.
Inserting feature alignment 133->102 to the queue for round 9.
Inserting feature alignment 133->85 to the queue for round 15.
Inserting feature alignment 134->55 to the queue for round 9.
Inserting feature alignment 134->81 to the queue for round 15.
Inserting feature alignment 135->130 to the queue for round 6.
Inserting feature alignment 135->137 to the queue for round 18.
Inserting feature alignment 136->139 to the queue for round 2.
Inserting feature alignment 136->128 to the queue for round 22.
Inserting feature alignment 136->138 to the queue for round 22.
Inserting feature alignment 137->135 to the queue for round 5.
Inserting feature alignment 138->136 to the queue for round 1.
Inserting feature alignment 139->140 to the queue for round 3.
Inserting feature alignment 139->136 to the queue for round 21.
Inserting feature alignment 140->112 to the queue for round 4.
Inserting feature alignment 140->139 to the queue for round 20.
Inserting feature alignment 141->82 to the queue for round 9.
Inserting feature alignment 142->144 to the queue for round 5.
Inserting feature alignment 142->37 to the queue for round 19.
Inserting feature alignment 143->116 to the queue for round 7.
Inserting feature alignment 143->144 to the queue for round 17.
Inserting feature alignment 144->143 to the queue for round 6.
Inserting feature alignment 144->142 to the queue for round 18.
Inserting feature alignment 144->145 to the queue for round 18.
Inserting feature alignment 145->144 to the queue for round 5.

And the specific link that fails:

Aligned runs: 103 111: rmseComb = 0.278724 rmse1 = 0.161232 rmse2 = 0.227358

It is presumably significant that there are only 48 links.

Based on the line number counts in the file list, 103 and 111 should correspond to two samples from the same lab and instrument, run in different batches. The main difference is that (for unclear reason) the MS was switched from FT/FT mode to FT/IT mode in the second batch.

If my numbering is off, and these two samples come from different instruments (see https://github.com/statisticalbiotechnology/quandenser/issues/5) could there be an issue that 50 minutes in one lab corresponds to 80-100 minutes in the other?

Thanks,

Andrew

MatthewThe commented 4 years ago

It's indeed very suspicious that there are only 48 links. Did Quandenser run a new percolator run on this alignment that you cut out from the log? If you reused an old results folder, Quandenser tries to reuse these results, including old alignments and percolator runs.

Do you still have access to this file: quandenser_output\percolator/search_and_link_111_to_103.psms.pout.dinosaur_targets.tsv? It would be good to know where it encounters the nan value.

There shouldn't be an issue with different gradient lengths, as the retention time alignment nodes are set by using retention times from MS2 spectra that are highly similar between the two runs.

andrewjmc commented 4 years ago

I reran as I accidentally closed and lost the log. The error appeared the same.

Here is the search_and_link file. Lots of NaNs!

search_and_link_111_to_103.psms.pout.dinosaur_targets.txt

Thanks,

Andrew

MatthewThe commented 4 years ago

Thanks! Based on all the NaNs something seems to be going wrong with the alignment, but in that case it's strange that it can compute a RMSE at all. Does the percolator output file for the first feature matching also contain lots of NaNs (quandenser_output\percolator/link_111_to_103.psms.pout)?

Perhaps you could run Quandenser on just these two files and see if you encounter the same error?

andrewjmc commented 4 years ago

No NaNs there

PSMId   score   q-value posterior_error_prob    peptide proteinIds
69083_1158.09607_79.5922394_9867386_5   0   0.0208333   6.30512e-16 A.71071_1158.09521_79.6734238_21687124_5.A
65265_1052.43396_79.5922394_5451348_5   0   0.0208333   6.30512e-16 A.68195_1052.43335_79.6003113_8555419_5.A
43684_753.471436_79.5922394_12393395_2  0   0.0208333   6.30512e-16 A.51727_753.471313_79.3168488_3107870.75_2.A
62714_990.797363_79.5922394_722188.438_5    -0.0415158  0.0208333   6.30512e-16 A.66004_990.796326_79.5830383_2084346.25_5.A
63178_1000.60437_79.5922394_4546595_2   -0.0519794  0.0208333   6.30512e-16 A.66349_1000.60352_79.4725037_6449816.5_2.A
62191_978.590942_79.5922394_4530321_2   -0.0701809  0.0208333   6.30512e-16 A.65603_978.589905_79.4687119_5697950_2.A
55834_877.959229_79.5922394_1321549.5_1 -0.073912   0.0208333   6.30512e-16 A.60750_877.958313_79.7451782_2143934.5_1.A
68117_1132.93701_79.5922394_176173.688_2    -0.0753579  0.0208333   6.30512e-16 A.70509_1132.9353_79.5922394_154608.109_2.A
54565_859.233948_79.5922394_1935866.5_1 -0.0842571  0.0208333   6.30512e-16 A.59914_859.23291_79.7226257_3340290.5_1.A
67642_1122.87622_79.5922394_9664615_5   -0.0854519  0.0208333   6.30512e-16 A.70214_1122.87451_79.6266251_19748648_5.A
57271_896.970825_79.5922394_1712437_1   -0.0870146  0.0208333   6.30512e-16 A.62048_896.969849_79.771225_2954083.25_1.A
67629_1122.67761_79.5922394_181001.406_2    -0.0919205  0.0208333   6.30512e-16 A.70202_1122.6781_79.9992142_251870.375_2.A
55835_878.102478_79.5922394_1909160.63_1    -0.0938794  0.0208333   6.30512e-16 A.60752_878.101379_79.7455902_3875439.25_1.A
68811_1149.29175_79.5922394_11231218_5  -0.125874   0.0208333   6.30512e-16 A.70868_1149.28979_79.6541367_21809064_5.A
56206_884.701843_79.5922394_3232924_6   -0.131739   0.0208333   6.30512e-16 A.61067_884.700195_79.6153259_5635440.5_6.A
68103_1132.68323_79.5922394_3743936.25_2    -0.131897   0.0208333   6.30512e-16 A.70506_1132.68103_79.513443_3095626.25_2.A
56179_884.391113_79.5922394_1817283.25_1    -0.134161   0.0208333   6.30512e-16 A.60974_884.390503_79.7536774_2750433.25_1.A
70064_1193.31836_79.5922394_10748685_5  -0.136301   0.0208333   6.30512e-16 A.71731_1193.31616_79.7194595_23114330_5.A
46185_775.485413_79.5922394_13580459_2  -0.137928   0.0208333   6.30512e-16 A.53549_775.484497_79.3367386_2740740.75_2.A
62554_987.103943_79.5922394_4283580_2   -0.148797   0.0208333   6.30512e-16 A.65874_987.102356_79.4484711_3083504.75_2.A
57255_896.827515_79.5922394_1182196.25_1    -0.149785   0.0208333   6.30512e-16 A.62038_896.825867_79.772377_1886706.75_1.A
54942_865.380615_79.5922394_1396423.75_1    -0.154068   0.0208333   6.30512e-16 A.60253_865.378906_79.744873_2335942_1.A
55772_877.19574_79.5922394_3157520_6    -0.157709   0.0208333   6.30512e-16 A.60689_877.193909_79.6029358_5227106.5_6.A
35874_687.433105_79.5922394_14412073_2  -0.172794   0.0208333   6.30512e-16 A.45225_687.432312_79.248436_3160381_2.A
70278_1202.12244_79.5922394_10076528_5  -0.222349   0.0208333   6.30512e-16 A.71883_1202.12134_79.7182465_21186520_5.A
38646_709.446228_79.5922394_13804845_2  -0.222652   0.0208333   6.30512e-16 A.47090_709.445068_79.2879105_3184310_2.A
67628_1122.67236_79.5922394_1444623.25_2    -0.253684   0.0208333   6.30512e-16 A.70201_1122.66882_79.4345932_1391342.25_2.A
72801_1316.79248_79.5922394_3836211_5   -0.311015   0.0208333   6.30512e-16 A.73405_1316.78931_79.8508301_10001912_5.A
49652_808.775085_79.5922394_888644.188_1    -0.314138   0.0208333   6.30512e-16 A.55589_808.774048_79.637825_1575919.88_1.A
73511_1340.04895_79.5922394_52482728_3  -0.351173   0.0208333   6.30512e-16 A.73642_1340.047_78.5182114_27413240_3.A
50300_815.064636_79.5922394_980652.125_1    -0.391056   0.0208333   6.30512e-16 A.56115_815.063538_79.6485748_1513772.63_1.A
71738_1270.10706_79.5922394_4998787.5_2 -0.437672   0.0208333   6.30512e-16 A.72857_1270.10583_78.5798645_10231779_2.A
59175_925.907593_79.5922394_3594029.75_2    -0.456716   0.0208333   6.30512e-16 A.63561_925.906006_78.7770844_2752469_2.A
50289_814.920959_79.5922394_534390.188_1    -0.556058   0.0208333   6.30512e-16 A.56108_814.919678_79.6442719_802431.063_1.A
75681_1492.71667_79.5922394_2962652.5_2 -0.577428   0.0208333   6.30512e-16 A.75148_1492.71387_78.553978_9237638_2.A
56175_884.248718_79.5922394_1052687.25_1    -0.612151   0.0208333   6.30512e-16 A.60971_884.247559_79.7508926_2125916.5_1.A
54558_859.090881_79.5922394_1400203.75_1    -0.61304    0.0208333   6.30512e-16 A.59912_859.089661_79.7162399_2126793.5_1.A
41180_731.459106_79.5922394_14211587_2  -0.64223    0.0208333   6.30512e-16 A.49267_731.458435_79.2953262_3462668.75_2.A
73430_1337.3042_79.5922394_4272242_4    -0.715045   0.0208333   6.30512e-16 A.73600_1337.30164_79.5993881_7851266_4.A
71781_1272.27905_79.5922394_26329606_3  -0.756543   0.0208333   6.30512e-16 A.72891_1272.27515_78.5241928_39882752_3.A
69754_1181.89514_79.5922394_6670183_3   -0.764955   0.0208333   6.30512e-16 A.71517_1181.89136_78.5469971_12181422_3.A
65299_1053.1438_79.5922394_3704413.25_2 -0.810912   0.0208333   6.30512e-16 A.68233_1053.14197_79.4887543_2754978.5_2.A
69678_1179.11182_79.5922394_4377825.5_5 -0.884162   0.0208333   6.30512e-16 A.71473_1179.10962_79.6858215_6849529.5_5.A
52883_840.2229_79.5922394_1234591.88_1  -1.05405    0.0208333   6.30512e-16 A.58306_840.22113_79.6777039_1934246.63_1.A
55859_878.532166_79.5922394_2660550_4   -1.24688    0.0208333   6.30512e-16 A.60761_878.530762_79.9195251_1524402.88_4.A
60254_944.012756_79.5922394_56645680_2  -1.86135    0.0208333   6.30512e-16 A.64272_944.012268_78.667244_66986920_2.A
62535_986.604248_79.5922394_323278.938_3    -2.24556    0.0208333   6.30512e-16 A.65865_986.600525_79.5515518_254646.766_3.A
65755_1068.02258_79.5922394_41686404_2  -3.54903    0.0208333   6.30512e-16 A.68670_1068.01953_78.7238998_1706508.13_2.A

I'll try the two alone.

MatthewThe commented 4 years ago

Thanks, the problem actually seems to be that all features have the same retention time (79.5922394). This is probably also the reason that the alignment fails as well. If we have results from Quandenser on just these two runs we can probably distill if the problem lies in the input or in the parsing/processing.

andrewjmc commented 4 years ago

Aha! Perhaps two files are identical. I will investigate...

andrewjmc commented 4 years ago

Going off zero indexing of the files, they correspond to different samples and batches and the files are different sizes.

MatthewThe commented 4 years ago

I don't think that the problem stems from identical files. Rather, all features from one of the runs (111) all have the same retention time. This could be because this is what the input file actually contains, or it could be a bug somewhere else. Do all the lines in quandenser_output\percolator/link_111_to_103.psms.pout contain 79.5922394 in the third field (separated by underscores) of the PSMid column? If so, it is either a problem with the input file, or with Dinosaur.

andrewjmc commented 4 years ago

Do you mean quandenser_output\percolator/link_111_to_103.psms? Yes, all contain the same.

MatthewThe commented 4 years ago

Okay, that narrows things down quite a bit! There should be a tsv file in quandenser_output/dinosaur/ with the same name as the file at index 111. Do they all contain 79.5922394 in the rtApex column (the 5th column)?

andrewjmc commented 4 years ago

File 111 (line 112 of the file list, I believe) contains RT apices from 0.004752 to 102.9914m. The paired file (103) also seems to have a wide range of RTs.

MatthewThe commented 4 years ago

That's strange, the parsing of these dinosaur output files should be very straightforward.. Could you check quandenser_output\percolator/link_<x>_to_<y>.psms for another alignments and see if it has the same issue with the same value for the retention time in each row for the PSMid?

If you haven't done so yet, please try to run Quandenser on just the two files (111 and 103, i.e. line 112 and 104 in your file list).

andrewjmc commented 4 years ago

I had to a add a third file (though the text says at least 2, it actually imposes a limit of 3 or more). Interestingly, seems to have worked fine, and is now outputting consensus spectra

andrewjmc commented 4 years ago

No issue noted with alignments. The pout files for linking the offending files now contain multiple RTs.

However did receive many thousands of index out of bounds errors during consensus spectrum writing (from multiple files):

Warning: index 3036693 out of bounds: E:\RAW\FL948_MSQ1388_20180605_144.mzML: 2 36693
MatthewThe commented 4 years ago

Thanks for the information and all your patience. I took another look at the code and it seems very likely that it is just the retention time alignment (111->103) going wrong.

Unfortunately, it's not so easy to debug this without the data. Would you be able to share the concerning mzML files (111 and 103) and the original MS2 clustering output (quandenser_output/maracluster/MaRaCluster.clusters_p10.tsv from the run with all files)?

andrewjmc commented 4 years ago

I replied by e-mail but it didn't seem to go through. Of course, thanks. I'll get you a link offline, what's the best address?

Andrew

MatthewThe commented 4 years ago

Great, you can send it to matthewt [at] kth [dot] se

MatthewThe commented 4 years ago

I could actually reproduce the error by running Quandenser on just the two files. The problem was indeed the alignment which failed due to several retention time knots having the exact same retention time value (this created some -inf values in the equations for solving the spline regression). I will create a fix for this and create a new windows binary.

MatthewThe commented 4 years ago

Here is the binary installer for windows: quandenser-v0-02-win64-msvc-x64.zip

Let me know if it works!

andrewjmc commented 4 years ago

That's great, thanks! Will get the rerun going later today.

andrewjmc commented 4 years ago

Reran (with the original Dinosaur and Maracluster files, and only deleting the offending 111>103 alignment files.

Hit a different error, similar to #13 but at a different point:

Matching features 32->31 (1/290)
Features:
ppmDiff rtDiff precMz rTime queryIsPlaceHolder targetIsPlaceHolder charge1 charge2 charge3 charge4plus
Percolator version 3.02.1, Build Date Jan 15 2020 09:13:06
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms .\quandenser_output\percolator/link_32_to_31.psms --decoy-results-psms .\quandenser_output\percolator/link_32_to_31.psms.decoys
Started Thu Jan 16 13:05:18 2020
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 172154 positives and 24219 negatives, size ratio=7.10822 and pi0=1
Selecting Cpos by cross-validation.
Selecting Cneg by cross-validation.
Found 139142 test set positives with q<0.02 in initial direction
Reading in data and feature calculation took 0.358 cpu seconds or 0 seconds wall clock time.
---Training with Cpos selected by cross validation, Cneg selected by cross validation, initial_fdr=0.02, fdr=0.02
Iteration 1:    Estimated 139933 PSMs with q<0.02
Iteration 2:    Estimated 140284 PSMs with q<0.02
Iteration 3:    Estimated 140453 PSMs with q<0.02
Iteration 4:    Estimated 140564 PSMs with q<0.02
Iteration 5:    Estimated 140602 PSMs with q<0.02
Iteration 6:    Estimated 140647 PSMs with q<0.02
Iteration 7:    Estimated 140682 PSMs with q<0.02
Iteration 8:    Estimated 140690 PSMs with q<0.02
Iteration 9:    Estimated 140693 PSMs with q<0.02
Iteration 10:   Estimated 140693 PSMs with q<0.02
Learned normalized SVM weights for the 3 cross-validation splits:
 Split1  Split2  Split3 FeatureName
-2.3880 -2.3652 -2.4107 ppmDiff
-0.5740 -0.5751 -0.5800 rtDiff
-0.0485 -0.0360 -0.0399 precMz
-0.0013  0.0032 -0.0114 rTime
 0.0000  0.0000  0.0000 queryIsPlaceHolder
 0.0000  0.0000  0.0000 targetIsPlaceHolder
 0.0253  0.0421  0.0402 charge1
-0.0226 -0.0343 -0.0261 charge2
-0.0292 -0.0392 -0.0384 charge3
 0.0458  0.0597  0.0496 charge4plus
-0.4913 -0.4620 -0.4890 m0
Found 140633 test set PSMs with q<0.02.
Selected best-scoring PSM per scan+expMass (target-decoy competition): 98061 target PSMs and 10859 decoy PSMs.
Calculating q values.
Final list yields 84524 target PSMs with q<0.02.
Calculating posterior error probabilities (PEPs).
Processing took 32.23 cpu seconds or 32 seconds wall clock time.
Links before 0
Links after 86273
java -Xmx48G -jar "C:\Program Files\quandenser-v0-02\share\java/Dinosaur-1.1.3.free.jar" --force  --profiling=true --nReport=0  --concurrency=11 --seed=1 --outDir=.\quandenser_output\percolator/search_and_link_32_to_31.psms.pout_dinosaur --advParams="C:\Program Files\quandenser-v0-02\share\java/advParams_dinosaur_targeted.txt" --mode=target --targets=.\quandenser_output\percolator/search_and_link_32_to_31.psms.pout.dinosaur_targets.tsv E:\RAW\Lab1\Febrile4793.mzML
Dinosaur 1.1.3    built:${maven.build.timestamp}
  mzML file: E:\RAW\Lab1\Febrile4793.mzML
    out dir: .\quandenser_output\percolator/search_and_link_32_to_31.psms.pout_dinosaur
   out name: Febrile4793

.                              .
[==============================]all hills, n=1278927
hill checkSum = 619739993971331
peaky hills, n=1278927
peaky hill checkSum = 619739993971331
  nScans    nHills
         2        0
         3   331506
         4   185629
      5-10   411630
     10-20   224404
     20-50    94229
    50-100    22254
   100-200     9275
   200-500        0
  500-1000        0
 1000-2000        0
 2000-5000        0
5000-10000        0
    >10000        0
writing hill reports...
hill reports written
=== IN TARGETED MODE ===
deisotoping complete
isotopes, n=322834

=== OUTPUT ===
  writing targeted isotope pattern file .\quandenser_output\percolator\search_and_link_32_to_31.psms.pout_dinosaur\Febrile4793.targets.csv

=== PROFILING ===
         mzml parse time: _ days __:_1:33.712
           centroid time: _ days __:__:__.___
        hill report time: _ days __:__:50.012
          deisotope time: _ days __:__:14.311
 iso pattern report time: _ days __:__:__.___
         mass calib time: _ days __:__:14.124
       target match time: _ days __:__:_4.281
              write time: _ days __:__:_1.391

         deiso edge time: _ days __:__:__.___
          deiso rip time: _ days __:__:__.___
       deiso deconv time: _ days __:__:__.___

total time: _ days __:_2:58.534
Features:
ppmDiff rtDiff precMz rTime queryIsPlaceHolder targetIsPlaceHolder charge1 charge2 charge3 charge4plus
Percolator version 3.02.1, Build Date Jan 15 2020 09:13:06
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms .\quandenser_output\percolator/search_and_link_32_to_31.psms.pout --decoy-results-psms .\quandenser_output\percolator/search_and_link_32_to_31.psms.pout.decoys
Started Thu Jan 16 13:08:55 2020
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 51772 positives and 31350 negatives, size ratio=1.65142 and pi0=1
Selecting Cpos by cross-validation.
Selecting Cneg by cross-validation.
Found 7827 test set positives with q<0.02 in initial direction
Reading in data and feature calculation took 0.153 cpu seconds or 0 seconds wall clock time.
---Training with Cpos selected by cross validation, Cneg selected by cross validation, initial_fdr=0.02, fdr=0.02
Iteration 1:    Estimated 8393 PSMs with q<0.02
Iteration 2:    Estimated 8574 PSMs with q<0.02
Iteration 3:    Estimated 8872 PSMs with q<0.02
Iteration 4:    Estimated 8892 PSMs with q<0.02
Iteration 5:    Estimated 8942 PSMs with q<0.02
Iteration 6:    Estimated 8958 PSMs with q<0.02
Iteration 7:    Estimated 9070 PSMs with q<0.02
Iteration 8:    Estimated 9017 PSMs with q<0.02
Iteration 9:    Estimated 9054 PSMs with q<0.02
Iteration 10:   Estimated 9049 PSMs with q<0.02
Learned normalized SVM weights for the 3 cross-validation splits:
 Split1  Split2  Split3 FeatureName
-7.0440 -3.4775 -4.9978 ppmDiff
-3.5696 -2.3786 -2.7348 rtDiff
-0.1170 -0.2574 -0.1404 precMz
-0.3405 -0.1617 -0.2032 rTime
 0.0000  0.0000  0.0000 queryIsPlaceHolder
 0.0000  0.0000  0.0000 targetIsPlaceHolder
 0.0471  0.0485  0.0185 charge1
-0.1110 -0.0090 -0.0629 charge2
 0.0257 -0.0416  0.0007 charge3
 0.0657  0.0119  0.0656 charge4plus
-10.0062        -5.3384 -7.7162 m0
Found 8327 test set PSMs with q<0.02.
Selected best-scoring PSM per scan+expMass (target-decoy competition): 43547 target PSMs and 11696 decoy PSMs.
Calculating q values.
Final list yields 9760 target PSMs with q<0.02.
Calculating posterior error probabilities (PEPs).
Processing took 6.3 cpu seconds or 7 seconds wall clock time.
Links before 86273
Links after 105920
Added 40375 placeholders.
Matching features 45->31 (2/290)
Links before 0
Links after 75594
Links before 75594
Exception caught: invalid vector<T> subscript
MatthewThe commented 4 years ago

Do I understand correctly that you reused the old results folder? This might not work because a different alignment tree might be formed because of the bug fix.

andrewjmc commented 4 years ago

Ah, OK. I will clear the percolator directory, but leave the others.

andrewjmc commented 4 years ago

Rerunning now, 111->103 no longer appears in list of alignments to be computed!

andrewjmc commented 4 years ago

However, new error at percolator step, this time about no decoys. I presume again this is due to the small numbers of matching features. Or is it because I need to rerun the original dinosaur and maracluster steps?

Thanks,

Andrew

java -Xmx48G -jar "C:\Program Files\quandenser-v0-02\share\java/Dinosaur-1.1.3.free.jar" --force  --profiling=true --nReport=0  --concurrency=11 --seed=1 --outDir=.\quandenser_output\percolator/search_and_link_16_to_122.psms.pout_dinosaur --advParams="C:\Program Files\quandenser-v0-02\share\java/advParams_dinosaur_targeted.txt" --mode=target --targets=.\quandenser_output\percolator/search_and_link_16_to_122.psms.pout.dinosaur_targets.tsv E:\RAW\Lab1\FL948_MSQ1388_20180605_SM_24.mzML
Dinosaur 1.1.3    built:${maven.build.timestamp}
  mzML file: E:\RAW\Oxford\FL948_MSQ1388_20180605_StephanieMenikou_24.mzML
    out dir: .\quandenser_output\percolator/search_and_link_16_to_122.psms.pout_dinosaur
   out name: FL948_MSQ1388_20180605_SM_24

.                              .
[==============================]all hills, n=544657
hill checkSum = 112754986694271
peaky hills, n=544657
peaky hill checkSum = 112754986694271
  nScans    nHills
         2        0
         3   119145
         4    68137
      5-10   160296
     10-20   125287
     20-50    62595
    50-100     7153
   100-200     2044
   200-500        0
  500-1000        0
 1000-2000        0
 2000-5000        0
5000-10000        0
    >10000        0
writing hill reports...
hill reports written
=== IN TARGETED MODE ===
deisotoping complete
isotopes, n=25

=== OUTPUT ===
  writing targeted isotope pattern file .\quandenser_output\percolator\search_and_link_16_to_122.psms.pout_dinosaur\FL948_MSQ1388_20180605_SM_24.targets.csv

=== PROFILING ===
         mzml parse time: _ days __:__:22.420
           centroid time: _ days __:__:__.___
        hill report time: _ days __:__:22.857
          deisotope time: _ days __:__:_2.140
 iso pattern report time: _ days __:__:__.___
         mass calib time: _ days __:__:__._16
       target match time: _ days __:__:__.140
              write time: _ days __:__:__.656

         deiso edge time: _ days __:__:__.___
          deiso rip time: _ days __:__:__.___
       deiso deconv time: _ days __:__:__.___

total time: _ days __:__:48.979
Features:
ppmDiff rtDiff precMz rTime queryIsPlaceHolder targetIsPlaceHolder charge1 charge2 charge3 charge4plus
Percolator version 3.02.1, Build Date Jan 15 2020 09:13:06
Copyright (c) 2006-9 University of Washington. All rights reserved.
Written by Lukas Käll (lukall@u.washington.edu) in the
Department of Genome Sciences at the University of Washington.
Issued command:
percolator --only-psms --post-processing-tdc input_file_placeholder --trainFDR 0.02 --testFDR 0.02 --results-psms .\quandenser_output\percolator/search_and_link_16_to_122.psms.pout --decoy-results-psms .\quandenser_output\percolator/search_and_link_16_to_122.psms.pout.decoys
Started Thu Jan 16 16:51:17 2020
Hyperparameters: selectionFdr=0.02, Cpos=0, Cneg=0, maxNiter=10
FeatureNames::getNumFeatures(): 10
Train/test set contains 14 positives and 0 negatives, size ratio=inf and pi0=1
Exception caught: Error: no decoy PSMs were provided.
MatthewThe commented 4 years ago

There should be no need to rerun the dinosaur and maracluster steps. The problem could indeed be a low number of matching features. How many matching features are found by percolator on the same alignment before the targeted dinosaur run? And what is the rmse of this alignment?

andrewjmc commented 4 years ago

The file search_and_link_16_to_122.psms.pout.dinosaur_targets.tsv has 129625 pairs. The search_and_link_16_to_122.psms.pout and search_and_link_16_to_122.psms.pout.decoys files are empty.

I had to rerun (now with yesterday evening's version) with a minimal set of files to get the retention time stats (still struggling to get piping of PowerShell output with Tee-Object.

Aligned runs: 0 1: rmseComb = 3.64676 rmse1 = 1.58766 rmse2 = 3.28301 numPairs = 237

This analysis is proceeding fine, having passed the same error step. However, the search_and_link...dinosaur_targets.tsv file is much smaller (only 50742 pairs). I'm not sure why there is such a difference.

MatthewThe commented 4 years ago

The difference in number of pairs in dinosaur_targets.tsv is probably due to features that were added due to alignments ahead in the alignment tree, so this is somewhat expected.

It's likely that the alignment is not very good, since only 237 pairs were used to generate the alignment. Could you either share link_16_to_122.psms, or at least look up how many matches have a q-value below 0.05 in that file?

andrewjmc commented 4 years ago

Sorry for delay. Smallest q value is 0.076, of the 62 from the original run. link_16_to_122.psms.txt

Looking at the same samples from the reduced rerun, there are 103,272 in the link_*.psms and 58,833 have q values below 0.05.

MatthewThe commented 4 years ago

Okay, that indeed seems to indicate that the alignment is going wrong. Did you try to run the entire set with the newest version already?

andrewjmc commented 4 years ago

I am losing track a little, but I think I had only used the new version for the reduced rerun. When the machine is available, I'll rerun all.