search

claczny / VizBin

Repository of our application for human-augmented binning
27 stars 14 forks source link

Some snapshots from doing testing before merging `revision01` to `master` #20

Open claczny opened 9 years ago

claczny commented 9 years ago

Below will be some snapshots/ideas that I found to be important to note down before the merge.

claczny commented 9 years ago

From Windows console output:

C:\Users\Jennifer>"C:\Program Files (x86)\Java\jre7\bin\java.exe" -jar -Xmx1g C:
\Users\Jennifer\Desktop\VizBin\VizBin-dist.jar
2014-12-10 10:51:04,320 DEBUG [main] (MainFrame.java:61) - Init of Main applicat
ion frame
2014-12-10 10:51:28,750 DEBUG [AWT-EventQueue-0] (ProcessInput.java:61) - Init o
f ProcessInput
2014-12-10 10:51:28,765 DEBUG [Thread-3] (ProcessInput.java:115) - C:\Users\Jenn
ifer\AppData\Local\Temp\
2014-12-10 10:51:29,062 DEBUG [Thread-3] (ProcessInput.java:124) - Loading data
from file.
Contig length treshold: 1000
2014-12-10 10:51:29,062 DEBUG [Thread-3] (ProcessInput.java:91) - Loading fasta
file: C:\Users\Jennifer\Desktop\VizBin\37A\37A\37A.concatenated.fa
2014-12-10 10:51:34,740 DEBUG [Thread-3] (ProcessInput.java:91) - DataSet loaded
 (13042 sequences)
2014-12-10 10:51:34,740 DEBUG [Thread-3] (ProcessInput.java:91) - Creating kmers
 (k=5, merge = true)
2014-12-10 10:51:34,756 DEBUG [Thread-3] (DataSetUtils.java:65) - Creating usedV
al for k=5
2014-12-10 10:51:41,604 DEBUG [Thread-3] (ProcessInput.java:91) - Normalizing ve
ctors...
2014-12-10 10:51:41,869 DEBUG [Thread-3] (ProcessInput.java:91) - Clr normalizat
ion...
2014-12-10 10:51:42,805 DEBUG [Thread-3] (ProcessInput.java:91) - Running PCA...
 (Mtj)
Dez 10, 2014 10:51:46 AM com.github.fommil.netlib.LAPACK <clinit>
Warnung: Failed to load implementation from: com.github.fommil.netlib.NativeSyst
emLAPACK
Dez 10, 2014 10:51:47 AM com.github.fommil.jni.JniLoader liberalLoad
Information: successfully loaded C:\Users\Jennifer\AppData\Local\Temp\jniloader2
365430519185301729netlib-native_ref-win-i686.dll
2014-12-10 10:52:21,587 DEBUG [Thread-3] (DataSetUtils.java:249) - DONE: Compute
d the new basis.
2014-12-10 10:52:42,085 DEBUG [Thread-3] (DataSetUtils.java:256) - DONE: Project
ed from sample to eigen space.
2014-12-10 10:52:42,085 DEBUG [Thread-3] (ProcessInput.java:91) - Running T-SNE.
..
2014-12-10 10:52:45,049 DEBUG [Thread-3] (DataSetUtils.java:264) - Running comma
nd: "C:\Users\Jennifer\.vizbin\pbh_tsne.exe" in directory: C:\Users\Jennifer\App
Data\Local\Temp\map7054369329860871950

Number of threads: 1
Seed: 0
2014-12-10 10:52:47,093 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Read
the 13042 x 50 data matrix successfully!
2014-12-10 10:52:47,093 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Runni
ng BH_TSNE with 1 threads and seed = 0
2014-12-10 10:52:47,093 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Using
 no_dims = 2, perplexity = 30.000000, theta = 0.500000
claczny commented 9 years ago

From Yohan's console output (OSX Yosemite10.10.1 / Retina): 

2014-12-10 10:43:28,928 DEBUG [main] (MainFrame.java:61) - Init of Main application frame
2014-12-10 10:43:43,777 DEBUG [AWT-EventQueue-0] (ProcessInput.java:61) - Init of ProcessInput
2014-12-10 10:43:43,780 DEBUG [Thread-1] (ProcessInput.java:115) - /var/folders/gn/p47pzlx16v7bnwnwv4b10rv9skqz7l/T/
2014-12-10 10:43:43,794 DEBUG [Thread-1] (ProcessInput.java:124) - Loading data from file.
Contig length treshold: 1000
2014-12-10 10:43:43,794 DEBUG [Thread-1] (ProcessInput.java:91) - Loading fasta file: /Users/yohan.jarosz/Downloads/37A/37A.concatenated.fa
2014-12-10 10:43:44,664 DEBUG [Thread-1] (ProcessInput.java:91) - DataSet loaded (13042 sequences)
2014-12-10 10:43:44,665 DEBUG [Thread-1] (ProcessInput.java:91) - Creating kmers (k=5, merge = true)
2014-12-10 10:43:44,666 DEBUG [Thread-1] (DataSetUtils.java:65) - Creating usedVal for k=5
2014-12-10 10:43:47,529 DEBUG [Thread-1] (ProcessInput.java:91) - Normalizing vectors...
2014-12-10 10:43:47,597 DEBUG [Thread-1] (ProcessInput.java:91) - Clr normalization...
2014-12-10 10:43:48,247 DEBUG [Thread-1] (ProcessInput.java:91) - Running PCA... (Mtj)
Dec 10, 2014 10:43:48 AM com.github.fommil.jni.JniLoader liberalLoad
INFO: successfully loaded /var/folders/gn/p47pzlx16v7bnwnwv4b10rv9skqz7l/T/jniloader3997748089661723248netlib-native_system-osx-x86_64.jnilib
2014-12-10 10:43:49,150 DEBUG [Thread-1] (DataSetUtils.java:249) - DONE: Computed the new basis.
2014-12-10 10:43:49,629 DEBUG [Thread-1] (DataSetUtils.java:256) - DONE: Projected from sample to eigen space.
2014-12-10 10:43:49,630 DEBUG [Thread-1] (ProcessInput.java:91) - Running T-SNE...
2014-12-10 10:43:49,965 DEBUG [Thread-1] (DataSetUtils.java:264) - Running command: "/Users/yohan.jarosz/.vizbin/pbh_tsne_osx" in directory: /var/folders/gn/p47pzlx16v7bnwnwv4b10rv9skqz7l/T/map2098942954547252279
Number of threads: 1
Seed: 0
2014-12-10 10:43:50,377 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Read the 13042 x 50 data matrix successfully!
2014-12-10 10:43:50,379 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Running BH_TSNE with 1 threads and seed = 0
2014-12-10 10:43:50,380 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Using no_dims = 2, perplexity = 30.000000, theta = 0.500000
2014-12-10 10:43:50,380 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Computing input similarities...
2014-12-10 10:43:50,459 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Building tree...
2014-12-10 10:43:50,459 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE:  - point 0 of 13042
2014-12-10 10:43:52,338 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE:  - point 10000 of 13042
2014-12-10 10:43:53,175 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Done in 2.79 seconds (sparsity = 0.010317)!
2014-12-10 10:43:53,175 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Learning embedding...
2014-12-10 10:43:55,891 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 50: error is 98.882329 (50 iterations in 0.00 seconds)
2014-12-10 10:43:58,439 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 100: error is 87.618302 (50 iterations in 0.00 seconds)
2014-12-10 10:44:00,705 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 150: error is 81.454664 (50 iterations in 0.00 seconds)
2014-12-10 10:44:02,919 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 200: error is 80.281685 (50 iterations in 0.00 seconds)
2014-12-10 10:44:05,062 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 250: error is 4.155104 (50 iterations in 0.00 seconds)
2014-12-10 10:44:07,134 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 300: error is 3.178465 (50 iterations in 0.00 seconds)
2014-12-10 10:44:09,252 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 350: error is 2.854119 (50 iterations in 0.00 seconds)
2014-12-10 10:44:11,337 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 400: error is 2.672712 (50 iterations in 0.00 seconds)

-> MTJ is working at maximum performance, as expected for a Mac.

claczny commented 9 years ago

From Ubuntu-14.10VM:

test@test-VirtualBox:~/VizBin$ java -jar VizBin-dist.jar 
2014-12-10 11:08:56,816 DEBUG [main] (MainFrame.java:61) - Init of Main application frame
2014-12-10 11:09:44,795 DEBUG [AWT-EventQueue-0] (ProcessInput.java:61) - Init of ProcessInput
2014-12-10 11:09:44,800 DEBUG [Thread-0] (ProcessInput.java:115) - /tmp
2014-12-10 11:09:44,835 DEBUG [Thread-0] (ProcessInput.java:124) - Loading data from file.
Contig length treshold: 1000
2014-12-10 11:09:44,835 DEBUG [Thread-0] (ProcessInput.java:91) - Loading fasta file: /home/test/Downloads/37A/37A/37A.concatenated.fa
2014-12-10 11:09:46,574 DEBUG [Thread-0] (ProcessInput.java:91) - DataSet loaded (13042 sequences)
2014-12-10 11:09:46,575 DEBUG [Thread-0] (ProcessInput.java:91) - Creating kmers (k=5, merge = true)
2014-12-10 11:09:46,579 DEBUG [Thread-0] (DataSetUtils.java:65) - Creating usedVal for k=5
2014-12-10 11:09:50,596 DEBUG [Thread-0] (ProcessInput.java:91) - Normalizing vectors...
2014-12-10 11:09:50,707 DEBUG [Thread-0] (ProcessInput.java:91) - Clr normalization...
2014-12-10 11:09:51,061 DEBUG [Thread-0] (ProcessInput.java:91) - Running PCA... (Mtj)
Dec 10, 2014 11:09:52 AM com.github.fommil.jni.JniLoader liberalLoad
INFO: successfully loaded /tmp/jniloader9144771906803982692netlib-native_system-linux-x86_64.so
2014-12-10 11:09:56,600 DEBUG [Thread-0] (DataSetUtils.java:249) - DONE: Computed the new basis.
2014-12-10 11:09:58,336 DEBUG [Thread-0] (DataSetUtils.java:256) - DONE: Projected from sample to eigen space.
2014-12-10 11:09:58,337 DEBUG [Thread-0] (ProcessInput.java:91) - Running T-SNE...
2014-12-10 11:09:59,068 DEBUG [Thread-0] (DataSetUtils.java:264) - Running command: "/home/test/.vizbin/pbh_tsne" in directory: /tmp/map6211623356938677046
Number of threads: 2
Seed: 0
2014-12-10 11:09:59,415 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Read the 13042 x 50 data matrix successfully!
2014-12-10 11:09:59,421 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Running BH_TSNE with 2 threads and seed = 0
2014-12-10 11:09:59,422 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Using no_dims = 2, perplexity = 30.000000, theta = 0.500000 
2014-12-10 11:09:59,422 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Computing input similarities...
2014-12-10 11:09:59,557 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Building tree...
2014-12-10 11:09:59,557 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE:  - point 0 of 13042
2014-12-10 11:10:01,859 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE:  - point 10000 of 13042
2014-12-10 11:10:02,907 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Done in 3.48 seconds (sparsity = 0.010317)!
2014-12-10 11:10:02,907 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Learning embedding...
2014-12-10 11:10:05,780 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 50: error is 98.882329 (50 iterations in 0.00 seconds)
2014-12-10 11:10:08,680 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 100: error is 87.618302 (50 iterations in 0.00 seconds)
2014-12-10 11:10:11,312 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 150: error is 81.455067 (50 iterations in 0.00 seconds)
2014-12-10 11:10:13,889 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 200: error is 80.282551 (50 iterations in 0.00 seconds)
2014-12-10 11:10:16,425 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 250: error is 4.155243 (50 iterations in 0.00 seconds)
2014-12-10 11:10:18,812 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 300: error is 3.178847 (50 iterations in 0.00 seconds)
2014-12-10 11:10:21,273 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 350: error is 2.854269 (50 iterations in 0.00 seconds)
2014-12-10 11:10:23,672 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 400: error is 2.672754 (50 iterations in 0.00 seconds)
2014-12-10 11:10:26,154 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 450: error is 2.554703 (50 iterations in 0.00 seconds)
2014-12-10 11:10:28,864 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 500: error is 2.470680 (50 iterations in 0.00 seconds)
2014-12-10 11:10:31,203 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 550: error is 2.407209 (50 iterations in 0.00 seconds)
2014-12-10 11:10:33,592 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 600: error is 2.357605 (50 iterations in 0.00 seconds)
2014-12-10 11:10:35,978 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 650: error is 2.317692 (50 iterations in 0.00 seconds)
2014-12-10 11:10:38,360 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 700: error is 2.285571 (50 iterations in 0.00 seconds)
2014-12-10 11:10:40,739 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 750: error is 2.259456 (50 iterations in 0.00 seconds)
2014-12-10 11:10:43,100 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 800: error is 2.238512 (50 iterations in 0.00 seconds)
2014-12-10 11:10:45,503 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 850: error is 2.222015 (50 iterations in 0.00 seconds)
2014-12-10 11:10:47,846 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 900: error is 2.209019 (50 iterations in 0.00 seconds)
2014-12-10 11:10:50,261 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 950: error is 2.199357 (50 iterations in 0.00 seconds)
2014-12-10 11:10:52,644 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Iteration 999: error is 2.192653 (50 iterations in 0.00 seconds)
2014-12-10 11:10:52,645 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Fitting performed in 0.00 seconds.
2014-12-10 11:10:52,664 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: Wrote the 13042 x 2 data matrix successfully!
2014-12-10 11:10:52,664 DEBUG [TSNERunner] (DataSetUtils.java:300) - TSNE: 
2014-12-10 11:10:52,686 DEBUG [Thread-0] (ProcessInput.java:91) - Points created.
2014-12-10 11:10:52,989 DEBUG [Thread-0] (ProcessInput.java:91) - Creating png files....
2014-12-10 11:10:55,109 DEBUG [Thread-0] (ProcessInput.java:91) - Done.

-> MTJ is working at maximum performance, as expected for a Linux with the necessary packages installed (i.e., BLAS and LAPACK).

claczny commented 9 years ago

Average error between the points file returned by Windows and the one under Mac OS X

cedric.laczny@CLMBP:~/Documents/phd/projects/BINNING/publication/VizBin_-_Application_Note/revision_01/data/examples/37A $ paste -d"," 37A.concatenated.points /Users/cedric.laczny/Downloads/37A_Windows_points.txt | awk -F"," '{sum += sqrt(($1-$3)*($1-$3))} END {print sum}'
4999.66
cedric.laczny@CLMBP:~/Documents/phd/projects/BINNING/publication/VizBin_-_Application_Note/revision_01/data/examples/37A $ paste -d"," 37A.concatenated.points /Users/cedric.laczny/Downloads/37A_Windows_points.txt | awk -F"," '{sum += sqrt(($1-$3)*($1-$3))} END {print sum/NR}'
0.383351
cedric.laczny@CLMBP:~/Documents/phd/projects/BINNING/publication/VizBin_-_Application_Note/revision_01/data/examples/37A $ paste -d"," 37A.concatenated.points /Users/cedric.laczny/Downloads/37A_Windows_points.txt | awk -F"," '{sum += sqrt(($2-$4)*($2-$4))} END {print sum/NR}'
0.647647
cedric.laczny@CLMBP:~/Documents/phd/projects/BINNING/publication/VizBin_-_Application_Note/revision_01/data/examples/37A $ head 37A.concatenated.points /Users/cedric.laczny/Downloads/37A_Windows_points.txt
==> 37A.concatenated.points <==
15.148317,-10.563461
15.546716,-10.248951
15.815515,-10.100727
14.651723,-10.931276
15.734991,-10.211598
16.043028,-9.728845
15.835106,-10.058832
15.644402,-10.170950
15.822641,-10.046160
15.184782,-10.477033

==> /Users/cedric.laczny/Downloads/37A_Windows_points.txt <==
15.084483,-10.184800
15.497696,-9.890971
15.773701,-9.769374
14.572603,-10.524224
15.691000,-9.870220
16.049392,-9.413775
15.798445,-9.729780
15.601806,-9.814748
15.783813,-9.717228
15.123478,-10.100430
cedric.laczny@CLMBP:~/Documents/phd/projects/BINNING/publication/VizBin_-_Application_Note/revision_01/data/examples/37A $

Seems to be related mainly to differences in precision etc. (really hard to avoid and would not add any information as it is anyways a locally optimal solution, albeit a very good one.)