SHAPEWarp on the test file results in "[!] No significant match with current search parameters"

[vshanka23]

Followed all installation procedures. Make test succeeded.

Running: ./SHAPEwarp -q t/query.txt -d t/ -o test_out -ow Results in: [!] No significant match with current search parameters

This should not happen. I do not get any errors to use to debug. So, do you have any ideas?

[dincarnato]

Hello,

I have just tried to do a fresh git clone from the repo on a clean machine, compiled and ran the same test, and I get the expected output. Did you run the "perl Makefile.PL" followed by "make"? What did the "make test" return? Also, after having run SHAPEwarp, if you scroll up in your terminal, is there any other message you might have not seen?

Best, Danny

[vshanka23]

Installed Rust and Cargo. Downloaded RNAFramework (make test won't succeed without it anyway) and added PERL5LIB path to it. Installed both FFI modules (Platypus and Platypus::Lang::Rust). All builds and compiles passed with "OKs" perl MakeFile.PL make make test (Result; PASS)

Running SHAPEwarp, starts the Kmer part of the pipeline, then instantly quits to [!] No significant match with current search parameters.

Any other ideas?

[dincarnato]

Can you please run the following from within the SHAPEwarp directory and show me the output?

./swKmerLookup --seq AGGTTAAAACTCAAATGAATTGACGGGGGCCCGCACAAGCGGTGGAGCATGTGGTTTAATTCGATGCAACGCGAAGAACCTTACCTGGTCTTGACATCCACGGAAGTTTTCAGAGATGAGAATGTGCCTTCGGGAACCGTGAGACAGGTGCTGCATGGCTGTCGTCAGCTCGTGTTGTGAAATGTTGGGTTAAGTCCCGC --react 0.885,0.181,0.189,0.239,0.531,0.141,0.126,0.491,0.648,0.114,0.171,0.332,0.099,0.601,0.849,1.015,0.928,0.221,0.464,0.482,0.398,0.082,0.079,0.141,0.649,0.419,0.172,0.242,0.391,0.000,0.000,0.005,0.220,0.022,0.270,0.195,0.252,0.140,0.055,0.060,0.042,0.150,0.122,0.146,0.063,0.000,0.038,0.042,0.000,0.066,0.075,0.190,0.144,0.333,0.025,0.203,0.374,0.239,0.219,0.076,0.123,0.427,1.066,0.698,3.256,2.983,1.165,1.075,0.681,1.411,0.561,0.632,0.716,0.341,0.412,0.234,0.142,0.049,0.029,0.194,0.346,0.390,0.842,0.096,0.011,0.054,0.022,0.135,0.120,0.868,1.492,2.061,0.645,0.327,0.502,0.611,0.236,0.129,0.095,0.072,0.041,NaN,0.251,0.222,0.407,NaN,0.040,0.016,0.023,NaN,0.063,0.075,0.183,0.071,0.518,0.520,0.127,0.040,NaN,0.150,0.055,NaN,NaN,0.937,2.314,0.550,0.028,0.116,0.380,0.589,2.475,0.653,0.081,0.079,0.164,0.117,0.003,NaN,0.069,0.031,0.084,1.128,0.397,0.443,0.767,0.376,0.148,0.443,1.367,0.486,0.090,0.352,0.411,0.768,0.045,0.034,0.012,0.034,0.028,0.045,0.085,0.080,0.065,0.232,0.234,0.295,0.346,0.056,0.040,0.002,0.145,0.709,0.990,1.410,1.006,3.039,2.495,0.820,1.965,0.641,1.749,2.138,2.212,1.593,1.623,2.142,0.252,0.081,0.240,0.474,1.106,1.346,1.535,4.573,0.715,0.003,0.009,0.021,0.064,0.168 --threads 50 --minKmers 2 --maxReactivity 1 --maxKmerDist 30 --kmerMaxSeqDist 0 --kmerLen 15 --kmerOffset 1 --kmerMinComplexity 0.3 --kmerMaxMatchEveryNt 200 --db t/reactivity.db --debug

[vshanka23]

Some more details: The terminal clears when running SHAPEwarp. Meaning, the moment it is executed, I get a clean output from the terminal and previous messages and content are cleared. Not sure this is expected or designed behavior.

I have a strong suspicion that the kmerlookup part of the compile likely failed because it instantly quits when trying to run the query look up part. I even downloaded the sets from the Application Case portion of the readthedocs: https://shapewarp-docs.readthedocs.io/en/latest/case/ and got the same result. So, there is definitely a problem with my installation. I just don't know where the issue specifically is currently.

[dincarnato]

Please try the command above and let me know. Also, the terminal clears, but if you scroll up, the full output should still be visible.

[vshanka23]

Found the problem!

The header of the swKmerLookup makes the assumption that system perl (#!/usr/bin/perl) should be used. I switched to custom location of our Perl (we run on HPC and inside specialized environments per user) and that seems to have generated this:

{ dbId => 16S_Bsubtilis, db => [38,59], query => [0,21] }
{ dbId => 16S_Bsubtilis, db => [86,101], query => [0,15] }
{ dbId => 16S_Bsubtilis, db => [740,764], query => [0,24] }
{ dbId => 16S_Bsubtilis, db => [1102,1118], query => [0,16] }
{ dbId => 16S_Bsubtilis, db => [1371,1418], query => [0,47] }
{ dbId => 16S_Bsubtilis, db => [144,183], query => [1,40] }
{ dbId => 16S_Bsubtilis, db => [1017,1036], query => [1,20] }
{ dbId => 16S_Bsubtilis, db => [167,189], query => [4,26] }
{ dbId => 16S_Bsubtilis, db => [182,204], query => [4,26] }
{ dbId => 16S_Bsubtilis, db => [1331,1363], query => [4,36] }
{ dbId => 16S_Bsubtilis, db => [90,109], query => [5,24] }
{ dbId => 16S_Bsubtilis, db => [446,462], query => [7,23] }
{ dbId => 16S_Bsubtilis, db => [505,521], query => [7,23] }
{ dbId => 16S_Bsubtilis, db => [497,516], query => [9,28] }
{ dbId => 16S_Bsubtilis, db => [347,378], query => [11,42] }
{ dbId => 16S_Bsubtilis, db => [1261,1279], query => [11,29] }
{ dbId => 16S_Bsubtilis, db => [1302,1318], query => [13,29] }
{ dbId => 16S_Bsubtilis, db => [590,605], query => [14,29] }
{ dbId => 16S_Bsubtilis, db => [1486,1501], query => [14,29] }
{ dbId => 16S_Bsubtilis, db => [925,1009], query => [16,100] }
{ dbId => 16S_Bsubtilis, db => [379,398], query => [17,36] }
{ dbId => 16S_Bsubtilis, db => [1029,1045], query => [17,33] }
{ dbId => 16S_Bsubtilis, db => [1210,1227], query => [17,34] }
{ dbId => 16S_Bsubtilis, db => [633,649], query => [21,37] }
{ dbId => 16S_Bsubtilis, db => [1339,1357], query => [21,39] }
{ dbId => 16S_Bsubtilis, db => [927,943], query => [26,42] }
{ dbId => 16S_Bsubtilis, db => [1030,1046], query => [26,42] }
{ dbId => 16S_Bsubtilis, db => [1245,1261], query => [26,42] }
{ dbId => 16S_Bsubtilis, db => [1199,1216], query => [29,46] }
{ dbId => 16S_Bsubtilis, db => [249,264], query => [35,50] }
{ dbId => 16S_Bsubtilis, db => [380,395], query => [35,50] }
{ dbId => 16S_Bsubtilis, db => [1117,1132], query => [35,50] }
{ dbId => 16S_Bsubtilis, db => [1146,1161], query => [35,50] }
{ dbId => 16S_Bsubtilis, db => [1345,1360], query => [35,50] }
{ dbId => 16S_Bsubtilis, db => [1189,1205], query => [36,52] }
{ dbId => 16S_Bsubtilis, db => [1009,1024], query => [37,52] }
{ dbId => 16S_Bsubtilis, db => [668,683], query => [38,53] }
{ dbId => 16S_Bsubtilis, db => [145,160], query => [39,54] }
{ dbId => 16S_Bsubtilis, db => [599,616], query => [39,56] }
{ dbId => 16S_Bsubtilis, db => [288,305], query => [42,59] }
{ dbId => 16S_Bsubtilis, db => [2,21], query => [51,70] }
{ dbId => 16S_Bsubtilis, db => [959,977], query => [51,69] }
{ dbId => 16S_Bsubtilis, db => [1104,1119], query => [51,66] }
{ dbId => 16S_Bsubtilis, db => [1128,1147], query => [51,70] }
{ dbId => 16S_Bsubtilis, db => [757,773], query => [54,70] }
{ dbId => 16S_Bsubtilis, db => [1036,1051], query => [68,83] }
{ dbId => 16S_Bsubtilis, db => [1235,1251], query => [70,86] }
{ dbId => 16S_Bsubtilis, db => [1467,1482], query => [71,86] }
{ dbId => 16S_Bsubtilis, db => [1161,1178], query => [77,94] }
{ dbId => 16S_Bsubtilis, db => [206,227], query => [79,100] }
{ dbId => 16S_Bsubtilis, db => [445,463], query => [82,100] }
{ dbId => 16S_Bsubtilis, db => [496,511], query => [83,98] }
{ dbId => 16S_Bsubtilis, db => [507,523], query => [84,100] }
{ dbId => 16S_Bsubtilis, db => [187,202], query => [85,100] }
{ dbId => 16S_Bsubtilis, db => [507,522], query => [85,100] }
{ dbId => 16S_Bsubtilis, db => [296,311], query => [138,153] }
{ dbId => 16S_Bsubtilis, db => [1078,1093], query => [138,153] }
{ dbId => 16S_Bsubtilis, db => [996,1011], query => [145,160] }
{ dbId => 16S_Bsubtilis, db => [1363,1380], query => [145,162] }
{ dbId => 16S_Bsubtilis, db => [1300,1315], query => [148,163] }
{ dbId => 16S_Bsubtilis, db => [3,22], query => [161,180] }
{ dbId => 16S_Bsubtilis, db => [837,853], query => [161,177] }
{ dbId => 16S_Bsubtilis, db => [960,979], query => [161,180] }
{ dbId => 16S_Bsubtilis, db => [1128,1147], query => [161,180] }
{ dbId => 16S_Bsubtilis, db => [1129,1148], query => [161,180] }
{ dbId => 16S_Bsubtilis, db => [3,19], query => [162,178] }
{ dbId => 16S_Bsubtilis, db => [798,813], query => [182,197] }
{ dbId => 16S_Bsubtilis, db => [581,597], query => [183,199] }
{ dbId => 16S_Bsubtilis, db => [1030,1046], query => [183,199] }
{ dbId => 16S_Bsubtilis, db => [1092,1108], query => [183,199] }
{ dbId => 16S_Bsubtilis, db => [1211,1227], query => [183,199] }
{ dbId => 23S_Bsubtilis, db => [462,477], query => [0,15] }
{ dbId => 23S_Bsubtilis, db => [853,872], query => [0,19] }
{ dbId => 23S_Bsubtilis, db => [2025,2041], query => [0,16] }
{ dbId => 23S_Bsubtilis, db => [2754,2807], query => [0,53] }
{ dbId => 23S_Bsubtilis, db => [1742,1757], query => [1,16] }
{ dbId => 23S_Bsubtilis, db => [1849,1864], query => [1,16] }
{ dbId => 23S_Bsubtilis, db => [2567,2582], query => [1,16] }
{ dbId => 23S_Bsubtilis, db => [1711,1730], query => [2,21] }
{ dbId => 23S_Bsubtilis, db => [438,455], query => [3,20] }
{ dbId => 23S_Bsubtilis, db => [490,513], query => [3,26] }
{ dbId => 23S_Bsubtilis, db => [1144,1164], query => [3,23] }
{ dbId => 23S_Bsubtilis, db => [1417,1442], query => [3,28] }
{ dbId => 23S_Bsubtilis, db => [2646,2661], query => [3,18] }
{ dbId => 23S_Bsubtilis, db => [288,305], query => [4,21] }
{ dbId => 23S_Bsubtilis, db => [840,858], query => [4,22] }
{ dbId => 23S_Bsubtilis, db => [960,975], query => [5,20] }
{ dbId => 23S_Bsubtilis, db => [1105,1121], query => [7,23] }
{ dbId => 23S_Bsubtilis, db => [2000,2023], query => [7,30] }
{ dbId => 23S_Bsubtilis, db => [2326,2344], query => [7,25] }
{ dbId => 23S_Bsubtilis, db => [426,447], query => [8,29] }
{ dbId => 23S_Bsubtilis, db => [1624,1644], query => [9,29] }
{ dbId => 23S_Bsubtilis, db => [2884,2901], query => [9,26] }
{ dbId => 23S_Bsubtilis, db => [972,991], query => [10,29] }
{ dbId => 23S_Bsubtilis, db => [1459,1477], query => [10,28] }
{ dbId => 23S_Bsubtilis, db => [2631,2647], query => [10,26] }
{ dbId => 23S_Bsubtilis, db => [661,678], query => [12,29] }
{ dbId => 23S_Bsubtilis, db => [2531,2584], query => [13,66] }
{ dbId => 23S_Bsubtilis, db => [97,112], query => [14,29] }
{ dbId => 23S_Bsubtilis, db => [551,566], query => [14,29] }
{ dbId => 23S_Bsubtilis, db => [1504,1519], query => [14,29] }
{ dbId => 23S_Bsubtilis, db => [1770,1785], query => [17,32] }
{ dbId => 23S_Bsubtilis, db => [1807,1826], query => [17,36] }
{ dbId => 23S_Bsubtilis, db => [2378,2394], query => [17,33] }
{ dbId => 23S_Bsubtilis, db => [2459,2474], query => [17,32] }
{ dbId => 23S_Bsubtilis, db => [2499,2515], query => [17,33] }
{ dbId => 23S_Bsubtilis, db => [1366,1406], query => [20,60] }
{ dbId => 23S_Bsubtilis, db => [1693,1709], query => [20,36] }
{ dbId => 23S_Bsubtilis, db => [1832,1848], query => [21,37] }
{ dbId => 23S_Bsubtilis, db => [2248,2263], query => [21,36] }
{ dbId => 23S_Bsubtilis, db => [1248,1264], query => [25,41] }
{ dbId => 23S_Bsubtilis, db => [2198,2214], query => [25,41] }
{ dbId => 23S_Bsubtilis, db => [544,564], query => [27,47] }
{ dbId => 23S_Bsubtilis, db => [2008,2031], query => [27,50] }
{ dbId => 23S_Bsubtilis, db => [134,158], query => [29,53] }
{ dbId => 23S_Bsubtilis, db => [293,308], query => [29,44] }
{ dbId => 23S_Bsubtilis, db => [2720,2741], query => [29,50] }
{ dbId => 23S_Bsubtilis, db => [444,459], query => [30,45] }
{ dbId => 23S_Bsubtilis, db => [943,958], query => [30,45] }
{ dbId => 23S_Bsubtilis, db => [1274,1291], query => [30,47] }
{ dbId => 23S_Bsubtilis, db => [2867,2882], query => [30,45] }
{ dbId => 23S_Bsubtilis, db => [607,626], query => [31,50] }
{ dbId => 23S_Bsubtilis, db => [2631,2647], query => [31,47] }
{ dbId => 23S_Bsubtilis, db => [2530,2545], query => [32,47] }
{ dbId => 23S_Bsubtilis, db => [1316,1333], query => [33,50] }
{ dbId => 23S_Bsubtilis, db => [56,71], query => [37,52] }
{ dbId => 23S_Bsubtilis, db => [2587,2604], query => [39,56] }
{ dbId => 23S_Bsubtilis, db => [11,32], query => [49,70] }
{ dbId => 23S_Bsubtilis, db => [956,971], query => [49,64] }
{ dbId => 23S_Bsubtilis, db => [1414,1430], query => [49,65] }
{ dbId => 23S_Bsubtilis, db => [1849,1870], query => [49,70] }
{ dbId => 23S_Bsubtilis, db => [2072,2090], query => [49,67] }
{ dbId => 23S_Bsubtilis, db => [2106,2121], query => [49,64] }
{ dbId => 23S_Bsubtilis, db => [237,252], query => [50,65] }
{ dbId => 23S_Bsubtilis, db => [488,503], query => [50,65] }
{ dbId => 23S_Bsubtilis, db => [1743,1759], query => [50,66] }
{ dbId => 23S_Bsubtilis, db => [2252,2267], query => [50,65] }
{ dbId => 23S_Bsubtilis, db => [1205,1220], query => [55,70] }
{ dbId => 23S_Bsubtilis, db => [1996,2013], query => [69,86] }
{ dbId => 23S_Bsubtilis, db => [2238,2255], query => [69,86] }
{ dbId => 23S_Bsubtilis, db => [2360,2376], query => [70,86] }
{ dbId => 23S_Bsubtilis, db => [2448,2464], query => [70,86] }
{ dbId => 23S_Bsubtilis, db => [2450,2465], query => [71,86] }
{ dbId => 23S_Bsubtilis, db => [1198,1214], query => [76,92] }
{ dbId => 23S_Bsubtilis, db => [2585,2600], query => [76,91] }
{ dbId => 23S_Bsubtilis, db => [2801,2816], query => [76,91] }
{ dbId => 23S_Bsubtilis, db => [736,751], query => [78,93] }
{ dbId => 23S_Bsubtilis, db => [1422,1438], query => [84,100] }
{ dbId => 23S_Bsubtilis, db => [2340,2355], query => [84,99] }
{ dbId => 23S_Bsubtilis, db => [1107,1122], query => [85,100] }
{ dbId => 23S_Bsubtilis, db => [2353,2368], query => [85,100] }
{ dbId => 23S_Bsubtilis, db => [1870,1885], query => [138,153] }
{ dbId => 23S_Bsubtilis, db => [1689,1704], query => [140,155] }
{ dbId => 23S_Bsubtilis, db => [2016,2031], query => [140,155] }
{ dbId => 23S_Bsubtilis, db => [1364,1381], query => [142,159] }
{ dbId => 23S_Bsubtilis, db => [2038,2056], query => [142,160] }
{ dbId => 23S_Bsubtilis, db => [2119,2136], query => [143,160] }
{ dbId => 23S_Bsubtilis, db => [1682,1697], query => [145,160] }
{ dbId => 23S_Bsubtilis, db => [659,675], query => [147,163] }
{ dbId => 23S_Bsubtilis, db => [12,33], query => [159,180] }
{ dbId => 23S_Bsubtilis, db => [488,503], query => [159,174] }
{ dbId => 23S_Bsubtilis, db => [956,971], query => [159,174] }
{ dbId => 23S_Bsubtilis, db => [1415,1430], query => [159,174] }
{ dbId => 23S_Bsubtilis, db => [1743,1759], query => [159,175] }
{ dbId => 23S_Bsubtilis, db => [1849,1869], query => [159,179] }
{ dbId => 23S_Bsubtilis, db => [2072,2089], query => [159,176] }
{ dbId => 23S_Bsubtilis, db => [2106,2121], query => [159,174] }
{ dbId => 23S_Bsubtilis, db => [2567,2584], query => [159,176] }
{ dbId => 23S_Bsubtilis, db => [2337,2352], query => [164,179] }
{ dbId => 23S_Bsubtilis, db => [2808,2824], query => [164,180] }
{ dbId => 23S_Bsubtilis, db => [1570,1585], query => [165,180] }
{ dbId => 23S_Bsubtilis, db => [173,189], query => [182,198] }
{ dbId => 23S_Bsubtilis, db => [688,704], query => [182,198] }
{ dbId => 23S_Bsubtilis, db => [1756,1771], query => [182,197] }
{ dbId => 23S_Bsubtilis, db => [829,844], query => [184,199] }

I was also able to run the small test from the https://github.com/dincarnato/SHAPEwarp#testing-the-shapewarp-installation.

Thank you for your help and your lightning fast responses! If I am allowed make a wish list, I wish there was an error catch on the swKmerLookup that trickled over to the main script or a log file generated for child processes.

[dincarnato]

Yes, i am aware of this, and don't worry. FYI, we are about to release a full rewrite of SHAPEwarp in Rust, that will take care of all these issues. Also, it will be light-speed fast (some benchmarks that took >1 day with the current version on 56 cores, run in seconds on 1 core in the new version).

Best, Danny

[dincarnato]

If you issue a git pull it should be fixed now to use the env perl.

[vshanka23]

Danny,

I did a pull and it works flawlessly now! Also, thanks for the heads-up on the Rust version of SHAPEwarp. We are looking forward to this. Thanks again for your help!

Cheers, Vijay