Kill Received

[jerryyyyy]

Hello,

I have compiled this and it is running but I get a "kill" result on Step 1 and the log last lines reads "TRF output lines processed 32481800". The mail at the bu web site is inop.

When I hit CtrC after the "kill" it says executing step #2.. no valid pairs for 0-0 command exited with value 255 at line 782.

Any help appreciated,

Jerry Y

[yzhernand]

Please try the version in branch "v1.09_maintenance" and see if that works for you. This is a test release which automatically splits up the input as it processes it, so it may produce a high number of intermediate files, but it shouldn't present much of a problem unless your input has a very large number of reads.

If it does work for you, I can tweak how often it splits the input.

[jerryyyyy]

Hey thanks for responding,

I just ran 108 with your test data and it died also in Step 3. So not my data.

Couldn’t execute statement… line 283

We have just completed WGS on 28 cases from Macrogen in Boston, guess local to you, and has run well on other software.

I’ll download and try it.

Jerry Y

Jerome Yesavage, M.D.

Jared and Mae Tinklenberg Professor

Associate Chair of Psychiatry and Behavioral Sciences and Professor of Neurology and Neurological Sciences (By Courtesy)

Stanford University School of Medicine

Internet E-Mail: mailto:yesavage@stanford.edu yesavage@stanford.edu

URL: http://www.stanford.edu/people/yesavage/ http://www.stanford.edu/people/yesavage/

Voice: 650-852-3287 Fax: 650-852-3297

From: Yozen Hernandez [mailto:notifications@github.com] Sent: Tuesday, February 13, 2018 6:50 AM To: yzhernand/VNTRseek VNTRseek@noreply.github.com Cc: jerryyyyy yesavage@stanford.edu; Author author@noreply.github.com Subject: Re: [yzhernand/VNTRseek] Kill Received (#7)

Please try the version in branch "v1.09_maintenance" and see if that works for you. This is a test release which automatically splits up the input as it processes it, so it may produce a high number of intermediate files, but it shouldn't present much of a problem unless your input has a very large number of reads.

If it does work for you, I can tweak how often it splits the input.

— You are receiving this because you authored the thread. Reply to this email directly, https://github.com/yzhernand/VNTRseek/issues/7#issuecomment-365288892 view it on GitHub, or https://github.com/notifications/unsubscribe-auth/AfoESwr1fmjwWhYgn32lAzAQR6m4CIT4ks5tUaEHgaJpZM4SD3Av mute the thread. https://github.com/notifications/beacon/AfoESyK10CK4MhxCsBF8uD7GVOl_Reaeks5tUaEHgaJpZM4SD3Av.gif

[jerryyyyy]

Got an install version on the Branch:

Found PerlModules: TRUE
CMake Error at perl/CMakeLists.txt:31: Parse error. Expected "(", got identifier with text "bin".

[yzhernand]

Sorry, left that in. Please pull and try again.

[jerryyyyy]

OK, that compiled and installed!

But:

lab@SLE-5B4-422-CKN:~$ vntrseek 0 19 --fasta_dir ~/Downloads/samplereads108/watsontest --DBSUFFIX kaci --SERVER Ubuntu --LOGIN root --PASS XXXXXXXXXXXXXXXX Can't locate Config/Simple.pm in @INC (you may need to install the Config::Simple module) (@INC contains: /usr/local/vntrseek1.09.1/lib /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.0 /usr/local/share/perl/5.26.0 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/local/vntrseek1.09.1/lib/vutil.pm line 6. BEGIN failed--compilation aborted at /usr/local/vntrseek1.09.1/lib/vutil.pm line 6. Compilation failed in require at /usr/local/bin/vntrseek line 53. BEGIN failed--compilation aborted at /usr/local/bin/vntrseek line 53.

[yzhernand]

I've pushed another change to that branch. That dependency is not actually necessary for VNTRseek but I didn't see the error because I have that module installed. Thanks for testing this out and thanks for your patience.

[jerryyyyy]

Nothing is easy:

lab@SLE-5B4-422-CKN:~/Downloads/VNTRseek-1.09_maintenance/VNTRseek-1.09_maintenance/build$ vntrseek 0 19 --fasta_dir ~/Downloads/samplereads108/watsontest --DBSUFFIX kaci --SERVER Ubuntu --LOGIN root --PASS XXXXXXXXXXXXXXXXXXXXXX Could not read global config File 'trf409-ngs.linux.exe' not found! at /usr/local/bin/vntrseek line 352.

[yzhernand]

Looks like I will need to test this in a clean environment and get back to you. Not sure why it says trf is not found, either. Thanks for the feedback.

[jerryyyyy]

Yes, tons of dependencies I am sure. I am running the latest Ubuntu builds.

[yzhernand]

Hi Jerry, could you please send the output of the following command: ls -l /usr/local/vntrseek1.09.1 and attach the install_manifest.txt file from the source build directory. I just want to check if the appropriate files were installed.

[jerryyyyy]

Will do this but have to transfer this mail over to the machine to copy properly.

From: Yozen Hernandez [mailto:notifications@github.com] Sent: Saturday, February 17, 2018 11:13 AM To: yzhernand/VNTRseek VNTRseek@noreply.github.com Cc: jerryyyyy yesavage@stanford.edu; Author author@noreply.github.com Subject: Re: [yzhernand/VNTRseek] Kill Received (#7)

Hi Jerry, could you please send the output of the following command: ls -l /usr/local/vntrseek1.09.1 and attach the install_manifest.txt file from the source build directory. I just want to check if the appropriate files were installed.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/yzhernand/VNTRseek/issues/7#issuecomment-366464349 , or mute the thread https://github.com/notifications/unsubscribe-auth/AfoES5kc8L6_eqii1BqBnGxhKd7Qr7V2ks5tVyS9gaJpZM4SD3Av . https://github.com/notifications/beacon/AfoES-JlSx8XNwft6IubGPHQkOk4JQPSks5tVyS9gaJpZM4SD3Av.gif

[jerryyyyy]

lab@SLE-5B4-422-CKN:/usr/local/include/bamtools/api$ ls -l /usr/local/vntrseek1.09.1/ total 1380 -rwxr-xr-x 1 root root 1586 Feb 13 10:09 checkleb36.pl -rw-r--r-- 1 root root 72 Feb 13 10:09 CLUSTER -rw-r--r-- 1 root root 35129 Feb 13 10:09 COPYING -rw-r--r-- 1 root root 501513 Feb 13 10:09 eucledian.dst -rwxr-xr-x 1 root root 2755 Feb 13 10:09 extra_index.pl -rwxr-xr-x 1 root root 42424 Feb 13 11:11 flankalign.exe -rwxr-xr-x 1 root root 525 Feb 13 10:09 ge7.pl -rwxr-xr-x 1 root root 9400 Feb 13 10:09 hist_flank_filter.pl -rw-r--r-- 1 root root 6900 Feb 13 10:09 HISTORY -rwxr-xr-x 1 root root 15112 Feb 13 10:09 insert_reads.pl -rwxr-xr-x 1 root root 42640 Feb 13 11:11 join_clusters.exe drwxr-xr-x 2 root root 4096 Feb 13 11:11 lib -rwxr-xr-x 1 root root 9158 Feb 13 10:09 map_dup.pl -rwxr-xr-x 1 root root 2930 Feb 13 10:09 master_for_qstub_test_advanced.sh -rwxr-xr-x 1 root root 330 Feb 13 10:09 master_for_qstub_test.sh -rwxr-xr-x 1 root root 3833 Feb 13 10:09 master_for_qsub_test_advanced_useDB.sh -rwxr-xr-x 1 root root 22440 Feb 13 11:11 pcr_dup.exe -rwxr-xr-x 1 root root 15176 Feb 13 10:09 pcr_dup.pl -rwxr-xr-x 1 root root 1482 Feb 13 10:09 print_gained_by_patsize.pl -rwxr-xr-x 1 root root 7654 Feb 13 10:09 print_spanned_reads.pl -rwxr-xr-x 1 root root 1417 Feb 13 10:09 print_unclustered.pl -rwxr-xr-x 1 root root 1402 Feb 13 10:09 print_unmapped.pl -rwxr-xr-x 1 root root 5542 Feb 13 10:09 produce_indist.pl -rwxr-xr-x 1 root root 92368 Feb 13 11:11 psearch.exe -rwxr-xr-x 1 root root 1186 Feb 13 10:09 qsub_test_advanced.sh -rwxr-xr-x 1 root root 3209 Feb 13 10:09 qsub_test_advanced_useDB.sh -rwxr-xr-x 1 root root 288 Feb 13 10:09 qsub_test.sh -rw-r--r-- 1 root root 1002 Feb 13 10:09 README -rwxr-xr-x 1 root root 65120 Feb 13 11:11 redund.exe -rwxr-xr-x 1 root root 50832 Feb 13 11:11 refflankalign.exe -rwxr-xr-x 1 root root 5033 Feb 13 10:09 remove_sw_fpvntrs.pl -rwxr-xr-x 1 root root 5606 Feb 13 10:09 renumber.pl -rwxr-xr-x 1 root root 12407 Feb 13 10:09 run_assemblyreq.pl -rwxr-xr-x 1 root root 22400 Feb 13 10:09 run_edges.pl -rwxr-xr-x 1 root root 10811 Feb 13 10:09 run_flankcomp.pl -rw-r--r-- 1 root root 72 Feb 13 10:09 RUNNING -rwxr-xr-x 1 root root 5816 Feb 13 10:09 run_proclu.pl -rwxr-xr-x 1 root root 12827 Feb 13 10:09 run_rankflankmap.pl -rwxr-xr-x 1 root root 5651 Feb 13 10:09 run_trf.pl -rwxr-xr-x 1 root root 20357 Feb 13 10:09 run_variability.pl -rwxr-xr-x 1 root root 2976 Feb 13 10:09 setdbstats.pl -rwxr-xr-x 1 root root 1425 Feb 13 10:09 sim_sw_reads.sh -rwxr-xr-x 1 root root 5402 Feb 13 10:09 stats.pl -rwxr-xr-x 1 root root 91608 Feb 13 11:11 trf2proclu-ngs.exe -rwxr-xr-x 1 root root 86960 Feb 13 10:09 updaterefs.pl -rwxr-xr-x 1 root root 4423 Feb 13 10:09 vntrpipe_xml2sql.pl -rwxr-xr-x 1 root root 56012 Feb 13 11:10 vntrseek.pl -rw-r----- 1 root root 2428 Feb 13 09:08 vs.cnf -rw-r----- 1 root root 2428 Feb 13 10:09 vs.cnf.new

On Tue, Feb 20, 2018 at 8:27 AM, Jerome Yesavage yesavage@stanford.edu wrote:

Will do this but have to transfer this mail over to the machine to copy properly.

From: Yozen Hernandez [mailto:notifications@github.com] Sent: Saturday, February 17, 2018 11:13 AM To: yzhernand/VNTRseek VNTRseek@noreply.github.com Cc: jerryyyyy yesavage@stanford.edu; Author <author@noreply.github.com

Subject: Re: [yzhernand/VNTRseek] Kill Received (#7)

Hi Jerry, could you please send the output of the following command: ls -l /usr/local/vntrseek1.09.1 and attach the install_manifest.txt file from the source build directory. I just want to check if the appropriate files were installed.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/yzhernand/VNTRseek/issues/7#issuecomment-366464349, or mute the thread https://github.com/notifications/unsubscribe-auth/AfoES5kc8L6_eqii1BqBnGxhKd7Qr7V2ks5tVyS9gaJpZM4SD3Av .

/usr/local/vntrseek1.09.1/vs.cnf.new /usr/local/vntrseek1.09.1/HISTORY /usr/local/vntrseek1.09.1/COPYING /usr/local/vntrseek1.09.1/CLUSTER /usr/local/vntrseek1.09.1/README /usr/local/vntrseek1.09.1/RUNNING /usr/local/vntrseek1.09.1/eucledian.dst /usr/local/vntrseek1.09.1/pcr_dup.exe /usr/local/vntrseek1.09.1/redund.exe /usr/local/vntrseek1.09.1/flankalign.exe /usr/local/vntrseek1.09.1/join_clusters.exe /usr/local/vntrseek1.09.1/trf2proclu-ngs.exe /usr/local/vntrseek1.09.1/psearch.exe /usr/local/vntrseek1.09.1/refflankalign.exe /usr/local/vntrseek1.09.1/checkleb36.pl /usr/local/vntrseek1.09.1/extra_index.pl /usr/local/vntrseek1.09.1/ge7.pl /usr/local/vntrseek1.09.1/hist_flank_filter.pl /usr/local/vntrseek1.09.1/insert_reads.pl /usr/local/vntrseek1.09.1/map_dup.pl /usr/local/vntrseek1.09.1/pcr_dup.pl /usr/local/vntrseek1.09.1/print_gained_by_patsize.pl /usr/local/vntrseek1.09.1/print_spanned_reads.pl /usr/local/vntrseek1.09.1/print_unclustered.pl /usr/local/vntrseek1.09.1/print_unmapped.pl /usr/local/vntrseek1.09.1/produce_indist.pl /usr/local/vntrseek1.09.1/remove_sw_fpvntrs.pl /usr/local/vntrseek1.09.1/renumber.pl /usr/local/vntrseek1.09.1/run_assemblyreq.pl /usr/local/vntrseek1.09.1/run_edges.pl /usr/local/vntrseek1.09.1/run_flankcomp.pl /usr/local/vntrseek1.09.1/run_proclu.pl /usr/local/vntrseek1.09.1/run_rankflankmap.pl /usr/local/vntrseek1.09.1/run_trf.pl /usr/local/vntrseek1.09.1/run_variability.pl /usr/local/vntrseek1.09.1/setdbstats.pl /usr/local/vntrseek1.09.1/stats.pl /usr/local/vntrseek1.09.1/updaterefs.pl /usr/local/vntrseek1.09.1/vntrpipe_xml2sql.pl /usr/local/vntrseek1.09.1/lib/vutil.pm /usr/local/vntrseek1.09.1/lib/ProcInputReads.pm /usr/local/vntrseek1.09.1/vntrseek.pl /usr/local/vntrseek1.09.1/master_for_qstub_test.sh /usr/local/vntrseek1.09.1/master_for_qstub_test_advanced.sh /usr/local/vntrseek1.09.1/master_for_qsub_test_advanced_useDB.sh /usr/local/vntrseek1.09.1/qsub_test.sh /usr/local/vntrseek1.09.1/qsub_test_advanced.sh /usr/local/vntrseek1.09.1/qsub_test_advanced_useDB.sh /usr/local/vntrseek1.09.1/sim_sw_reads.sh

[yzhernand]

Hi Jerry, apologies for taking to long to get back to you. I think I've sorted things out on my end finding a few more bugs, and I think I've figured out your issue.

At the end of out wiki page on installing VNTRseek I note that the vs.cnf file should have group ownership changed so that certain users can read it. The current wording is probably misleading so I will change that.

In your case, please try changing the group of /usr/local/vntrseek1.09.1/vs.cnf to a group which your uses is a member of, preferably some meaningful group name. Then make sure group read permissions are enabled for that file. I'd also suggest you remove the vs.cnf file which was created in your home directory for this run as it might not have initialized properly.

Then pull the latest commit I've made, make, reinstall, and initialize your run like you did the first time. Let me know how it goes.

[yzhernand]

Hi Jerry, just checking in to see if this is still an issue for you.

[jerryyyyy]

Hi, have done nothing on this since the last e-mail. I wonder if I should try again. The repeats at the 5-HTP promoter are very very complex. We have been looking at the actual sequences….

On the other hand, I have been running other software out of the VMWare VM using the new Windows 10 Ubuntu shell and it is working out much better. Perhaps I should give that a try. [I will do that when the latest run finishes... probably in day or so]. Two groups have published some SNPs in LD to try to impute the L/S repeats.

[jerryyyyy]

Reporting back it does seem to run in the new Ubuntu Linux instance using the compiled version. I just (not easy) need to set up the file for the pipeline.....

[yzhernand]

Yes this is a sore point in getting started, and it's something I'm addressing in the next version. But I'm glad to hear that it seems to work so far.

[jerryyyyy]

Well, I was a bit too optimistic. I have gotten a long way further but am having a lot of trouble with DBI and DBD... I have tried multiple ways to install and always run into one failure or another 2h effort). Any suggestions as to an easy method to get these installed. Running on an instance of Ubuntu.

Update, I have gotten DBI on... still DBD error on compilation.

Yes! sudo apt-get install libdbd-mysql-perl

Installed! Now, let's get it to work!

[jerryyyyy]

Yes, it is compiled and running using your test data, but when I run it I get this error:

yesavage@SLE-5B4-421-H5T:/mnt/e/Kaci1$ vntrseek 0 19 --dbsuffix run1 --server MIRECC --nprocesses 8 --html_dir /home/yesavage/ww --fasta_dir /mnt/e/Kaci1/watsontest/ --output_root /home/yesavage --tmpdir /tmp Using config file '/home/yesavage/run1.vs.cnf'... File 't26__.leb36' not found! at /usr/local/bin/vntrseek line 385.

[yzhernand]

If you are following the instructions for the example, you will need the prepared hg19 data files from our downloads page and install the files as described in our GitHub wiki.

For your own reference sets, you can use TRDB to generate the input for you.

[jerryyyyy]

Yes, I followed that wiki and downloaded the hg19 files, decompressed them and put then in the directory that is in the command line argument.

--fasta_dir /mnt/e/Kaci1/watsontest/

I also decompressed the .gz file and it is not the same name as was in your wiki, i.e. not "samplereads" rather is is "fast_Watson_selected":

yesavage@SLE-5B4-421-H5T:/mnt/e/Kaci1/watsontest$ ls -l total 16 -rwxrwxrwx 1 root root 11310 Mar 22 11:57 fasta_Watson_selected -rwxrwxrwx 1 root root 2684 Mar 22 10:43 fasta_Watson_selected.gz

yesavage@SLE-5B4-421-H5T:/mnt/e/Kaci1/watsontest$ vntrseek 0 19 --dbsuffix run1 --server MIRECC --nprocesses 8 --html_dir /home/yesavage/ww --fasta_dir /mnt/e/Kaci1/watsontest/ --output_root /home/yesavage --tmpdir /tmp Using config file '/home/yesavage/run1.vs.cnf'... File 't26__.leb36' not found! at /usr/local/bin/vntrseek line 385.

[yzhernand]

Sorry, they should go in the install directory. So if VNTRseek is installed in /usr/local/vntrseek1.09.1 then that's where those files should go. The fasta dir option is only for setting the location of the fasta/fastq files.

[jerryyyyy]

I looked at line 385 and it was a reference to $reference_file and in the code that was blank: "". Not sure it will find it.

You are saying move the Watson folder over to the install location in /usr/local/vntrseek1.09.1... or is this the hg19 set of files? I guess it must be the hg19 files?

[yzhernand]

Yes, the hg19 files. That setting in the vntrseek file is just the initialization line. It gets set later when reading the config file.

[jerryyyyy]

Thanks, that got it to run. Now I get: Can't connect to local MySQL server through socket.... Will read on that. Hope the mysql server is running. Looks like not... I remember this from the dark past and will try to find a solution.

We are almost there.

[jerryyyyy]

Got past the server and it is running to step 3:

Executing step #3 (eliminating cyclic redundancies)...

redund.exe: Input File sorted by minimum representation. Now call this program again without -s switch to remove redundancy.

Opening files... /home/yesavage/vntr_run1/reference//start2.leb36

230306 profiles read, 224584 profiles marked nonredundant. (time: 3 seconds)

Opening files... /home/yesavage/vntr_run1/data_out//0-0.leb36.renumbered

54 profiles read, 24 profiles marked nonredundant. (time: 0 seconds)

setting additional statistics... Using config file '/home/yesavage/run1.vs.cnf'... DBD::mysql::st execute failed: Data truncated for column 'NUMBER_REF_TRS' at row 1 at /usr/local/vntrseek1.08/vntr/vutil.pm line 283, line 1. Couldn't execute statement: Data truncated for column 'NUMBER_REF_TRS' at row 1 at /usr/local/vntrseek1.08/vntr/vutil.pm line 283, line 1. command exited with value 255 at /usr/local/bin/vntrseek line 810.

I tried adding:

innodb_buffer_pool_size=1G innodb_additional_mem_pool_size=20M

to the my.cfn file and got the same result...

I think we are very close....

[jerryyyyy]

1. Line 283 is the last one below.

Seems the database must be open and mysql is working otherwise the statements above would bomb.

my $dbh = DBI->connect("DBI:mysql:$DBNAME;host=$HOST", "$LOGIN", "$PASS" ) || die "Could not connect to database: $DBI::errstr";

my $sth = $dbh->prepare("UPDATE stats SET $NAME=?") or die "Couldn't prepare statement: " . $dbh->errstr;

$sth->execute($VALUE) # Execute the query or die "Couldn't execute statement: " . $sth->errstr;

2. I tried adding the innod... statements and it is not clear where to add them. I cannot find the my.cnf file with the instructions... the ones in my mysql build do not have [mysqld]:

IMPORTANT: for correct execution, please add these lines to the [mysqld] section of the my.cnf file and restart the mysql process:

innodb_buffer_pool_size=1G innodb_additional_mem_pool_size=20M

3. I found what appears to be the relevant file in /etc/mysql/mysql.config.d mysqld.cnf, but adding those lines did not change the errors.

[yzhernand]

Hi Jerry,

Sorry for the delay. Lots of things to address this time, so I'll try to get them all!

Using config file '/home/yesavage/run1.vs.cnf'... DBD::mysql::st execute failed: Data truncated for column 'NUMBER_REF_TRS' at row 1 at /usr/local/vntrseek1.08/vntr/vutil.pm line 283, line 1. Couldn't execute statement: Data truncated for column 'NUMBER_REF_TRS' at row 1 at /usr/local/vntrseek1.08/vntr/vutil.pm line 283, line 1. command exited with value 255 at /usr/local/bin/vntrseek line 810.

Interesting. I should have fixed this bug before I asked you to update your install. Check again that you have the latest version. I think I may have made another change since we last talked, but it shouldn't have affected anything that you've done so far. The fix was in commit 6e378fcc4425e5f775f68dde59001a184ddceef3.

I found what appears to be the relevant file in /etc/mysql/mysql.config.d/mysqld.cnf, but adding those lines did not change the errors.

It looks like since the documentation was written, the config structure for mysql in debian (and maybe others) has changed a bit.

You are correct in editing that file, as it does have a [mysqld] section. Though I think that the Debian/Ubuntu developers intend package maintainers and installers to make their own file which goes into /etc/mysql/mysql.config.d/ with the settings they need. So an alternative would be to make a new file in that directory, say vntrseek.cnf, with a [mysqld] section with those lines.

In any case, version 1.10 (maybe we'll call this 2.00) will do away with mysql for zero database configuration on the user's part. We'll use sqlite instead.

Thanks for looking into the code yourself to see if you could find out what's happening! We are aware that it's a bit convoluted in places.

[jerryyyyy]

Thanks, Not sure i will get much done remotely over the weekend, but I made the modification of the config file... indeed the structure looked different.
On those two files that are changed, can I just replace them in my current build? I am not experienced with Perl and they appear to be scripts and not compiled.
JY

[yzhernand]

Yes, Perl is a scripting language so just copying the files or making the same changes should be fine. However, I would recommend pulling the latest changes and reinstalling.

[jerryyyyy]

Hi, Perhaps I am not fully sure how these changes are reflected in github but the "latest" commit does not have the same identifier as listed above. So not clear how to download the correct whole bundle to reinstall.
I do hate to reinstall since I have made all my corrections in many settings files including getting mysql to work. I'll try to replace the files if I can download the two in question. Thx, JY

[yzhernand]

Hi Jerry, the latest commit would be different from the one I mentioned since there were other changes since then.

Anyway, all that would be required is a git pull in the source directory, and then make install in the build directory as before. Your edited files should be unaffected unless you were making changes to the source files. The MySQL and run configuration files should be fine.

[jerryyyyy]

Thanks,

Sorry to be so concrete, but I see that what I had installed was V 1.08, and I see the changes you noted were in V 1.09 but it looks like the latest is V 1.10.

Can you give me a link to the tar.gz file that would be the most likely to be stable with the crucial changes included?

Thanks again,

JY

[yzhernand]

This should work. It's the zip of the latest commit to the v1.09_maintenance branch. https://github.com/yzhernand/VNTRseek/archive/v1.09_maintenance.zip

[jerryyyyy]

Thanks, we are back up and running with that set of files, however, I seem to have a few wayward settings. I moved the reference genome files to the right place and restarted the mysql server.

1. The globalconf? Settings

2. Also line 186 below... not clear if this is a server error. I looked at the line and not an obvious issue.

Thanks again,

JY

yesavage@SLE-5B4-421-H5T:/usr/local/vntrseek1.08$ vntrseek 0 19 --dbsuffix run1 --server MIRECC --nprocesses 8 --html_dir /home/yesavage/ww --fasta_dir /mnt/e/Kaci1/watsontest Could not read global config

Executing step #0 (creating MySQL database)... Warning: Failed to create data directory! mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Warning: Failed to create html directory!

Warning: Failed to create output directory! DBI connect('VNTRPIPE_run1;host=localhost','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/local/vntrseek1.09.1/lib/vutil.pm line 186. Could not connect to database: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/local/vntrseek1.09.1/lib/vutil.pm line 186.

[yzhernand]

Make sure your user has read access to the /usr/local/vntrseek1.09.1 directory and its contents.

[jerryyyyy]

Yes, the permissions are OK and I moved over the old configuration file with the relevant mysql passwords. It looks like (another) mysql issue:

yesavage@SLE-5B4-421-H5T:/usr/local/vntrseek1.09.1$ sudo service mysql start

• Starting MySQL database server mysqld No directory, logging in with HOME=/ [fail] yesavage@SLE-5B4-421-H5T:/usr/local/vntrseek1.09.1$ mysql ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I wish I could look back and see what I did a week ago to get it running right.... need to keep this histories.

[yzhernand]

I'm not sure but maybe something went wrong when installing MySQL? Perhaps this like might help: https://askubuntu.com/questions/998224/mysql-ubuntu-16-04-not-starting

[jerryyyyy]

Well I uninstalled mysql and mysqld and still the server does not run right. Finally I found this:

https://www.zyxware.com/articles/5539/solved-cant-connect-to-local-mysql-server-through-socket-var-run-mysqld-mysqld-sock

See below for solution.

[jerryyyyy]

OK, it seems to be running.... !!!!!!!!!!!!!!!!!!!

It was not an sql problem but in the command line I had to renumber Run1 to Run2 etc.... would not overwrite prior data (I believe this invoked the security program).

Then it seems to need a program called: seqtk

Looks like it ran to completion in about 1h!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Now I need to look at my data... suspect I cannot use the whole genome?

As well, I better see how I interpret the output.

Thanks

JY

[yzhernand]

I'm glad it seems to have run, but that run time of 1 hour is a bit suspicious. Do you not have seqtk installed? If not, it may not have run the TRF step, so I am cautiously optimistic.

As for the output, a VCF file is produced in the output directory you specified, under your_output_dir/vntr_runname/data_out_clean/results

[yzhernand]

Nevermind my caution. I saw your comment before your edit. You ran the sample reads, so that run time is fine, if not very slow. I think it usually takes me about 10 minutes so that may be an issue.

[jerryyyyy]

Thanks, tomorrow I will have to figure out how to make sense of results, but tried to run on our own data and it does not seem to recognize my fastq.gz file:

fastq_21.gz

yesavage@SLE-5B4-421-H5T:~$ vntrseek 0 19 --dbsuffix run9_Kaci --server MIRECC --nprocesses 8 --html_dir /home/yesavage/ww --fasta_dir /mnt/h/Kaci2/Test --tmpdir / tmp --output_root /home/yesavage Could not read run config (harmless if this is a new run)

Executing step #0 (creating MySQL database)...mysql: [Warning] Using a password on the command line interface can be insecure.

Warning: Failed to create html directory! done!

Executing step #1 (searching for tandem repeats in reads, producing profiles and sorting)... 1 supported files (fastq format, compressed as gzip) found in /mnt/h/Kaci2/Test Will use 8 processes Processing file fastq_21.gz seq: invalid option -- 'S'

Usage: seqtk seq [options] |

Options: -q INT mask bases with quality lower than INT [0] -n CHAR masked bases converted to CHAR; 0 for lowercase [0] -l INT number of residues per line; 0 for 2^32-1 [0] -Q INT quality shift: ASCII-INT gives base quality [33] -s INT random seed (effective with -f) [11] -f FLOAT sample FLOAT fraction of sequences [1] -M FILE mask regions in BED or name list FILE [null] -L INT drop sequences with length shorter than INT [0] -c mask complement region (effective with -M) -r reverse complement -A force FASTA output (discard quality) -C drop comments at the header lines

gzip: stdout: Broken pipe Running child, current_file = 0 (fastq_21.gz)...

Error: Error while loading sequenceCould not read TRF header. Aborting. Process 0 did not find any VNTRs. Removing leb36 file... Processing complete -- processed 8 file(s). done!

Executing step #2 (reassigning IDs to repeats)...0 LEB36 files found in directory /home/yesavage/vntr_run9_Kaci/data_out/ 0 records processed Only changing the sign Not using indexes 1 LEB36 files found in directory /home/yesavage/vntr_run9_Kaci/reference/ 230306 records processed done!

Executing step #3 (eliminating cyclic redundancies)...

redund.exe: Input File sorted by minimum representation. Now call this program again without -s switch to remove redundancy.

Opening files... /home/yesavage/vntr_run9_Kaci/reference//start2.leb36

230306 profiles read, 224584 profiles marked nonredundant. (time: 2 seconds)

Opening files...

ERROR: No input files found in the directory.

command exited with value 1 at /usr/local/bin/vntrseek line 867. yesavage@SLE-5B4-421-H5T:~$

[yzhernand]

Hmm, it failed on seqtk. Are you using the latest release?

[jerryyyyy]

I installed with: sudo apt-get install seqtk; I had trouble with the github version.

It is Version: 1.0-r31

I just reran the scrip using the same one that ran on your test data after moving my file to the old directory (to minimize changes) and got the same error.

The version I am using does not seem to have the option "S", but I am not sure why that would be invoked on my data and not your data. Line 867 is the last "die" command.

system( "./redund.exe $read_profiles_folder/ $read_profiles_folder_clean/allreads.leb36 -i" ); if ( $? == -1 ) { SetError( $STEP, "command failed: $!", -1 ); die "command failed: $!\n"; } else { my $rc = ( $? >> 8 ); if ( 0 != $rc ) { SetError( $STEP, "calling redund.exe on read profiles folder failed", $rc ); die "command exited with value $rc"; } }

[yzhernand]

I've just made another push of the code. This commit includes the version of seqtk VNTRseek needs, and it is build along with the other binaries we use. I also added some fixes for BAM files, if that is relevant to you.

Please let me know if that works for you. Here is a direct link in case you need it: https://github.com/yzhernand/VNTRseek/archive/v1.09_maintenance.zip

[jerryyyyy]

Well, back to mysql:

CMake Error at FindDeps.cmake:59 (MESSAGE): Error: Unable to detect your version of mysql. Please ensure that mysql_config is in your PATH. Call Stack (most recent call first): CMakeLists.txt:49 (MYSQL_REQ_VER)

***did not see this error before

yesavage@SLE-5B4-421-H5T:/mnt/e/Kaci1/VNTRseek-1.09_maintenance1/VNTRseek-1.09_maintenance/build$ which mysql /usr/bin/mysql yesavage@SLE-5B4-421-H5T:/mnt/e/Kaci1/VNTRseek-1.09_maintenance1/VNTRseek-1.09_maintenance/build$ which mysql_config yesavage@SLE-5B4-421-H5T:/mnt/e/Kaci1/VNTRseek-1.09_maintenance1/VNTRseek-1.09_maintenance/build$

[yzhernand]

It's bizarre that you are getting that error now. Have some packages been removed accidentally? Make sure you have both libmysqlclient-dev and libdbd-mysql-perl installed

[jerryyyyy]

I found this and it compiled:

sudo apt install libmysqlclient-dev...

Will try to run it.

[jerryyyyy]

It found my file but I still get the "S" error:

Executing step #1 (searching for tandem repeats in reads, producing profiles and sorting)... 1 supported files (fastq format, compressed as gzip) found in /mnt/e/Kaci1/watsontest Will use 8 processes Processing file fastq_21.gz seq: invalid option -- 'S'

I will try it with your old test file.