Unable to run autotune on example data or my data

[quinn-ca]

Hello,

I successfully ran countLociImmanc.sh on both my data and the example data (spd_ARU+182+BOU.str.immanc), resulting in 28816 and 307 loci, respectively. I noticed the code in the readme specified 4819 loci for the example data, so maybe there is some discrepancy? However, countLociImmanc returned the number of loci I would expect for my data.

When I run BA3-SNPS-autotune on my data and the example data, I get a SystemExit Error. The code I ran for my data was BA3-SNPS-autotune.py -i autosomal_genstr_filtered_maxMeanDP49_rmRelated.immanc -l 28816 -b 10000 -g 100000 -o autotune_BA3_autosomal_genstr_filtered_maxMeanDP49_rmRelated -r 10 -s 100

Output for my data:

Running repetition 1 of 10
BA3-SNPS -F autosomal_genstr_filtered_maxMeanDP49_rmRelated.immanc -l 28816 -i 100000 -s 100 -b 10000 -m 0.1000000000000000055511151231257827021181583404541015625 -a 0.1000000000000000055511151231257827021181583404541015625 -f 0.1000000000000000055511151231257827021181583404541015625 -o autotune_BA3_autosomal_genstr_filtered_maxMeanDP49_rmRelated -t -v -g -u
Non-zero exit status:
137
Unexpected error:
(<class 'SystemExit'>, SystemExit(), <traceback object at 0x7f0cbfbe3240>)

Output for the example data:

Running repetition 1 of 5
BA3-SNPS -F spd_ARU+182+BOU.str.immanc -l 4819 -i 10000 -s 10 -b 1000 -m 0.1000000000000000055511151231257827021181583404541015625 -a 0.1000000000000000055511151231257827021181583404541015625 -f 0.1000000000000000055511151231257827021181583404541015625 -o output.txt -t -v -g -u
Non-zero exit status:
1
Unexpected error:
(<class 'SystemExit'>, SystemExit(), <traceback object at 0x7ff2c35fb1c0>)

The stdout file from my data returned (note: "JI" is the designation for one of my populations):

Made new Indiv object
Going to read input file
Setting alleles
JI      0
Read input file

Input file: autosomal_genstr_filtered_maxMeanDP49_rmRelated.immanc
Output file: autotune_BA3_autosomal_genstr_filtered_maxMeanDP49_rmRelated
Individuals: 124 Populations: 4 Loci: 28816 Missing genotypes: 26859

Locus:(Number of Alleles)
locus 0:2 
...
locus28656:2 locus2865

The stdout file from the example data returned:

Made new Indiv object
Going to read input file
Setting alleles

I ran BA3-SNPS with the default parameters for the example data to see if that would inform any errors I was getting. BA3-SNPS -v -F spd_ARU+182+BOU.str.immanc -l 4819 and BA3-SNPS -v -F spd_ARU+182+BOU.str.immanc -l 4819 both returned the same output (and a blank BA3out.txt file):

Made new Indiv object
Going to read input file
Setting alleles
Error: Incorrect number of entries in line>>  >>of input file spd_ARU+182+BOU.str.immanc

When I ran BA3-SNPS with my data, the output was:

Made new Indiv object
Going to read input file
Setting alleles
JI  0
Read input file

Input file: autosomal_genstr_filtered_maxMeanDP49_rmRelated.immanc
Output file: BA3out.txt
Individuals: 124 Populations: 4 Loci: 28816 Missing genotypes: 26859

Locus:(Number of Alleles)

locus0:2 locus1:2
....
locus28815:2

Killed

I sincerely appreciate any help you can provide!

[stevemussmann]

I'm not sure what is happening here. The error you are getting looks like it is being thrown from BA3-SNPs not reading the input files correctly for some reason. I downloaded the docker container to a new computer on which it had never been installed, and ran the example file with the example command in the documentation without issue. the countLociImmanc script also returned the correct number of loci for the file (4819).

I'm not sure what is wrong, but might have a wild guess of something to try. Could you run ls -al on the directory containing your input files and post the output of that command here?

[quinn-ca]

Hmm that's very strange! When I run ls -al, the output is:

ba3user@0433176e8bb3:/app/data$ ls -al
total 132056
drwxr-xr-x 14 ba3user ba3user       448 Jun 13 17:40 .
drwxr-xr-x  1 ba3user root         4096 Feb 19  2022 ..
-rw-r--r--  1 ba3user ba3user         0 Jun 13 17:27 BA3out.txt
-rw-r--r--  1 ba3user ba3user 129216472 Jun 12 17:20 autosomal_genstr_filtered_maxMeanDP49_rmRelated.immanc
-rw-r--r--  1 ba3user ba3user       906 Jun 13 17:38 autosomal_genstr_filtered_maxMeanDP49_rmRelated.immanc.1.stdout
-rw-r--r--  1 ba3user ba3user         0 Jun 13 17:36 autosomal_genstr_filtered_maxMeanDP49_rmRelated.indiv.txt
-rw-r--r--  1 ba3user ba3user         0 Jun 13 17:36 autosomal_genstr_filtered_maxMeanDP49_rmRelated.trace.txt
-rw-r--r--  1 ba3user ba3user         0 Jun 13 17:36 autotune_BA3_autosomal_genstr_filtered_maxMeanDP49_rmRelated
-rw-r--r--  1 ba3user ba3user         0 Jun 13 17:22 autotune_BA3_spd_ARU+182+BOU.str.immanc
-rw-r--r--  1 ba3user ba3user         0 Jun 13 16:40 output.txt
-rw-r--r--  1 ba3user ba3user    142687 Jun 13 16:40 spd_ARU+182+BOU.str.immanc
-rw-r--r--  1 ba3user ba3user       901 Jun 13 17:22 spd_ARU+182+BOU.str.immanc.1.stdout
-rw-r--r--  1 ba3user ba3user         0 Jun 13 17:22 spd_ARU+182+BOU.str.indiv.txt
-rw-r--r--  1 ba3user ba3user         0 Jun 13 17:22 spd_ARU+182+BOU.str.trace.txt

[stevemussmann]

Ok, I thought it could perhaps be a permissions issue but that seems unlikely since you have read access on all files.

Just curious what kind of computer this is being run on. Is it a Mac?

[quinn-ca]

Yes, I am running this on a mac, originally I hoped to install it on our lab's remote linux cluster, but that seemed too complicated given my restricted permissions and the remote connection.

Maybe part of the problem is how I re-started Docker after it had closed the first time? I opened the Docker desktop app and then ran docker container run -v /Users/quinncarvey/bayesass_snps/data/:/app/data --rm -it mussmann/ba3snps:1.0 /bin/bash in my terminal and that seems to have re-activated the container, but perhaps it did so incorrectly?

When I look at in the Docker GUI, I can't run the image. It says: "Critical vulnerabilities were found in this image". Perhaps of note, this is under "images" and not under "containers". The status is "in use". See attached pictures.

[stevemussmann]

Is it an M1 or M2 mac? I had to do some work on one of my other Docker containers to get it functioning properly on my M1 Macbook Pro. I thought I did that with this container as well, but perhaps I am remembering incorrectly.

[stevemussmann]

Just compared some Docker container build scripts and I'm pretty sure this is the issue if you have an M1 or M2 Mac (I did my test earlier on a machine running Ubuntu). I haven't used this container since I got my new laptop. I need to rebuild the Docker container and push it to Docker hub. I'll do that as soon as I can but it might take a couple days.

[quinn-ca]

Thanks for the update, I really appreciate it! I'm working on an older mac (2014), so it has an Intel core processor.

[stevemussmann]

I remembered after this conversation that the problem I had with Docker on Macs was building the containers, not running them. I downloaded the container as-is and it ran on my M1 Mac (with the latest Docker version) just fine. The countLociImmanc script produced the expected output on the example file, and the autotune program ran just fine with the example file (if a little slow since it was in a compatibility mode).

I'm currently at a loss for why it's not working for you, unless it's something specific to Intel-based Macs. Do you have any ability to try the container on a different computer? I can keep looking into it, but I don't have any great ideas coming to mind at the moment.

[quinn-ca]

I'm also having problems running the container on my Windows pc with ubuntu. I'm heading into the field through the end of the month, but I'll reach out to IT here at my university to see if I can get the container working with their help. If I have further questions, I'll reach out. I really appreciate your help!

[stevemussmann]

Sorry for the delayed response; I was at a conference.

I assume by "Windows pc with ubuntu" you are trying to run under the Windows Subsystem for Linux? If so, I've hit problems with running Docker containers under that setup. Sometimes they work; other times they do not. I think some of the variability relates to the multiple methods in which WSL can now be configured on Windows, but I haven't played around with it much yet to determine the exact conditions that create problems.

[quinn-ca]

Hi Steve,

I've been away and focused on other tasks, but am back at it, trying to get this to run! I worked with my IT department, and it seems like Docker itself is running okay, although they aren't experienced with this software. When I run runDocker.sh, it activates the container (my user changes to 'ba3user@#####'). Unfortunately, I'm getting all the same errors (with the example data and my data). Of note, my mac is old, so it's running Big Sur 11.7.9. I tried a manual installation on my mac, but I couldn't get it to work (it exceeded my gsl/c++ library compatibility knowledge).

Next, I tried a manual installation on ubuntu. I ran countLociImmanc.sh, autotune, and BA3-SNPS on the example file, and these all seemed to work. However, autotune and BA3-SNPs aren't working with my data. So (on ubuntu at least), it seems that the problem may be with my input file. I converted my .structure file output from Stacks to .immanc with the stacksStr2immanc.pl converter, and it seemed to be successful (attached).

When I ran autotune on my data file (./BA3-SNPS-autotune.py -i autosomal_genstr_filetered_maxMeanDP49_rmRelated.immanc -l 28816), I got the error: Non-zero exit status: 137 Unexpected Error: (<class 'SystemExit'>, SystemExit (), <traceback object at 0x7fd9794f8089>). When I ran BA3-SNPS (./BA3-SNPS -F autosomal_gentr_filtered_maxMeanDP49_rmRelated.immanc -l 28816), it runs for a few seconds, then says killed. autosomal_genstr_filtered_maxMeanDP49_rmRelated.immanc.zip

I'd appreciate any troubleshooting help you can provide!

[quinn-ca]

Just an update trying to troubleshoot my input file:

I edited the sample names of the example data file to see if having special characters (e.g., '_') or them being a longer text string was the problem. However, autotune still runs when the sample names have been edited. None of my loci have all missing data, which is noted in the manual as being a problem. There are no headings in my data file, the sample IDs/popIDs and alleles are in the correct locations. The alleles are coded correctly (0-4). The file is the correct length (n*l). I re-ran the stacks structure --> immanc pipeline and also ran a roundabout vcf-->phylip-->structure-->immanc pipeline. The files are the exact same across conversions and I still get the same Non-zero exit status: 137 error.

I investigated exit status: 137 and found: "A 137 code is issued when a process is terminated externally because of its memory consumption". This is plausible, my data file is 26 times larger than the example data. Could this be the problem? I'm running the program on a standard desktop PC with 8 GB RAM with an i5-8500 Intel Core CPU processor. I have access to a linux cluster, but installation there is even more difficult due to lack of sudo permissions.

[stevemussmann]

Apologies for the delayed response - github apparently decided on its own that I didn't need to be updated on issues that people have opened so I just saw your last two responses today.

Anyways, I think there is a very good chance you are exceeding the memory limits in your computer from what you have described with the 137 exit status.

[stevemussmann]

I'm closing this issue since it seems to no longer be active. Feel free to re-open, or open a new issue if you experience problems.

-Steve