FAIL: test_common-uniq-inds.bash

[pcarbo]

Hi Tim,

I'm using eqtlbma 1.3.1. I successfully ran ./configure and make (on a compute cluster with Scientific Linux 6). However, when I ran make test, this is the output I got (in test-suite.log):

   eqtlbma 1.3.1: tests/test-suite.log
=========================================

# TOTAL: 11
# PASS:  10
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test_common-uniq-inds.bash
================================

START test_common-uniq-inds.bash 2016-10-10 17:38:26
cmd-line: ./test_common-uniq-inds.bash
temp dir: /project/mstephens/eqtlbma-1.3.1/tests/tmp_test_30612
simulate data and calculate expected results ...
analyze data to get observed results ...
compare obs vs exp results ...

Further:

> cat test_common-uniq-inds.bash.log
START test_common-uniq-inds.bash 2016-10-10 17:38:26
cmd-line: ./test_common-uniq-inds.bash
temp dir: /project/mstephens/eqtlbma-1.3.1/tests/tmp_test_30612
simulate data and calculate expected results ...
analyze data to get observed results ...
compare obs vs exp results ...
file 'obs_bf_l10abfs_raw.txt.gz' has differences with exp

A quick scan of the two files, exp_bf_l10abfs_raw.txt.gz and obs_bf_l10abfs_raw.txt.gz, and the results are clearly not even close. I've attached the files from the test foilder so you can look at them yourself.

Any idea what might be the problem? Perhaps it has something to do with the simulated data being different (e.g., because the sequence of pseudorandom numbers is different)?

Thanks, Peter

tmp_test_30612.tar.gz

[timflutre]

(Everything works fine on my laptop with xUbuntu 14.04, R 3.3.1, MASS package 7.3.45, gcc 4.8.5, and GSL 1.16.)

This test is executed from the bash script test_common-uniq-inds.bash.

By comparing the log message and the bash script from line 124, I can say that the summary stats are fine, and that the problem arises when calculating the "raw" Bayes factors ("raw" because "not averaged over grid nor config").

Here is how I set up the tests. Each test has its own bash script, which does everything:

1. executes the R script functional_tests.R (which simulates data, writes them into files, performs the analyses labelled "expected");
2. executes the binaries from eqtlbma on the data simulated at the previous stage (which performs analyses labelled "observed");
3. compare exp versus obs.

The test we are interested in, test_common-uniq-inds.bash, looks at the scenario where some individuals are common to all subgroups, but some are not. You can see this via the options --ris and --mvlr given by the bash script to the R script on line 108.

Something must have changed in the C++ side or the R side. at this stage, I think something is more likely to have changed in the R side (from one version to a new one), no? (In any case, can you check your R version, as well as the MASS package, GSL and gcc versions?)

So let's look at the R script. It "looks like" a C program: when it is executed, it loads all the functions and, at the very bottom of the file, it executes the function "main". Inside this 1st function, the problem must come from "getResultsOnSimulatedData". Inside this 2nd function, the problem must come from "calcRawAbfsOnSimulatedData". Inside this 3rd function, the problem may come from the "else" statement on line 1317. At this stage, one now has to debug where it comes from. I would suggest starting by re-running the C++ code and the R code on a single gene-snp pair.

[pcarbo]

Tim, these are the versions of the software/libraries you and I are using:

        you     me
     ------  -----
gcc   4.8.5  4.8.5
gsl    1.16   1.16
R     3.3.1  3.2.1
MASS 7.3-45 7.3-45

So the only difference is the version of R. In your instructions, I don't see anywhere where you require a certain version of R. Have you tried running your tests with an earlier version of R (e.g., R 3.2)?

Aside from the tests, it appears that you only use R to summarize and visualize the results of eqtlbma. Is this correct? If so, it seems that requiring a specific version of R is only critical for the tests; that is, it seems that tutorial_eqtlbma.R and utils_eqtlbma.R are not necessary for correct usage of eqtlbma. Or are they?

[timflutre]

In the installation instructions, I wrote: "If you want to run the tests successfully with make check, you will need R >= 2.15 (notably for paste0)".

As far as I remember, make check always worked for me, whatever the computer environment I was using, but maybe I jumped from R 3.0 to R 3.3 directly.

R is used in eqtlbma at 3 places:

1. for testing (via functional_tests.R which requires MASS);
2. for the tutorial (via tutorial_eqtlbma.R which sources utils_eqtlbma.R);
3. for preprocessing/summary/visualization (via utils_eqtlbma.R in which a function requires MASS and another GenomicRanges from Bioconductor).

I always use paste0 instead of paste, thus requiring R >= 2.15. But I don't see what could cause a single test to fail, and not the others.

ps: to answer one of your previous question, in functional_tests.R, I always set the same seed at the beginning of function getSimulatedData on line 239.

[pcarbo]

You are right, Tim---I didn't see the sentence about requiring R >= 2.15 for make check.

I was able to run all the tests successfully, but only after using R 3.3.1 instead of R 3.2.1. So it seems that the final test (test_common-uniq-inds.bash) requires R version > 3.2.1. (I didn't try versions of R between 3.2.1 and 3.3.1 to find out which version breaks the test.) Should I update the Wiki with this info?

[timflutre]

Thanks Peter for finding this! I updated the wiki.

However, I didn't change the R code of functional_tests.R (lines 28-31) as, for most users, it may not be necessary to update their R version if the tests work well between 2.15 and 3.2.0, don't work for 3.2.1, but work again for >= 3.3.

[pcarbo]

Thanks, Tim!

Peter

On Thu, Oct 13, 2016 at 12:59 AM, Timothée Flutre notifications@github.com wrote:

Thanks Peter for finding this! I updated the wiki.

However, I didn't change the R code of functional_tests.R (lines 28 https://github.com/timflutre/eqtlbma/blob/master/tests/functional_tests.R#L28-31) as, for most users, it may not be necessary to update their R version if the tests work well between 2.15 and 3.2.0, don't work for 3.2.1, but work again for >= 3.3.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/timflutre/eqtlbma/issues/23#issuecomment-253421538, or mute the thread https://github.com/notifications/unsubscribe-auth/ABbyoChdrgrfiBVfMlwgch-kX0qeQqqQks5qzci3gaJpZM4KTv9J .