planemo test not handling tool-data/example.loc.sample

[peterjc]

See also #96 for linting. Since switching to planemo test, I have been having problems with planemo test on TravisCI.

        <param name="module" type="select" display="radio" label="Classification module">
            <options from_file="effectiveT3.loc">
                <column name="value" index="0"/>
                <column name="name" index="1"/>
                <column name="path" index="2"/>
            </options>
        </param>

I provide file tool-data/effectiveT3.loc.sample and expect Galaxy or planemo test to auto-generate tool-data/effectiveT3.loc but it does not, resulting in a test failure, e.g.

https://travis-ci.org/peterjc/pico_galaxy/jobs/150084912

======================================================================
ERROR: Effective T3 ( effectiveT3 ) > Test-1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/peterjc/pico_galaxy/galaxy-dev/test/functional/test_toolbox.py", line 299, in test_tool
    self.do_it( td )
  File "/home/travis/build/peterjc/pico_galaxy/galaxy-dev/test/functional/test_toolbox.py", line 60, in do_it
    raise e
RunToolException: Error creating a job for these tool inputs - Parameter module requires a value, but has no legal values defined.
-------------------- >> begin captured logging << --------------------

Running locally, planemo test tools/effectiveT3/effectiveT3.xml works when it uses the local Galaxy install's tool-data/effectiveT3.loc - but it will successful regenerate it from the sample if I remove it:

$ mv /mnt/galaxy/galaxy-central/tool-data/effectiveT3.loc /mnt/galaxy/galaxy-central/tool-data/effectiveT3.loc.bak
$ planemo test tools/effectiveT3/effectiveT3.xml
...
Initializing tool-data/effectiveT3.loc from effectiveT3.loc.sample...
...
All 3 test(s) executed passed.
effectiveT3[0]: passed
effectiveT3[1]: passed
effectiveT3[2]: passed
$ diff /mnt/galaxy/galaxy-central/tool-data/effectiveT3.loc.bak /mnt/galaxy/galaxy-central/tool-data/effectiveT3.loc
$ planemo --version
planemo, version 0.24.0.dev0

This is running an old planemo 0.24.0dev0 and old Galaxy release_15.10, so the problem on TravisCI could be a regression in either?

I can workaround this on TravisCI using https://github.com/peterjc/pico_galaxy/commit/da75fc4ab1279493acb9b1b8a29910e027ba0e31 (now on master branch), see https://travis-ci.org/peterjc/pico_galaxy/builds/151178750 (test branch) and https://travis-ci.org/peterjc/pico_galaxy/builds/151184006 (master branch)

$ git show
commit da75fc4ab1279493acb9b1b8a29910e027ba0e31
Author: peterjc <p.j.a.cock@googlemail.com>
Date:   Wed Aug 10 11:49:32 2016 +0100

    Explicitly generate tool-data/*.loc from *.loc.sample

diff --git a/.travis.yml b/.travis.yml
index 1570b92..cfc644a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -125,6 +125,8 @@ install:
  - for i in $( ls ${TRAVIS_BUILD_DIR}/tools/ ); do ln -s ${TRAVIS_BUILD_DIR}/tools/$i/ tools/$i ; done
  - ln -s ${TRAVIS_BUILD_DIR}/.travis.datatypes_conf.xml datatypes_conf.xml
  - cp ${TRAVIS_BUILD_DIR}/test-data/* test-data/
+# Explicitly use sample files to generate default *.loc files
+ - for f in ${TRAVIS_BUILD_DIR}/tool-data/*.loc.sample; do cp "$f" "${f%.sample}"; done
  - cp ${TRAVIS_BUILD_DIR}/tool-data/* tool-data/
  - bash scripts/common_startup.sh --no-create-venv --dev-wheels

[jmchilton]

For what it is worth - I don't think this ever worked with planemo. A fix for this is needed but I'd call it an enhancement more than a bug. My proposed solution was to build test versions of these .loc files - http://planemo.readthedocs.io/en/latest/writing_how_do_i.html#test-index-loc-data. I have recently realized this needs to be revised though for files with default data included - my original attempt at this was conceived of as a way to test bwa for instance where default non-test entries really wouldn't make sense.

So the logic should be something like - if a test variant is available, use it - otherwise use the sample.

[peterjc]

That is what I think Galaxy's test framework did - test-data/example.loc would be the first choice, failing that using tool-data/example.loc.sample (which would become tool-data/example.loc on installation).

[peterjc]

Not sure if all of this was needed, but after removing all my manual TravisCI tool installation in https://github.com/peterjc/pico_galaxy/commit/0df534215fc9690cef04d9db3c77d78a6c5bdca8 (and just letting planemo do it), this commit was needed to get my tests to pass via TravisCI: https://github.com/peterjc/pico_galaxy/commit/f84c1f773b07fa1eeba0d0a432ed4c1d899740b1

[peterjc]

Note that my workaround of manually copying *.loc files into place is only possible if you have a local Galaxy instance for planemo test --galaxy_root ...

This is impossible when Planemo itself creates the Galaxy instance for testing, e.g. with planemo test --install_galaxy ... or planemo test --galaxy_branch ... --galaxy_source ... as currently used in https://github.com/galaxyproject/tools-iuc/blob/master/.travis.yml#L45 and https://github.com/galaxyproject/tools-devteam/blob/master/.travis.yml#L44