search

ga4gh / compliance

Compliance test suite for the APIs defined in ga4gh-schemas. RETIRED 2018-01-24
https://ga4gh.org
Apache License 2.0
10 stars 23 forks source link

simplify compliance install/run #36

Closed diekhans closed 9 years ago

diekhans commented 9 years ago

Compliance test should run out-of-the-box with minimal configuration of only server base URL for normal case.

From: Jerome Kelleher I've run the tests, and it does seem to be quite a powerful framework. However, it's much too complicated to run at the moment; it took me a good hour to get the directory structure right (why do we need a lib directory? What do log4j2.xml and antRunTests.xml do?) and get the tests running. Flexibility is great, but I wonder if things have gone a bit overboard here. However, this is something we should try to fix after we merge.

diekhans commented 9 years ago

make that, supply base URL on command line

wstidolph commented 9 years ago

The 'dist' module builds a zip file that unzips to a working structure, but that zip doesn't seem to be getting published right now - TravisCI was building that and pushing it as a "release" zip so it should be reasonable to turn back on. When you use that mechanism, the only configuration you should need to do is to set the target server URL as a property (or, if the server is at http://localhost:8000/v0.5.1 you don't even need to do that)

As to why the lib, etc.:

I put log4j2.xml, antRunTests.xml , etc in a lib dir rather than leaving them packaged in the jars so that they would be easily editable by a use who didn't want to rebuild/repackage the CTK. Also, the lib/ provides a place to put overriding test class implementations (in a PR I submitted)

hjellinek commented 9 years ago

The Travis config looks correct for automatically building and pushing the dist directory when a release is tagged. The command-line arg stuff (supplying the base URL and supplying the dataset ID) are both implemented. Can this issue be closed, Mark @diekhans?

wstidolph commented 9 years ago

Actually, I've had a problem with the Travis build, since it looks like TravisCI only has Maven 3.2.5 and someone added an Enforcer rule to a pom.xml which causes the Travis build to fail (and, nothing gets pushed).

On Wed, Sep 2, 2015 at 9:43 AM, Herb Jellinek notifications@github.com wrote:

The Travis config looks correct for automatically building and pushing the dist directory when a release is tagged. The command-line arg stuff (supplying the base URL and supplying the dataset ID) are both implemented. Can this issue be closed, Mark @diekhans https://github.com/diekhans?

— Reply to this email directly or view it on GitHub https://github.com/ga4gh/compliance/issues/36#issuecomment-137164706.

Wayne Stidolph | (831) 419 7146 | LinkedIn http://www.linkedin.com/in/waynestidolph | Google+ https://google.com/+WayneStidolph | twitter https://twitter.com/#!/wstidolph | personal blog http://waynedge.blogspot.com/

hjellinek commented 9 years ago

I did, in response to a user with Maven 3.1.x who found it was too old to build. Maven 3.3.3 was/is the current version and I didn't want to go to the trouble of figuring out the minimum version that would work. I will see if 3.2.5 is recent enough (I'm sure it is) and change the rule if so.

wstidolph commented 9 years ago

Thx. Back to the original (fair) complaint - it should not take an hour to get started! And if someone gets to the right entry point (the cli zip, probably) then it doesn't. But the compliance toolkit does have inherent complexity due to supporting several use cases for how to run and report on tests (under an IDE, as part of a build, as a standalone command line client, as a standalone web server) and some of those take time to get going. I suspect a better/unified "getting started" guide would help a lot.

diekhans commented 9 years ago

cli and ide are the cases to optimize, as those would be the developers.

Wayne Stidolph notifications@github.com writes:

Thx. Back to the original (fair) complaint - it should not take an hour to get started! And if someone gets to the right entry point (the cli zip, probably) then it doesn't. But the compliance toolkit does have inherent complexity due to supporting several use cases for how to run and report on tests (under an IDE, as part of a build, as a standalone command line client, as a standalone web server) and some of those take time to get going. I suspect a better/ unified "getting started" guide would help a lot.

— Reply to this email directly or view it on GitHub.*

macieksmuga commented 9 years ago

I believe that the QuickStart documentation should alleviate this issue.