search

GoogleCloudPlatform / gradle-appengine-plugin

Gradle plugin that provides tasks for uploading, running and managing Google App Engine projects
Apache License 2.0
236 stars 60 forks source link

Question: Logging for status of functional tests? #245

Open ndtreviv opened 8 years ago

ndtreviv commented 8 years ago

When I run unit tests, the system outputs a line for each test with a SCCESS/FAILURE indicator to the console, which is then visible when watching the build process on a CI server. Also, the test results are output in such a way that I can visually examine them (ie: graph) on our CI server.

When running functional tests (using gradle appengineFunctionalTest) there is no output other than BUILD SUCCEEDED if they all passed. Am I missing a flag of some sort to indicate success of each test?

At the moment I'm considering using nightwatchjs instead, as the framework to control selenium already exists, the output is great, and it logs success/failures in a junit-style way so that I can see them on our CI server.

I would love appengineFunctionalTest to output the same sort of results format so that I can visualise them in the same way. Is this possible now?

loosebazooka commented 8 years ago

Sorry, the functionality for appengineFunctionalTest is before my time but it's just another test run, so if your functional tests were using a gradle supported test runner like junit or testNG then I believe it should be producing test results. What's your test setup here for starting your selenium tests? It's quite possible that using this other framework is the way to go.

ndtreviv commented 8 years ago

Sorry - I discovered why this was happening and fixed it. I do get failure reports to the console now, which is what I was looking for. Totally forgot I added this! Closing it now.

ndtreviv commented 8 years ago

Ahh - sorry! Most of this was resolved, but not all.

Whilst I do get failure notifications to the console, I don't get success notifications.

I have a lot of functional tests, they take a long time to run. System.out or log messages don't appear on the console. There is no way to see progress. When I run it locally, gradle will keep me updated on how many it's run. When I run it on Jenkins I get nothing.

When you run JUnit tests it outputs the name of the test and either FAILED or SUCCESS next to on the console. It would be really useful if the functional tests did this too. Is there any way to make it do this?

elharo commented 8 years ago

I'm not sure, but this may be working as intended. It's a general principle of TDD that passing tests generate no significant output other than "tests passed" or equivalent. The reason is so that it's really obvious whether tests passed or failed. This is especially important in a non-GUI environment since it avoids the antipattern where a single test failure gets hidden in page after page of logging from successful tests. The JUnit test runners have mostly gotten this right over the years, though maven and gradle do not.