Add Google Test support

ngie-eign commented 5 years ago

This change adds fine grained execution of Google Test test programs.

First, the Google Test test is executed with --gtest_list_tests. Next, based on the output from --gtest_list_tests, the testcases are run individually. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub here.

This support is a very rough cut to provide an initial working integration effort. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library.

This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via $TEST_ENV_ prefixed environment variables.

This support adds additional tests for verifying pass, fail, skip (will be available in version 1.9.0 and is available in FreeBSD base's version of Google Test), and disabled result determination, using mock output and a mock test program (engine/googletest_helpers), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on the GoogleTest test infrastructure, in order to limit Kyua's dependencies.

Signed-off-by: Enji Cooper yaneurabeya@gmail.com

ngie-eign commented 5 years ago

CC: @asomers, @emaste

asomers commented 5 years ago

:tada: One problem, though. When I use the new googletest test program, "kyua report" doesn't work.

> /home/somers/src/github/jmmv/kyua/kyua report
kyua: E: Column 'result_reason' is not a string (sqlite db: /home/somers/.kyua/store/results.usr_tests_sys_fs_fusefs.20190403-025838-736040.db).

ngie-eign commented 5 years ago

🎉 One problem, though. When I use the new googletest test program, "kyua report" doesn't work.
> /home/somers/src/github/jmmv/kyua/kyua report
kyua: E: Column 'result_reason' is not a string (sqlite db: /home/somers/.kyua/store/results.usr_tests_sys_fs_fusefs.20190403-025838-736040.db).

Hmmm... that's strange. I'll see if I can reproduce that.

asomers commented 5 years ago

"kyua report" still give me the same error even with your newest patch.

ngie-eign commented 5 years ago

"kyua report" still give me the same error even with your newest patch.

I suspect it's one of the cases involving the skipped or disabled status. Need to do some more digging here with your branch..

ngie-eign commented 5 years ago

As I suspected, the issue is that GTEST_SKIP is being called without a reason:

$ sudo kyua test -k /usr/tests/sys/fs/fusefs/Kyuafile write:Write.append
write:Write.append  ->  skipped  [0.017s]

Results file id is usr_tests_sys_fs_fusefs.20190405-171435-875837
Results saved to /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-171435-875837.db

1/1 passed (0 failed)
$ sudo kyua report --results-file=/root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-171435-875837.db
kyua: E: Column 'result_reason' is not a string (sqlite db: /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-171435-875837.db).
$ sudo kyua debug -k /usr/tests/sys/fs/fusefs/Kyuafile write:Write.append                                                                                             
Note: Google Test filter = Write.append
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Write
[ RUN      ] Write.append
[  SKIPPED ] Write.append (13 ms)
[----------] 1 test from Write (13 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (13 ms total)
[  PASSED  ] 0 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] Write.append
write:Write.append  ->  skipped

Wait... the fact that GTEST_SKIP is not properly emitting any messages is a bug in our (FreeBSD base) version of googletest too. Argh...

ngie-eign commented 5 years ago

Hmmm... that didn't work :/...

$ kyua report --results-file=/root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-182049-970358.db
kyua: E: Column 'result_reason' is not a string (sqlite db: /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-182049-970358.db).
$ sqlite3 /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-182049-970358.db 
SQLite version 3.27.1 2019-02-08 13:17:39
Enter ".help" for usage hints.
sqlite> SELECT * FROM test_results;
1|skipped|YOU HAVE 1 DISABLED TEST|1554488449986045|1554488449990167
2|skipped|YOU HAVE 1 DISABLED TEST|1554488449992589|1554488449997075
3|skipped|YOU HAVE 1 DISABLED TEST|1554488450000970|1554488450005545
4|skipped||1554488450007688|1554488450022533
5|passed||1554488450029400|1554488450039586
6|passed||1554488450042407|1554488450061054
7|passed||1554488450064575|1554488450077458
8|passed||1554488450080841|1554488450091814
9|passed||1554488450095204|1554488450107807
10|passed||1554488450110648|1554488450123577
11|passed||1554488450126588|1554488450139898
12|skipped||1554488450143172|1554488450153392
13|skipped||1554488450156506|1554488450165748
14|skipped||1554488450169125|1554488450178281
15|skipped||1554488450182709|1554488450193590
16|skipped|YOU HAVE 1 DISABLED TEST|1554488450196618|1554488450201177
17|skipped|YOU HAVE 1 DISABLED TEST|1554488450203280|1554488450207678
18|passed||1554488450209844|1554488450220627

asomers commented 5 years ago

As I suspected, the issue is that GTEST_SKIP is being called without a reason:

Shouldn't that be allowed? If you don't supply a reason to GTEST_SKIP, shouldn't it use a sane default, like ""?

ngie-eign commented 5 years ago

Shouldn't that be allowed? If you don't supply a reason to GTEST_SKIP, shouldn't it use a sane default, like ""?

That’s what I’m trying to implement as a workaround, but it’s a waste of space on disk.

ngie-eign commented 5 years ago

Ok. I'm starting to understand where the issue's coming from:

192 static model::test_result
193 parse_result(sqlite::statement& stmt, const char* type_column,
194              const char* reason_column)
195 {       
196     try {
197         const model::test_result_type type =
198             store::column_test_result_type(stmt, type_column);
199         if (type == model::test_result_passed) {
200             if (stmt.column_type(stmt.column_id(reason_column)) !=
201                 sqlite::type_null)
202                 throw store::integrity_error("Result of type 'passed' has a "
203                                              "non-NULL reason");
204             return model::test_result(type);
205         } else {
206             return model::test_result(type,
207                                       stmt.safe_column_text(reason_column));
208         }

ngie-eign commented 5 years ago

Well! It looks like store/write_transaction.cpp:425-426 does the field NULLing for me :D!

409 int64_t
410 store::write_transaction::put_result(const model::test_result& result,
411                                      const int64_t test_case_id,
412                                      const datetime::timestamp& start_time,
413                                      const datetime::timestamp& end_time)
414 {
415     try {
416         sqlite::statement stmt = _pimpl->_db.create_statement(
417             "INSERT INTO test_results (test_case_id, result_type, "
418             "                          result_reason, start_time, "
419             "                          end_time) "
420             "VALUES (:test_case_id, :result_type, :result_reason, "
421             "        :start_time, :end_time)");
422         stmt.bind(":test_case_id", test_case_id);
423 
424         store::bind_test_result_type(stmt, ":result_type", result.type());
425         if (result.reason().empty())
426             stmt.bind(":result_reason", sqlite::null());
427         else
428             stmt.bind(":result_reason", result.reason());

ngie-eign commented 5 years ago

And... magic!

$ sudo kyua test -k /usr/tests/sys/fs/fusefs/Kyuafile write
write:AioWrite.DISABLED_aio_write  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
write:Write.DISABLED_direct_io_evicts_cache  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
write:Write.DISABLED_mmap  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
write:Write.append  ->  skipped  [0.012s]
write:Write.append_direct_io  ->  passed  [0.010s]
write:Write.direct_io_short_write  ->  passed  [0.010s]
write:Write.direct_io_short_write_iov  ->  passed  [0.012s]
write:Write.indirect_io_short_write  ->  passed  [0.016s]
write:Write.write  ->  passed  [0.011s]
write:Write.write_large  ->  passed  [0.012s]
write:Write.write_nothing  ->  passed  [0.010s]
write:WriteBack.close  ->  skipped  [0.011s]
write:WriteBack.o_direct  ->  skipped  [0.009s]
write:WriteBack.rmw  ->  skipped  [0.010s]
write:WriteBack.writeback  ->  skipped  [0.013s]
write:WriteThrough.DISABLED_update_file_size  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.007s]
write:WriteThrough.DISABLED_writethrough  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
write:WriteThrough.pwrite  ->  passed  [0.010s]

Results file id is usr_tests_sys_fs_fusefs.20190405-223507-578754
Results saved to /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-223507-578754.db

18/18 passed (0 failed)
$ kyua report --results-file=/root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-223507-578754.db
===> Skipped tests
write:AioWrite.DISABLED_aio_write  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
write:Write.DISABLED_direct_io_evicts_cache  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
write:Write.DISABLED_mmap  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
write:Write.append  ->  skipped  [0.012s]
write:WriteBack.close  ->  skipped  [0.011s]
write:WriteBack.o_direct  ->  skipped  [0.009s]
write:WriteBack.rmw  ->  skipped  [0.010s]
write:WriteBack.writeback  ->  skipped  [0.013s]
write:WriteThrough.DISABLED_update_file_size  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.007s]
write:WriteThrough.DISABLED_writethrough  ->  skipped: YOU HAVE 1 DISABLED TEST  [0.004s]
===> Summary
Results read from /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-223507-578754.db
Test cases: 18 total, 10 skipped, 0 expected failures, 0 broken, 0 failed
Total time: 0.170s

ngie-eign commented 5 years ago

@jmmv: I realize this is a large change, but I was wondering if there's a way that I could work with you to get this change reviewed and into the kyua project? I'm asking, because there's a chance that I might get a job in the next few months which will invalidate my individual CLA.

Plus, this change makes it markably easier to run Google Test programs in FreeBSD with Kyua.

ngie-eign commented 5 years ago

@jmmv: ping?

ngie-eign commented 5 years ago

@jmmv: ping?

ngie-eign commented 5 years ago

@jmmv: ping?

ngie-eign commented 5 years ago

@jmmv: ping?

asomers commented 5 years ago

@jmmv could you please review this change? If this gets merged and released before I finish my fusefs work, it will change how I organize the tests.

ngie-eign commented 5 years ago

@jmmv: ping..?

ngie-eign commented 4 years ago

@emaste: given the lack of feedback in this PR, I think it's time to consider officially forking atf/kyua/lutok into a FreeBSD-org managed repo and install the ports from that repo. I'll reach out to you soon about this.

markjdb commented 2 weeks ago

@emaste: given the lack of feedback in this PR, I think it's time to consider officially forking atf/kyua/lutok into a FreeBSD-org managed repo and install the ports from that repo. I'll reach out to you soon about this.

Now that this is done, is there any chance this PR could be rebased and landed? The lack of googletest support in kyua bit me recently: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=281100

freebsd / kyua

Add Google Test support #203