Investigate Google Test for unit tests

[teusbenschop]

Bibledit now uses its own crafted version of unit tests.

• [x] Investigation of using gtest for this.
• [x] Start moving all unit tests to "gtest".
• [x] Change to refresh_sandbox (false) at the start of each test.
• [x] Cleanup.

[aranggitoar]

Hello @teusbenschop, I found something odd while doing unit tests today.

The whole journal history kept getting printed repeatedly after every couple of test function run.

Is it probably related to the recent switch to Google Test?

[teusbenschop]

This is weird because over at this side all the unit tests run fine, without any error output, and with 100% pass rate.

Just now while writing this response, the unit tests keep running fine over here.

But then over here I run the unit tests on macOS. What is your system where to run it on?

[aranggitoar]

This is weird because over at this side all the unit tests run fine, without any error output, and with 100% pass rate.

I see, I'll try to clean install the Bibledit instance.

But then over here I run the unit tests on macOS. What is your system where to run it on?

It's the latest Debian in a Docker container.

By the way, did you install Google Test manually from the source at GitHub? Here it's version 1.13.0, might that be a cause?

[teusbenschop]

It's the latest Debian in a Docker container.

Perhaps that is where the difference comes in, although it might also be something else.

By the way, did you install Google Test manually from the source at GitHub? Here it's version 1.13.0, might that be a cause?

It is version 1.12.1, installed through MacPorts. Version 1.13.0 is newer, it's unlikely that this is the cause, but one never knows.

[teusbenschop]

Running it here on Ubuntu 20.04, the unit tests run quite well, although some fail, because of uninstalled helper programs, like Python for example.

Here is the output:

[==========] Running 21 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from existing
[ RUN      ] existing.existing
Line number:    391
Function:       test_usfm2html2usfm
Desired result: \tr \th1 Tribe \th2 Leader \thr3 Number
\tr \tc1 Reuben \tc2 Elizur son of Shedeur \tcr3 46500
\tr \tc1 Simeon \tc2 Shelumiel son of Zurishaddai \tcr3 59300
\tr \tc1 Gad \tc2 Eliasaph son of Reuel \tcr3 45650
\tr \tcr2 Total: \tcr3 151450
Actual result:  \tr \th1 Tribe  | \th2 Leader  | \thr3 Number
\tr \tc1 Reuben  | \tc2 Elizur son of Shedeur  | \tcr3 46500
\tr \tc1 Simeon  | \tc2 Shelumiel son of Zurishaddai  | \tcr3 59300
\tr \tc1 Gad  | \tc2 Eliasaph son of Reuel  | \tcr3 45650
\tr \tcr2 Total:  | \tcr3 151450

test_workspaces
test_verses
test_client
test_sentences
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_sentences_setup
test_versification
test_usfm
test_pairs
test_hyphenate
test_database_noteassignment
test_database_strong
test_database_morphgnt
test_database_etcbc4
test_lexicons
test_database_cache
test_search
test_database_login
test_database_privileges
test_json
test_database_git
test_database_userresources
test_related
test_editone_logic
test_http
test_database_statistics
test_tasks_logic
test_biblegateway
test_rss_feed
test_space
test_roles
test_md5
test_string
test_date
test_export
test_html
test_archive
Line number:    149
Function:       test_archive
Desired result: between 618 and 634
Actual result:  606

test_odf
Line number:    58
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    70
Function:       test_odf
Desired result: Paragraph One

Paragraph Two

Heading One

Paragraph Three
Actual result:  sh: 1: python2.7: not found

Line number:    81
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    85
Function:       test_odf
Desired result: Should create new paragraph automatically
Actual result:  sh: 1: python2.7: not found

Line number:    100
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    106
Function:       test_odf
Desired result: Text†Note

.
Actual result:  sh: 1: python2.7: not found

Line number:    126
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    129
Function:       test_odf
Desired result: textaddnormal.
Actual result:  sh: 1: python2.7: not found

Line number:    150
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    156
Function:       test_odf
Desired result: Text𐌰Addnormal

.
Actual result:  sh: 1: python2.7: not found

Line number:    191
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    194
Function:       test_odf
Desired result: textaddndnormal.
Actual result:  sh: 1: python2.7: not found

Line number:    230
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    236
Function:       test_odf
Desired result: text𐌰addndnormal

.
Actual result:  sh: 1: python2.7: not found

Line number:    255
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    262
Function:       test_odf
Desired result: Paragraph with d style

Paragraph with d style at first, then Standard
Actual result:  sh: 1: python2.7: not found

Line number:    275
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    278
Function:       test_odf
Desired result: One apostrophy ' and two more ''.
Actual result:  sh: 1: python2.7: not found

Line number:    288
Function:       test_odf
Desired result: 0
Actual result:  32512

Line number:    291
Function:       test_odf
Desired result:     

Actual result:  sh: 1: python2.7: not found

test_filter_text
Line number:    156
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    174
Function:       test_filter_text
Desired result: Header4

Header4 Ⅰ

Ⅰ

This is the text of chapter 1, verse 1. This is the text of chapter 1, verse 1. This is the text of chapter 1, verse 1. This is the text of chapter 1, verse 1. This is the text of chapter 1, verse 1. This is the text of chapter 1, verse 1.

Header4 ②

②

This is the text of chapter 2, verse 2. This is the text of chapter 2, verse 2. This is the text of chapter 2, verse 2. This is the text of chapter 2, verse 2. This is the text of chapter 2, verse 2. This is the text of chapter 2, verse 2.
Actual result:  sh: 1: python2.7: not found

Line number:    201
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    227
Function:       test_filter_text
Desired result: Genesis

Genesis 1

Text Genesis 1

Genesis 2

Text Genesis 2

Matthew

Matthew 1

Text Matthew 1

Matthew 2

Text Matthew 2
Actual result:  sh: 1: python2.7: not found

Line number:    246
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    270
Function:       test_filter_text
Desired result: Ruth

The Book of

Ruth

Ruth 1

1 In the days when the judges judged, there was a famine in the land. A certain man of Bethlehem Judah went to live in the country of Moab with his wife and his two sons.

1 Peter

Peter’s First Letter

1 Peter 1

1 Peter, an apostle of Jesus Christ, to the chosen ones who are living as foreigners in the Dispersion in Pontus, Galatia, Cappadocia, Asia, and Bithynia,
Actual result:  sh: 1: python2.7: not found

Line number:    290
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    301
Function:       test_filter_text
Desired result: Genesis

1 Verse One.

Paragraph One. 2 Verse Two.

3 Verse Three. 4 Verse Four. 5 Verse Five.
Actual result:  sh: 1: python2.7: not found

Line number:    315
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    333
Function:       test_filter_text
Desired result: Genesis

1 Text 1aIsa. 1.1.

Isa. 2.2.

bIsa. 3.3.

, text 21Word1: Heb. Explanation1.

2Word2: Heb. Explanation2.

, text3.3Test: Heb. Note at the very end.
Actual result:  sh: 1: python2.7: not found

Line number:    364
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    406
Function:       test_filter_text
Desired result: Genesis

Genesis 1

1

Xref 1aXref 1.

. 2 Xref 2bXref 2.

. 3 Xref 3cXref 3.

 4 Note 11Note 1.

. 5 Note 22Note 2.

. 6 Note 33Note 3.

.

Genesis 2

2

Xref 4aXref 4.

. 2 Xref 5bXref 5.

. 3 Xref 6cXref 6.

 4 Note 41Note 4.

. 5 Note 52Note 5.

. 6 Note 63Note 6.

.
Actual result:  sh: 1: python2.7: not found

Line number:    427
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    436
Function:       test_filter_text
Difference:     1,5c1
< Genesis
< 
< Genesis 1
< 
< ၁။ ကိုယ်တော်သည် တမန်တော် တစ်ဆယ့်နှစ်ပါးတို့ကို အတူတကွခေါ်ပြီးလျှင် နတ်ဆိုးအပေါင်းတို့ကို နှင်ထုတ်နိုင်ရန်နှင့် အနာရောဂါများကို ပျောက်ကင်းစေနိုင်ရန် တန်ခိုးအာဏာတို့ကို သူတို့အား ပေးတော်မူ၏။- ၂။ ထို့နောက် ကိုယ်တော်သည် ဘုရားသခင်၏ နိုင်ငံတော်အကြောင်းကို ဟောပြောရန်နှင့် ဖျားနာသူများကို ကုသ ပျောက်ကင်းစေရန် သူတို့ကို စေလွှတ်တော်မူ၏။- ၃။ ကိုယ်တော်က သင်တို့သွားရမည့်လမ်းခရီးအတွက် မည်သည့်အရာကိုမျှ ယူဆောင်မသွားကြလေနှင့်။ တောင်ဝှေး၊ လွယ်အိတ်၊  စားစရာနှင့် ငွေကြေးတို့ကို သယ်ဆောင်မသွားကြနှင့်။ ဝတ်ရုံကိုလည်း နှစ်ထည်ယူမသွားလေနှင့်။- ၄။ မည်သည့်အိမ်ကိုမဆို သင်တို့ဝင်ကြလျှင် ထိုအိမ်၌နေထိုင်၍ ထိုနေရာမှပင် ပြန်လည်ထွက်ခွာကြလော့။-
\ No newline at end of file
---
> sh: 1: python2.7: not found
\ No newline at end of file

Line number:    813
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    822
Function:       test_filter_text
Desired result: Genesis

Genesis 1

1 I will sing to the Lord.
Actual result:  sh: 1: python2.7: not found

Line number:    836
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    842
Function:       test_filter_text
Desired result: Genesis

1 Text 1a text1 text1.
Actual result:  sh: 1: python2.7: not found

Line number:    859
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    868
Function:       test_filter_text
Desired result: Genesis

Genesis 1

1 I will sing to the Lord God.
Actual result:  sh: 1: python2.7: not found

Line number:    885
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    894
Function:       test_filter_text
Desired result: Genesis

Genesis 1

1 I will sing to the Lord.
Actual result:  sh: 1: python2.7: not found

Line number:    936
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    954
Function:       test_filter_text
Desired result: Genesis

Genesis 1

Chapter 1

1 I will sing to the LORD.

Genesis 2

Chapter 2

2 Jesus came to save the people.
Actual result:  sh: 1: python2.7: not found

Line number:    1004
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    1022
Function:       test_filter_text
Desired result: Genesis

Genesis 1

Chapter One

1 I will sing to the LORD.

Genesis 2

Chapter Two

2 Jesus came to save the people.
Actual result:  sh: 1: python2.7: not found

Line number:    1112
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    1119
Function:       test_filter_text
Desired result: Unknown 1

A Jesus is King.  B Jesus is the son of God.
Actual result:  sh: 1: python2.7: not found

Line number:    1154
Function:       test_filter_text
Desired result: 256
Actual result:  32512

Line number:    1157
Function:       test_filter_text
Desired result: true
Actual result:  false

Line number:    1204
Function:       test_filter_text
Desired result: 0
Actual result:  32512

Line number:    1218
Function:       test_filter_text
Desired result: Unknown 1

1 Verse one. 

caption

2 Verse two.
Actual result:  sh: 1: python2.7: not found

test_url
Line number:    191
Function:       test_url
Desired result: Internet connection failure: unknownhost: nodename nor servname provided, or not known
Actual result:  Internet connection failure: unknownhost: Name or service not known

test_passage
test_styles
test_diff
test_git
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
test_filter_git_setup
Line number:    511
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    512
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    526
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    527
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    528
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    547
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    548
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    560
Function:       test_git
Desired result: 6
Actual result:  1

Line number:    561
Function:       test_git
Desired result: 6
Actual result:  1

test_filter_git_setup
test_filter_git_setup
test_ipc
test_shell
test_database_sample
test_filter_webview
test_javascript
test_filter_mail
test_french
test_merge
test_paratext
test_nmt
test_html2format
test_studylight
test_gbs
test_database_bible_images
test_image
test_easy_english_bible
[       OK ] existing.existing (92362 ms)
[----------] 1 test from existing (92362 ms total)

[----------] 1 test from sqlite
[ RUN      ] sqlite.standard
[       OK ] sqlite.standard (195 ms)
[----------] 1 test from sqlite (195 ms total)

[----------] 7 tests from session
[ RUN      ] session.logic1
[       OK ] session.logic1 (113 ms)
[ RUN      ] session.logic2
[       OK ] session.logic2 (121 ms)
[ RUN      ] session.logic3
[       OK ] session.logic3 (133 ms)
[ RUN      ] session.logic4
[       OK ] session.logic4 (146 ms)
[ RUN      ] session.logic5
[       OK ] session.logic5 (2049 ms)
[ RUN      ] session.logic6
[       OK ] session.logic6 (150 ms)
[ RUN      ] session.users
[       OK ] session.users (169 ms)
[----------] 7 tests from session (2881 ms total)

[----------] 1 test from folders
[ RUN      ] folders.basic
[       OK ] folders.basic (24 ms)
[----------] 1 test from folders (24 ms total)

[----------] 1 test from flate
[ RUN      ] flate.basic
[       OK ] flate.basic (75 ms)
[----------] 1 test from flate (75 ms total)

[----------] 10 tests from database
[ RUN      ] database.users1
[       OK ] database.users1 (116 ms)
[ RUN      ] database.users2
[       OK ] database.users2 (135 ms)
[ RUN      ] database.users3
[       OK ] database.users3 (112 ms)
[ RUN      ] database.users4
[       OK ] database.users4 (119 ms)
[ RUN      ] database.config_general
[       OK ] database.config_general (186 ms)
[ RUN      ] database.config_bible
[       OK ] database.config_bible (0 ms)
[ RUN      ] database.config_user
[       OK ] database.config_user (800 ms)
[ RUN      ] database.logs_1
[       OK ] database.logs_1 (176 ms)
[ RUN      ] database.logs_2
[       OK ] database.logs_2 (189 ms)
[ RUN      ] database.logs_3
[       OK ] database.logs_3 (192 ms)
[----------] 10 tests from database (2025 ms total)

[----------] Global test environment tear-down
[==========] 21 tests from 6 test suites ran. (97562 ms total)
[  PASSED  ] 21 tests.

[aranggitoar]

Hello @teusbenschop, so if I'm not mistaken the log I shared was the journal log from logbook right? Because after I deleted the contents of that directory, there's now only the unit test texts being printed.

I'm curious, when you run the tests above, was it whilst the logbook directory empty or filled?

[teusbenschop]

Initially the logbook directory was completely empty before running the tests.

[aranggitoar]

So I guess the intended procedure before doing unit tests is to do a clean install?

On a side note, the logbook is printed out because of the refresh_sandbox function has "display any old journal entries" functionality that's activated in many of the tests. Hence the long journal entry prints.

[teusbenschop]

You are correct on the function refresh_sandbox.

The intended procedure is that every test runs refresh_sandbox(false); which will clear the logbook entries.

Then at the end of the test, if the logbook entries are of importance, then to run refresh_sandbox(true); but if the logbook entries are irrelevant then to run refresh_sandbox(false);.

So in your case it could be possible that a missing refresh_sandbox(false); at the start of a test causes the issues you were seeing.

[aranggitoar]

I see, thank you for the clarification.

I'm actually seeing many refresh_sandbox (true), and many of them are in the beginning of a test,

Screencast from 2023-07-09 01-01-15.webm

So I get it that they should've been refresh_sandbox (false)?

[teusbenschop]

At the start of the TEST function, yes, refresh_sandbox (false) is more robust.

[aranggitoar]

I got it, I'll try to get those ones right then.

[teusbenschop]

I got it, I'll try to get those ones right then.

I had some time today and fixed all the ones at the start of the test.

The unit tests keep running fine after that change.

[aranggitoar]

Thank you for that Teus, sorry I haven't been able to get to it.

What a great thing that it is now more convenient to do unit tests.

[teusbenschop]

Sure, yes, it is more convenient now:

Listing available tests:

./unittest --gtest_list_tests

Running one specific test only, e.g.:

./unittest --gtest_filter=gbs.logic

Result of that single test:

Note: Google Test filter = gbs.logic
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from gbs
[ RUN      ] gbs.logic
[       OK ] gbs.logic (18183 ms)
[----------] 1 test from gbs (18183 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (18183 ms total)
[  PASSED  ] 1 test.