Adding integration testing

[cr]

• Covering regression, scan, srcupdate and JSON logging

I am mildly excited about this important step towards 3.1.1. Knowing how trivial it was to implement, I now think we should have had this a long time.

[cr]

For now the integration tests are really simple: run some tlscanary scan that should produce N lines of logging, then check if the JSON log of the last run has a data vector with N lines.

I still think this will spot most major breakage due to its ridiculously large code coverage, and it's also a solid base for refinement.

[cr]

Fun fact: Adding just those two slender integration tests increases test coverage from 42 to 68%.

There are a lot of low-hanging fruits here.

Without integration tests

$ nosetests --with-coverage --cover-erase --cover-package=tlscanary -v -e zz_
[...]
Name                              Stmts   Miss  Cover
-----------------------------------------------------
tlscanary/__init__.py                 0      0   100%
tlscanary/cache.py                   53      4    92%
tlscanary/cert.py                    36     25    31%
tlscanary/firefox_app.py             59     12    80%
tlscanary/firefox_downloader.py      90     24    73%
tlscanary/firefox_extractor.py       47      9    81%
tlscanary/modes/__init__.py          15      0   100%
tlscanary/modes/basemode.py         160    133    17%
tlscanary/modes/log.py              147    130    12%
tlscanary/modes/performance.py       81     65    20%
tlscanary/modes/regression.py        95     77    19%
tlscanary/modes/scan.py              48     33    31%
tlscanary/modes/sourceupdate.py      89     71    20%
tlscanary/one_crl_downloader.py      52     44    15%
tlscanary/progress_bar.py            78     67    14%
tlscanary/report.py                 142    121    15%
tlscanary/runlog.py                 191     52    73%
tlscanary/sources_db.py             114     10    91%
tlscanary/worker_pool.py            130    102    22%
tlscanary/xpcshell_worker.py        120     27    78%
-----------------------------------------------------
TOTAL                              1747   1006    42%
----------------------------------------------------------------------
Ran 19 tests in 56.790s

With integration tests

$ nosetests --with-coverage --cover-erase --cover-package=tlscanary -v
[...]
Name                              Stmts   Miss  Cover
-----------------------------------------------------
tlscanary/__init__.py                 0      0   100%
tlscanary/cache.py                   53      4    92%
tlscanary/cert.py                    36     25    31%
tlscanary/cleanup.py                 17      4    76%
tlscanary/firefox_app.py             59     12    80%
tlscanary/firefox_downloader.py      90     24    73%
tlscanary/firefox_extractor.py       47      9    81%
tlscanary/loader.py                  13      3    77%
tlscanary/main.py                   111     44    60%
tlscanary/modes/__init__.py          15      0   100%
tlscanary/modes/basemode.py         160     39    76%
tlscanary/modes/log.py              147     88    40%
tlscanary/modes/performance.py       81     65    20%
tlscanary/modes/regression.py        95      8    92%
tlscanary/modes/scan.py              48      2    96%
tlscanary/modes/sourceupdate.py      89     10    89%
tlscanary/one_crl_downloader.py      52     15    71%
tlscanary/progress_bar.py            78     45    42%
tlscanary/report.py                 142    121    15%
tlscanary/runlog.py                 191     51    73%
tlscanary/sources_db.py             114      7    94%
tlscanary/worker_pool.py            130      8    94%
tlscanary/xpcshell_worker.py        120     17    86%
-----------------------------------------------------
TOTAL                              1888    601    68%
----------------------------------------------------------------------
Ran 21 tests in 169.157s

OK

[cr]

Fun fact 2: Those integration tests alone yield 65% test coverage:

$ nosetests --with-coverage --cover-erase --cover-package=tlscanary -v --tests=tests/__init__.py,tests/zz_tlscanary_integration_test.py
Downloading firefox instance for tests ... ok
TLS Canary can make regression runs ... ok
TLS Canary can update source DBs ... ok

Name                              Stmts   Miss  Cover
-----------------------------------------------------
tlscanary/__init__.py                 0      0   100%
tlscanary/cache.py                   53     18    66%
tlscanary/cert.py                    36     25    31%
tlscanary/cleanup.py                 17      4    76%
tlscanary/firefox_app.py             59     26    56%
tlscanary/firefox_downloader.py      90     32    64%
tlscanary/firefox_extractor.py       47     16    66%
tlscanary/loader.py                  13      3    77%
tlscanary/main.py                   111     44    60%
tlscanary/modes/__init__.py          15      0   100%
tlscanary/modes/basemode.py         160     39    76%
tlscanary/modes/log.py              147     88    40%
tlscanary/modes/performance.py       81     65    20%
tlscanary/modes/regression.py        95      8    92%
tlscanary/modes/scan.py              48      2    96%
tlscanary/modes/sourceupdate.py      89     10    89%
tlscanary/one_crl_downloader.py      52     15    71%
tlscanary/progress_bar.py            78     45    42%
tlscanary/report.py                 142    121    15%
tlscanary/runlog.py                 191     62    68%
tlscanary/sources_db.py             114     15    87%
tlscanary/worker_pool.py            130      8    94%
tlscanary/xpcshell_worker.py        120     19    84%
-----------------------------------------------------
TOTAL                              1888    665    65%
----------------------------------------------------------------------
Ran 3 tests in 157.965s

OK

[mwobensmith]

This is excellent! Let's land this!