Failing Travis tests

[lukas-vlcek]

Travis tests started to fail on code that was passing correctly in the past. To demonstrate this we created branch called test as a copy of master. The code always fails for tests/commands/test_nodes_hotthreads.py test.

Example: https://travis-ci.org/ViaQ/watches-cli/jobs/239557144#L1119

Unfortunately, I am unable to replicate this locally.

[lukas-vlcek]

When running tests locally (tests pass) I see this:

$ python setup.py test
/usr/local/lib/python2.7/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '1.0.3-dev' to '1.0.3.dev0'
  normalized_version,
running test
============================================================================= test session starts ==============================================================================
platform darwin -- Python 3.5.2, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/lvlcek/projects/lukas-vlcek/watches-cli, inifile:
plugins: cov-2.4.0
...

When the same tests are run in Travis (tests fail) I see this:

$ python setup.py test
/home/travis/virtualenv/python3.5.3/lib/python3.5/site-packages/setuptools/dist.py:334: UserWarning: Normalizing '1.0.3-dev' to '1.0.3.dev0'
  normalized_version,
running test
============================= test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /home/travis/build/ViaQ/watches-cli, inifile:
plugins: cov-2.5.1
...

In the test session I can see the following nifty diff:

platform linux -- Python 3.5.2
  ---- see the diff here ----|
platform linux -- Python 3.5.3

Can there be any significant diff between Python 3.5.3 and Python 3.5.2 ?

[lukas-vlcek]

Just for the record: https://docs.travis-ci.com/user/common-build-problems/#My-tests-broke-but-were-working-yesterday

[lukas-vlcek]

The is was caused by updated Python Elasticsearch client library. On my local system I had elasticsearch-5.0.1 which works fine and tests pass. But if the most recent version of Python ES client is used instead (as of writing it is elasticsearch-5.4.0) then I am able to replicate the issue locally as well.

See reproduction below:

$ pip install elasticsearch
Collecting elasticsearch
  Downloading elasticsearch-5.4.0-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 1.0MB/s 
Requirement already satisfied (use --upgrade to upgrade): urllib3<2.0,>=1.8 in /usr/local/lib/python2.7/site-packages (from elasticsearch)
Installing collected packages: elasticsearch
Successfully installed elasticsearch-5.4.0

What we have now:

$ pip list
certifi (2016.9.26)
coverage (4.2)
DateTime (4.1.1)
docopt (0.6.2)
elasticsearch (5.4.0)
gitdb2 (2.0.0)
GitPython (2.1.0)
logging (0.4.9.6)
pip (8.1.2)
py (1.4.31)
pyaml (16.9.0)
pytest (3.0.4)
pytest-cov (2.4.0)
python-dateutil (2.6.0)
pytz (2016.7)
PyYAML (3.12)
setuptools (23.1.0)
six (1.10.0)
smmap2 (2.0.1)
urllib3 (1.19.1)
vboxapi (1.0)
virtualenv (15.1.0)
watches (1.0.3.dev0, /Users/lvlcek/projects/lukas-vlcek/watches-cli)
wheel (0.29.0)
zope.interface (4.3.2)

Start the tests and see the tests/commands/test_nodes_hotthreads.py fail:

$ python setup.py test
/usr/local/lib/python2.7/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '1.0.3-dev' to '1.0.3.dev0'
  normalized_version,
running test
============================================================================= test session starts ==============================================================================
platform darwin -- Python 3.5.2, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/lvlcek/projects/lukas-vlcek/watches-cli, inifile:
plugins: cov-2.4.0
collected 46 items 

tests/test_cli.py ..
tests/commands/test_cluster_health.py ...................
tests/commands/test_cluster_state.py ......
tests/commands/test_cluster_stats.py ...
tests/commands/test_indices_stats.py .....
tests/commands/test_nodes_hotthreads.py F
tests/commands/test_nodes_info.py .....
tests/commands/test_nodes_stats.py .....

---------- coverage: platform darwin, python 3.5.2-final-0 -----------
Name                                   Stmts   Miss  Cover   Missing
--------------------------------------------------------------------
watches/__init__.py                        1      0   100%
watches/cli.py                            46      5    89%   87, 128-132
watches/commands/__init__.py               7      0   100%
watches/commands/base.py                  96     11    89%   45, 48, 68, 79, 96, 115-117, 136, 155, 173-175
watches/commands/cluster_health.py        10      0   100%
watches/commands/cluster_state.py         22      0   100%
watches/commands/cluster_stats.py          6      0   100%
watches/commands/indices_stats.py         15      0   100%
watches/commands/nodes_hotthreads.py       7      1    86%   10
watches/commands/nodes_info.py            10      0   100%
watches/commands/nodes_stats.py           10      0   100%
watches/util/ESClientProducer.py          27      5    81%   24, 35-41
watches/util/__init__.py                   0      0   100%
--------------------------------------------------------------------
TOTAL                                    257     22    91%

=================================================================================== FAILURES ===================================================================================
____________________________________________________________________ TestNodesHotThreads.test_returns_data _____________________________________________________________________

self = <test_nodes_hotthreads.TestNodesHotThreads testMethod=test_returns_data>

    def test_returns_data(self):
        cmd = self.appendSecurityCommands(['watches', 'nodes_hotthreads', '--threads', '10'])
        o = popen(cmd, stdout=PIPE).communicate()[0].decode('ascii')
>       self.assertTrue(len(o) > 0)
E       AssertionError: False is not true

tests/commands/test_nodes_hotthreads.py:12: AssertionError
----------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------
ERROR:root:
Traceback (most recent call last):
  File "/Users/lvlcek/projects/lukas-vlcek/watches-cli/watches/cli.py", line 142, in execute
    command.run()
  File "/Users/lvlcek/projects/lukas-vlcek/watches-cli/watches/commands/base.py", line 29, in run
    data = self.getData()
  File "/Users/lvlcek/projects/lukas-vlcek/watches-cli/watches/commands/nodes_hotthreads.py", line 19, in getData
    return self.es.nodes.hot_threads(**args)
  File "/usr/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
TypeError: hot_threads() got an unexpected keyword argument 'doc_type'
===================================================================== 1 failed, 45 passed in 18.10 seconds =====================================================================

[lukas-vlcek]

Fixed by #38