Replace avocado based python tests with shell script based tests

[coolbrg]

After going through the debugging of #121 , I am feeling that avocado based Python tests are good for reporting purpose however it is not able to properly display the logs. For example I enabled --show-libmachine-logs to see logs but this build

13:57:39 Executing command : /root/payload/tests/../build/bin/minishift start --vm-driver kvm --iso-url file:///root/payload/tests/../build/minishift-centos7.iso --show-libmachine-logs
******
13:57:39 *******
14:08:57 Error: E0907 15:01:48.099611   15230 start.go:344] Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded. Retrying.
14:08:57 E0907 15:05:22.787269   15230 start.go:344] Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded. Retrying.
14:08:57 E0907 15:08:57.903771   15230 start.go:344] Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded. Retrying.
14:08:57 Error starting the VM: Error configuring authorization on host: Maximum number of retries (60) exceeded
14:08:57 Error configuring authorization on host: Maximum number of retries (60) exceeded
14:08:57 Error configuring authorization on host: Maximum number of retries (60) exceeded
14:08:57 
14:08:57 
14:08:57 Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado/core/test.py:603
14:08:57 Traceback (most recent call last):
14:08:57   File "/root/payload/tests/test.py", line 98, in test_restart
14:08:57     self.execute_test({ 'cmd': cmd })
14:08:57   File "/root/payload/tests/test.py", line 131, in execute_test
14:08:57     self.assertEqual(0, process.returncode)
14:08:57   File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual
14:08:57     assertion_func(first, second, msg=msg)
14:08:57   File "/usr/lib64/python2.7/unittest/case.py", line 546, in _baseAssertEqual
14:08:57     raise self.failureException(msg)
14:08:57 AssertionError: 0 != 1
14:08:57 
14:08:57 Local variables:
14:08:57  -> start_args <type 'tuple'>: ('kvm', 'file:///root/payload/tests/../build/minishift-centos7.iso')
14:08:57  -> cmd <type 'str'>: /root/payload/tests/../build/bin/minishift start --vm-driver kvm --iso-url file:///root/payload/tests/../build/minishift-centos7.iso --show-libmachine-logs
14:08:57  -> self <class 'test.MinishiftISOTest'>: 11-tests/test.py:MinishiftISOTest.test_restart
14:08:57 FAIL 11-tests/test.py:MinishiftISOTest.test_restart -> AssertionError: 0 != 1

I wrote a temporary shell script to see logs in this build which seems to give expected logs where one can easily see what's wrong going on.

Also, shell based test feel natural .

Any thoughts @gbraad @praveenkumar @LalatenduMohanty ?

[gbraad]

Avocado is nice to do comparison checks, like assert on values to be equal, etc. but besides that, it does not really offer much more for me...

[coolbrg]

@amitkrout @agajdosi Your thoughts as well.

[amitkrout]

AFAIK whatever the outupt we get is piped to log file created by avocado. As part of test script is concerned avocado has better advantage over shell script, even we can leverage its uses in future. Not sure how replacing with shell script resolves the problem. IMO rather replacing this with shell script we should take this to avocado repo

If time is now the constraint then will take shell script as a solution and will create a new issue to track the avocado problem. WDYT ?

[coolbrg]

@amitkrout , my main concern is following

1. avocado based tests - https://ci.centos.org/job/minishift-centos-iso-pr/221/console (Unable to see logs of minishift operations)

2. Shell based tests - https://ci.centos.org/job/minishift-centos-iso-pr/225/console

[amitkrout]

I see.. @budhrg We have a option to call the shell script through avocado. Something like this way

avocado run --show-job-log tests/test.sh

Would you pleas try this one

[coolbrg]

@amitkrout , we are using that form only

https://github.com/minishift/minishift-centos-iso/blob/master/centos_ci.sh#L47

https://github.com/minishift/minishift-centos-iso/blob/master/Makefile#L90

[amitkrout]

@budhrg i think you did not get me what i am trying to say. Let me put in this way

Just call the shell script through avocado run as i mentioned above avocado run --show-job-log tests/test.sh, not like this sh tests/test.sh. So that we can atleast know the error coming from the test.py script or from avocado itself.

[coolbrg]

@amitkrout , not working as avocado framework assumes tests are written in Python language.

Error from job when I made above change

09:21:28 Done.
09:21:28 avocado run --show-job-log tests/test.sh
09:21:28 found extension EntryPoint.parse('journal = avocado.plugins.journal:JournalResult')
09:21:28 found extension EntryPoint.parse('tap = avocado.plugins.tap:TAPResult')
09:21:28 found extension EntryPoint.parse('human = avocado.plugins.human:Human')
09:21:28 found extension EntryPoint.parse('teststmpdir = avocado.plugins.teststmpdir:TestsTmpDir')
09:21:28 found extension EntryPoint.parse('jobscripts = avocado.plugins.jobscripts:JobScripts')
09:21:28 found extension EntryPoint.parse('human = avocado.plugins.human:HumanJob')
09:21:28 
09:21:28 Unable to resolve reference(s) 'tests/test.sh' with plugins(s) 'file', 'external', try running 'avocado list -V tests/test.sh' to see the details.
09:21:29 make: *** [test] Error 2

$ avocado list -V tests/test.sh
Type       Test                                                                         Tag(s)
NOT_A_TEST tests/test.sh: Does not look like an INSTRUMENTED test, nor is it executable 

TEST TYPES SUMMARY
==================
ACCESS_DENIED: 0
BROKEN_SYMLINK: 0
EXTERNAL: 0
INSTRUMENTED: 0
MISSING: 0
NOT_A_TEST: 1
SIMPLE: 0

[agajdosi]

@budhrg You can use --external-runner flag, it will be then --external-runner /bin/bash if I remember correctly.

[LalatenduMohanty]

Agree with @gbraad . Also now that we have integration test in Minishift which takes care of lots of system testing (black box) I am not inclined towards Avocado.