Unit tests for universal API

[ddemidov]

See the head of the discussion here: https://github.com/ddemidov/ev3dev-lang-python-1/commit/92828ecf6061b8e5f0a65664eb86082874b5ee89#commitcomment-14064704.

To keep the issue self contained:

@ddemidov:

Note that I did not actually test this on ev3. Creating a Device on my desktop computer works, but may be I'll do an ev3 test tonight and then create a proper PR.

Since we are talking about tests, I had this idea: what if we create a fixed representation of ev3-related nodes in /sys/class, but as normal files (just a deep copy of '/sys/class/{lego,tacho,etc}*')? Then we could alter this line in Device class to point to our 'virtual' or 'sandboxed' ev3 representation and do proper reproducible tests upon each commit. We could even use an online service like Travis-CI for that and have it report on decency of each pull request. @rhempel, @dlech, what do you think?

@dlech:

Sounds like it could be helpful. How would you simulate errors though? For example, if you try to set a non-existent mode, the file read fails with -EINVAL rather than reading the file. Since python is dynamic, you could just temporarily replace a function with something that calls raise, but this would not work for some languages.

@rhempel:

That's where the design of mocks for the CI/CT framework comes in. I tested the basics of the Driver class when I was first implementing it using the DKMS drivers for WeDo on my laptop. That worked well. I think what we are talking about is a "device" that you can load as a driver on the test machine that creates the same entries as are on the EV3 when devices are plugged in. We can start by testing the "rosy" or good path initially, then maybe adding support to fail certain operations as needed. Basically a DKMS driver that fakes out devices plugged in to the EV3. This is a non-trivial effort, so I think the right thing to do is to open up an issue (@ddemidov?) against ev3dev-lang so we don't lose it, then continue our focus on getting the Python binding stable and more complete.

[ddemidov]

@rhempel, fake device driver would make our life as bindings developers extremely easy, but I think the effort required in order to develop such fake drivers (and keep them in sync with the original ones) would be too high. What I thought of is something like this: https://github.com/ddemidov/ev3dev-lang-python-1/commit/0d4db5603e6256c5d724399b4bcccdd474c2dd4e.

Here, I created fake sys/class folder (currently contains single MediumMotor instance), and a unit test. The unit test replaces the Device.DEVICE_ROOT_PATH parameter with the fake path and checks that MediumMotor is able to connect and that the motor properties coincide with what actually is stored in the fake device attributes.

Immediate outcome of this is that the unittest reported to me that the FileCache class, contrary to what I thought, leaks file handles:

$ ./tests/api_tests.py 
/usr/lib/python3.5/unittest/case.py:597: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/demidov/work/ev3/ev3dev-lang/python/tests/fake_sys_class/tacho-motor/motor0/count_per_rot' mode='r' encoding='UTF-8'>
  testMethod()
/usr/lib/python3.5/unittest/case.py:597: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/demidov/work/ev3/ev3dev-lang/python/tests/fake_sys_class/tacho-motor/motor0/encoder_polarity' mode='r' encoding='UTF-8'>
  testMethod()
...

which resulted in https://github.com/ddemidov/ev3dev-lang-python-1/commit/54a8c32fc48381b960c9fd08619b2cef437b1047. Now the complete output of api_test.py looks like this:

.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

The advantage of such file-based tests is that we can easily run the tests on Travis-CI, after each commit pushed to github, fully automatically. In fact, I went ahead and implemented Travis-CI integration in the tests branch of my fork, so now I have a shiny badge at the top of the README, which says that all tests are passing. If you click on the badge, you see the log of the last test (triggered by the latest commit).

[rhempel]

That's a "good enough" mock and will do the job admirably!