Automated test coverage for Travis CI

[Harvie]

Let's continue the discussion with @coddingtonbear from #685, we have Travis working in bCNC project. What are the possibilities? I want to implement following:

• automatic .exe build using travis cloud
• automatic screenshots/screencasts using travis cloud
• functionality testing using fake GRBL script hooked in using pyserial URL handlers

[sfinexer]

• functionality testing using fake GRBL script hooked in using pyserial URL handlers

I would like a description of this, and a small example.

[coddingtonbear]

I've written a little fake grbl CLI app here: https://github.com/coddingtonbear/gcode-receiver ; it has a few limitations, and I haven't yet tried to wire it up to bCNC via socat or direct import, but it's at least a step.

Those limitations I mentioned come in two separate groups; the first is accuracy. The interpreter I wrote is not very faithful to reality in that, although move commands do take some time to complete, I haven't implemented interpolation of the machine position coordinates between its starting and ending positions, so it'll just hop over to the target position after a configurable delay. Additionally, only a handful of commands are implemented (see the methods around here), and currently it'll just ignore unknown commands.

The second set of limitations is around how this could be integrated into bCNC's integration tests; this current implementation is just a console app interacting with stdin/stdout as if it were the normal Grbl console; so to wire it up as-is to bCNC it would require starting this via socat. That being said, it wouldn't be very hard at all to import the module directly into bCNC when necessary or to instead have it open a socket -- we'd just need to make a subclass of the existing GcodeReceiver class that overrode get_command, send_output, and start, or, probably better yet: factoring out its existing terminal and stdout/in interactions into a subclass of its own.

I should note that I haven't tried connecting this to bCNC yet; it's totally possible bCNC requires commands I haven't implemented.

So; this is how far I am so far; hopefully this helps!

[Harvie]

I haven't implemented interpolation of the machine position coordinates between its starting and ending positions, so it'll just hop over to the target position after a configurable delay.

I don't think this is needed for testing purposes. I think this should be as minimal as possible. But i think we can calculate the delay based on feedrate and segment length...

For full simulation you should use grbl-sim anyway...

this current implementation is just a console app interacting with stdin/stdout as if it were the normal Grbl console

This is perfect. You can use socat, which i guess can be installed to travis environment.

And as i said. I want to add new URL handler to pyserial. Something like exec://python myscript.py so this will be very easy to do. There are already handlers used to connect to remote serial via TCP, so the infrastructure is ready:

https://pyserial.readthedocs.io/en/latest/url_handlers.html

You can use this directly in bCNC already:

Just add new handler here:

https://github.com/pyserial/pyserial/tree/master/serial/urlhandler

I've already started discussion about this, but so far no response, so please vote:

https://github.com/pyserial/pyserial/issues/393

Also note that such handler will also allow exec://ssh user@host wrapper /dev/ttyUSB0 to connect to remote serial over ssh. That would be usefull.

If we manage to get this merged directly to pyserial it will be beneficient for everyone, not just for bCNC.

[Harvie]

Test suite for GRBL streaming is more than necessary, because there seems to be some serious bug #1108 and it's blocking next bCNC release. I don't say we need to automate this using Travis, but at least we need way to test this manualy.

[coddingtonbear]

Posted #685

[Harvie]

There's pull request showing this problem and i think it might in fact be problem of the testsuite itself:

https://travis-ci.com/vlachoudis/bCNC/jobs/183593669

$ pytest --capture=no --verbose tests/
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.2.3, py-1.5.2, pluggy-0.4.0 -- /home/travis/virtualenv/python2.7.14/bin/python
cachedir: .cache
rootdir: /home/travis/build/vlachoudis/bCNC, inifile:
collected 0 items / 1 errors                                                    
==================================== ERRORS ====================================
_____________________ ERROR collecting tests/test_smoke.py _____________________
../../../virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/python.py:395: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
../../../virtualenv/python2.7.14/lib/python2.7/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
../../../virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/assertion/rewrite.py:212: in load_module
    py.builtin.exec_(co, mod.__dict__)
tests/test_smoke.py:4: in <module>
    import pyautogui
../../../virtualenv/python2.7.14/lib/python2.7/site-packages/pyautogui/__init__.py:65: in <module>
    import pyscreeze
E   SyntaxError: 'return' with argument inside generator (__init__.py, line 168)
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.23 seconds ============================
The command "pytest --capture=no --verbose tests/" exited with 2.

Any clues?

[MARIOBASZ]

I have little knowledge, but maybe the error comes from trochoud3d currently calls a function with more arguments than before? First merge the changes in cnc, main and trchoid_path files, in that order, solve the problem?

[Harvie]

First merge the changes in cnc, main and trchoid_path files, in that order, solve the problem?

It's not about merging. The tests runs on top of your repository. So no need to merge for test to be succesful. It already has all files from your repo, maybe something is missing in your repo.

[MARIOBASZ]

What steps do I follow? To update my repository I made an extraction request by changing the places viachoudis with mine (I do not know if it is the correct way)

[Harvie]

@coddingtonbear It's confirmed. there is problem with that test, because travis fails on it even on bCNC version that was previously OK. I guess they've upgraded something on Travis side. I temporarily disabled it until we figure this out...

[Harvie]

So bCNC should be mostly working in python3... Some testing might come in handy right now. But it's probably too much work to write tests for everything...