Set up continuous integration?

[desertsagebrush]

Would it be worth setting up a simple continuous integration system to make sure that changes to the codebase/setup.py do not break installation? This would also open up the ability to perform automated testing of changes once the repository stabilizes a bit more.

I would be willing to work on setting this up, if you all have a platform that you are interested in using.

[colinoflynn]

Hello,

It's something we'd love to run - actually we basically have a (long-term?) plan on automatically running our various tests on hardware. To that end we've got some setup that we could have for example a CW-Lite-Arm, CW-Lite-XMEGA, CW-Nano, CW-Pro, etc all connected to one computer.

Are there easier platforms for CI? Normally it's mostly SW tests I know which would be more generic, but if there is something that could be extended for hardware test would be good to know.

[desertsagebrush]

My primary experience has been with platforms like Gitlab CI or Circle CI, where you can set up a system to automatically build the software. However, one organization that I have worked for has CI (using Gitlab's CI) set up to connect to a production machine and deploy code when the automated tests pass, so something similar where it connects to a machine with hardware should be possible. However, that is all on an internal network, so I'm not sure how well/easily that could be done with Github.

[rob0tsunny]

I can potentially setup Jenkins or so in a cloud server. or even gitlab if it is appropriate. we might need to make code a bit more test friendly and decouple it a bit from hardware - in my experience most issues are not hardware related.

[FranzHeubach]

I had a thought related to this. It should be possible to do this if we replace the class that does the actual serial communications with a different class in testing. We can then compare the expected result of the API call to the serial message that was actually sent. This would allow us to test the result of API calls independently from hardware state. It would also make it possible to do something like CI tests.

[FranzHeubach]

If someone wants to try to make the code more test friendly for hardware free testing, have at it. I am working on improving the way our functional tests are run. For the person or people who want to do this I suggest taking a look at the naeusb.py module. Or you can try decoupling the software somewhere higher up. We have the beginning of some hardware-free tests in the tests directory. The test_api.py function tests parts of the API that do not need the hardware to be tested.

CI can be set up by someone who wants to take that on. It would be nice to have something that tests installing the software on different operating systems, as we do that mostly manually currently.

It would be great to keep it simple to start off with as @Wmyers559 suggested.

This issue will be for continued related discussion.

[FranzHeubach]

Related to this issue, We now have a local testing server up and running. It runs through every jupyter notebook tutorial on some of the boards every 4 hours. The plan is to add as many of the different types of board as possible. This way we can catch mistakes as early as possible when making changes to the develop branch... exciting!

The tests are functional tests and emulate the users experience when using the notebooks. It would still be nice to work towards hardware-less unit tests for faster testing, and using a CI test setup.

[alex-dewar]

Closing as we now have a test server that runs on each push to git that runs through some tutorials.