Build test automation

[TG9541]

This little project gained more support from users, and there are now minimum requirements on the code quality: at least unit tests need to be performed, and regression tests as a part of the release process need to be implemented. At the same time, contributions, improvements, and device support must not be blocked.

@larsbrinkhoff recently contributed the foundation of a continuous integration chain (and later improved the performance, and supported setting up branch protection). With #70 a containerized build- and simulation environment was added, which is planned to work in the C/I-chain.

The next step shall be automated unit tests, and regression tests. I'd like to use T{ .. -> .. }T style test definitions (e.g. similar to the Forth2012 test code).

This ticket shall be closed as soon as automated tests work in a C/I framework. Subsequent improvements shall reference this issue.

[larsbrinkhoff]

Any test at all running in a simulated target would be great progress. In lbForth, I have employed a simple smoke test: start Forth, run words, and print a success message. That will exercise most of the basics.

[TG9541]

A smoke test will of course be the first thing to perform, and running words is a good test for the dictionary. However, most of the bugs in STM8EF discovered by users during the last couple of months would have required more elaborate tests.

[TG9541]

@larsbrinkhoff here we have a case of "test driven development" applied to the test infrastructure ;-)

https://travis-ci.org/TG9541/stm8ef/builds/281604505?utm_source=github_status&utm_medium=notification

This is a good failure - it demonstrates the following:

• Docker integration in Travis-CI is easy to use
• pulling a Docker image in Travis-CI is reasonably fast (improvements possible by using a more lightweight base image)
• the tg9541/docker-sdcc image can be used to build the project
• smoke testing a µC Forth in Travis-CI works as intended

[TG9541]

The positive test worked, too. The simulation with uCsim is quite expensive compute-wise (about 4 Forth lines/second).

I reckon that a "sufficient" eForth unit test would require at least 1000 test steps (testing arithmetical operations require carefully designed tests to be effective and efficient).

However, I think that it would be sufficient to run such tests on demand, or before merging develop into master.

I'm a Travis-CI newbie, and I didn't see a way to detect whether there is the intention to merge into master. After browsing the list of Travis-CI variables, here is what I think how to do it:

• the build has been triggered by setting a tag (which would be to late)
• trigger the extended test only manually (which invites errors)
• run the extended tests deferred (by using a matrix build and the allow_failures feature)

[larsbrinkhoff]

What's the difference in time between the old cached build, and the new dockerized? I believe I see 20-30 seconds for the old, and 40-50 for the new. Is that correct?

[TG9541]

The cache-based builds I checked took anywhere from 24 to 54 seconds (I guess that depending on the load in the Travis-CI cluster sometimes jobs have to wait a little for "workers", or VMs get less CPU time).

According to the raw log starting the VM took 21s (the same as with the old configuration).

The difference is due to pulling a fat Ubuntu16.04 based image (17s), whereas copying the build cache in the old configuration is blazing fast.

I can try using a leaner image (e.g. debian:buster-slim), which might save some seconds. A more radical option might be caching the docker image. I have an idea how to do that, but there is a reason to believe that it won't be anywhere as fast as a plain cache.

The test with uCsim takes about 7s. Only a rewrite of uCsim (or a new simulator) can make it faster.

[larsbrinkhoff]

Thanks. I'm following this with great interest, to see if I should migrate my tests to Docker too.

As for doing quick vs extensive testing, I don't have a solution handy. One of my builds take close to 30 minutes. I just live with it.

[TG9541]

30min may sound like a lot but I have seen tests running for a day and a half. I wouldn't call that Continuous Integration, though ;-)

The good thing about Docker is, when done right, it takes care of all dependencies. One can test build- and test-steps locally, and expect that in the CI environment they work exactly the same.

Another advantage is that I now can offer all users a local build- and test environment that will just work (although I don't know how many guys in the maker scene are used to typing docker run).

[TG9541]

I think I now have the basics for console based unit, integration, and smoke-testing. A key element is T{e ... e-> ... }T, a construct for testing console output (character count and checksum). It also works with uCsim (although timeouts have been an issue due to bad performance).

Here is a demo:

[TG9541]

Here we go - basic test automation is in place now :-)

[larsbrinkhoff]

I like you asciinema demos!

[TG9541]

@larsbrinkhoff here is one more :-) The test is now a lot more complete.

[TG9541]

In Travis-CI, pull-build-test took about a minute (58s)

[larsbrinkhoff]

If I may submit some feedback on the videos? I find them a little bit hard to follow, because things happen a little too fast, and I can't read one screenful before the next is due. Some other videos have a soundtrack with verbal hints which help, but of course that's not an option with asciinema.

Here's a purely text-based video (except the comforting rattling of keys). The pace is slower, maybe too slow sometimes. But the action sequences are sandwiched between informational text blobs.
https://www.youtube.com/watch?v=zF-vy8b4r-Ys

[TG9541]

Feedback is always welcome :-)

The STM8S output is (surprisingly) hard to follow: thanks to e4thcom the transfer runs without any additional delays, and asciinema renders what it receives. Text blobs is one of the features many asciinema users asked for. I would be happy if text and pace could be edited in some way.

What I always wanted to do is a teaser video from flashing a W1209 to having it do some simple automation task. For most people testing is boring, anyway ;-)

However, making demos is yet another skill I need to acquire. Some of my newish colleagues are really good at it, and I know that can take a lot of time to get a demo video right.