Continuous integration/testing

[gabrielstuff]

Hey !

Me again. Didn't find anywhere this infos. I'm currently testing the travis-ci.org service. As many people are working on the project and as many many issue are raised, and fixed, I thought it could be great to add some test and thus the travis service.

It would avoid the problem met here : https://github.com/openframeworks/openFrameworks/commit/7ca7833ea1afb6bd5a6c54031e3fa688aa0c0ba8 and the discussion here : https://github.com/openframeworks/openFrameworks/issues/804 or there : https://github.com/openframeworks/openFrameworks/pull/921

I read that :

feel free to make a unitTests folder at the root level of OF - that is where all unitTests will go.

But didn't find them.

The travis service will permit to get the little badge we know well :

Also, why not adding some spec test, to ensure future development of features. I'm currently reading : http://www.squidoo.com/cplusplus-behaviour-driven-development-tools#module124841511 http://sourceforge.net/apps/mediawiki/turtle/index.php?title=Turtle

For those who want some other nice reading :

http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle

The travis test is here, currently it is a total fake, but I just proposed the idea of a test driven development http://travis-ci.org/#!/soixantecircuits/openFrameworks/builds/854259

[bilderbuchi]

Argh, a firefox crash ate my long reply, so please forgive my ultra-brevity while I read your links:

someone offered a CI server once: http://forum.openframeworks.cc/index.php/topic,5340.msg26506.html#msg26506 automated testing and building is definitely on the radar, some restructuring lately is aimed at facilitating this, so your input is definitely welcome. automated project generation on the roadmap is part of this, for example. btw, how knowledgeable are you with CI/automated testing/building?

unit tests are under consideration, the folder is not there simply cause nobody made it yet. one thing complicating stuff with OF is that many functions have visual/graphical output, so a testing framework would need to compare pictures. personally, I wonder if we, when we set up a testing harness, have to write unit test for (nearly) every single function/class in OF or if this can be done automatically.

so, posting before it crashes again, I'm sure other people will pitch in, too.

Edit: Is this our build script which exits with 0 even though there are errors, or is this from travis or whatever? ./install_dependencies.sh must be executed as sudo. passwordless sudo seems to be available on travis.

Edit2: you should probably work with the OF develop branch, not master. Many things have changed by now.

[bilderbuchi]

One thing I think will be a big issue is that Travis only seems to offer Ubuntu 32bit as build environment, but OF is heavily cross-platform. I think it's impossible to test linux64, Windows, macOS versions - this would be a k.o.-criterion I guess. Edit: seems they plan multi-OS support for second half of 2012...

Also, it seems to be heavily ruby/web oriented...

[joshuajnoble]

This is just my opinion but I'm not sure I see how this would really be all that useful. Business logic, server connections, DB scripts, all these things are definitely testable, but I'm not sure how you would test "image loading works", or "vertices are out of order" without writing something automated that compares a screenshot to a screenshot of the running application pixel by pixel. That's not that hard to do trivially but I have a hard time imagining that it's really worth the effort for all 5 platforms that OF supports. That doesn't even get into testing "sound is ok", "geometry shader uniforms work properly", etc. This might be part of the reason that TDD is really popular in the web world but not in the games world.

[gabrielstuff]

Hey ! Thanks. Well i'm not a game developer, i'm just a happy coder.

• "image loading works" ----> loading a file should give access to some function, so we could test that we get the access to this function, what ever images are loaded.

---> it is like upload, we do not verify that the uploaded photo of the dog is really a dog, but we verify that the avatar that photo is attribute to does get a photo and the photo that we show is not the default image nore a broken link

• "vertices are out of order"

---> this one is tricky are require a default image that we use as reference. We have to compare as you remark. Well, it's not hard but it takes time.

• "sound is ok"

---> well, we can test that we can load a file, access the file, gor at second 00:50 and check if value of volume or FFT is as we should expected.

• "geometry shader uniforms work properly"

---> hum, i do not know :)

Well, these cases are not the easiest and as always we can begin with littles ones, and thus avoid regression.

As for Travis ! I bump them about plateform issues :), see what they got to give.

Thanks

[bilderbuchi]

Well, we still have to make sure images are loaded correctly, right? :-) Wouldn't do to have people have blue faces cause the channels were switched, or upside down, or something. So I fear it's not as easy as you imagine..

[gabrielstuff]

hum, well just for images, don't we have to check pixel order on the R G B A channels with the corresponding ARRAYS that we had generated and we know are well formatted. Again, there are a bunch of work to do before even beginning verifying :

• color,
• sound,
• animation,
• 3DMesh model loading,
• Bump texture,
• ...

Lot's of function are related to inside manipulation and not concerning display stuff.

g

[joshuajnoble]

I don't really have any objection to this, and I doubt anyone else does either, but there are other things that I'd rather have the core team spend time working on, and I'd rather be working on other things. If someone, for instance, you, wants to handle this, that'd be really awesome and it could be really helpful.

[bilderbuchi]

Yeah It sure would be useful to have, I only wanted to point out that there would be some things which would be difficult to test - the question is how useful it would be to have test only for parts of OF? So as joshua said, if you want to go ahead and start working on this and find out how far we can take it, please do, it is appreciated.

The automated builds, though, that's definitely on the table afaik (but cross-platform is a must). Probably @arturoc can comment further on this.

[bilderbuchi]

While we're at it, regarding the multi-platform problem, it probably makes more sense to use a Jenkins server (which is cross-platform). I found a hosted Jenkins server solution which is free for FOSS projects, and also has github support etc., apparently can even do Android via emulator? http://jenkinshosting.com/

[jonbro]

I don't know how applicable it is, but the processing.js team has a nice set of unit tests. Part of them do pixel comparisons. While it is really nice there, I don't know the complexity of moving those features to OF. I would love to see them moved though.

There is also teamcity, which I believe is multiplatform: http://www.jetbrains.com/teamcity/

[diasbruno]

edit: forgot to ask if someone is already working on this. (?)

hi, i think it would be really nice to have unit tests on OF. in some cases, it can be a place to learn some tricks and it should make easy to tests fixes on cross platform, tests new features and it can make easy to test on other platforms.

i've started a new app (probably, it's a good way to make it testable in all platforms). it uses cpptest (simple to use). also, the source is included, so we don't need to install it - but it's good to have. :)

https://github.com/diasbruno/ofCoreTest (works as a normal OF app - clone in myApps or devApps). http://cpptest.sourceforge.net/tutorial.html#tutorial_screenshots

for now, it's xcode only.

maybe later, we can move to cloud service.

[bilderbuchi]

afaik, nobody is working on this at the moment. yes, tests (not necessarily unit tests) would be awesome to have. Right now we have a build server, but this just checks if stuff gets compiled, and doesn't check functionality. While unit tests are suitable for math etc., how do you propose solving the problem of testing for things which are not easily testable with unit tests (sound, images, etc, see above discussion)? btw, what did you base your choice of cpptest on, instead of one of the other testing frameworks out there? I'm curious, I did some digging a while back, and there's sooo many... :p

I think the major question is: who will write all the tests? ^^

edit: probably it's worthwhile to check how Processing.js does testing, as @jonbro indicated above.

[diasbruno]

the sintax is very commom in a lot of frameworks in ruby, python, java, flash!! and it was easy to setup and works everywhere (also tested with cppunit). :)

there is also CATCH which is on github, it is for c, c++, obj-c. it's really good, but the syntax looks quite different. so, i though it would be better to use cpptest.

it's small, simple, popular...so, it looks good. if someone has experience with another test tool, more robust, or whatever..we can try it. :)

i think we can easily tests all the small pieces, vectors, allocations,...maybe, we can find some stuff...maybe not. for things more complex to test, probably the best way is to write small scenarios, since we can use the 'testApp'...they don't need to be big scenarios, they just need to follow the script...ex.:

load image locally | fail loading locally load from buffer | fail loading from buffer load from url | fail loading from url

so, i think we won't spend much time testing and get the benefits.

i have some time available, so i think i can start. :)

i'll take a look at processing.js...

[bilderbuchi]

ah ic. also take a look through the forum thread you posted in - jonbro elaborates on what he did in more detail (he apparently used gunittest), and there are some keywords like "reftest" which sounds like what we could do with pictures etc. Also, we have to think how to deal with the cross-platform nature of OF.

I think it's ok/best to put all the test-related material in a /tests/ folder in the openframeworks root directory. I'll also try to take a look at processing.js tests.

Great to see some movement on this at last! Now that the build server is getting more and more stable, this is the next step!

[diasbruno]

yep! i saw the jonbro's branch with gunittest...it's looks great and i didn't know what was the status of this discussion...so i decided to make a new idea that could be simple enough to implement this...without dependencies on comp. libraries and stuff, and everybody can 'clone' and then just write the tests. (hope i'm not doing something wrong :) ).

i think it would be better to write as an app, cause it uses the compiled version with everything on OF, so we don't need to rewrite configuration and whatever...

edit: this is not difficult to setup. so, we can discuss about which solution can be used, than we can move the app to the root/test folder.

i'll look in more detail about gunittest from jonbro and if it's looks better than cpptest, we can continue his work...

[jonbro]

I had totally forgotten about that branch! back from the dead! So I don't have a great memory of why I picked GUnitTest, and I don't know what the maintenance level of that is.

Also, I am not really sure what processing.js is doing these days, but their integration testing back in the day was super awesome, basically pixel checking against a set of expected images. Mozilla also does this type of testing for their rendering engine: https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests

Good luck with this everyone!

[diasbruno]

@jonbro, this is great! thanks.

i picked cpptest for the simplicity and convention:

class ... : public Test::Suite {
public:
    setup()...
    teardown()...

    test_a()...
    test_b()...
}

since this integration will take some time, i think we can use cpptest for now, and move later for gunittest if needed.

suggestions?

[bilderbuchi]

I think most testing frameworks have more or less the structure above. Yes, I think the best course of action would be to look at a couple of testing frameworks, choose the most suited one and go with it. maybe @kpasko can contribute some wisdom and/or practical experience, I seem to remember him saying in Detroit that he used some similar approach?

[bilderbuchi]

Another additional idea: We got a load of examples already. maybe we can use/recycle those as reftests somehow? That way we could probably get significant code coverage.

[kpasko]

I've predominantly used pytest, for the following reasons. Firstly, I'm usually building with SCons, so the build dev is python the begin with, plus python is easier to script little quick tests and can leverage modules easily (sys/os, easy regex, etc). a simple 'py.test-2.7 --tb=short -v' and it globs all test_regression.py files in any subdirs and executes them, reporting any failures etc. These are just reported to console, but again since its python, we could easily set it up to notify/network/post the results or whathaveyou. As a side note, recently, all of the C++ code I'm using has been exposed to Python, which makes it even easier to create test runs for unit/regression testing. This is convenient, but certainly not necessary; before this, we just used subprocess.call for a command line with the test options. Anyhow, not sure if this is the right way to go for our needs, but just another option to put out there (plus perhaps motivation for me to stop slacking and finally expose OF to python.....)

[bilderbuchi]

thanks for you insight, Keith! yeah python is practical, I learned to really like it lately. :-)

[diasbruno]

@bilderbuchi great idea! +1

@kpasko python is really great! well, if we don't have to write lots of extra bindings to expose Of to python..it would be really nice. i'll read more about pytest.

also, the customization of the output can be done in cpptest. so, i think we have great options to go with.

[diasbruno]

@bilderbuchi i'm still trying to find a better way to make this automated tests work. if anyone have time to take a look, it would be awesome.

this should be a test for adding new features. (also, this should help with #1847, but i'm still working on it.)

https://github.com/diasbruno/ofCoreTest/tree/feature-ofVec-templates/src/template-ofVec/

[diasbruno]

[update]

i'm working on automated testing for the math core.

https://github.com/diasbruno/ofCoreTest/tree/templating-of-math-core/src

[bilderbuchi]

hey! great work, I'll try to take a look soon, hopefully over the weekend.

[jonbro]

Here is the blog post describing how unity handles its graphics integration testing: http://aras-p.info/blog/2011/06/17/testing-graphics-code-4-years-later/

[bilderbuchi]

awesome, pretty useful, thanks for sharing!

[diasbruno]

wrote a new project.

https://github.com/diasbruno/oftest

now it uses 'make' to run all tests.

running on osx.6.8, and it's seems to be running ok on linux and linux64 (tested on arch linux).

[bilderbuchi]

great to see you're still on this, and see progress! I still have to check it out and give it a spin :-/ I still think we should have tests in OF, so it's good to see that something starts to take shape!

[diasbruno]

yeah, finally got sometime to work on this...

it's now running on travis-ci, but i'm still trying to set up the environment to build. https://travis-ci.org/diasbruno/openFrameworks/builds/7304578

[bilderbuchi]

ha, travis-ci integration is awesome! I was hoping to get this one day, but always thought that compiling OF and all the tests takes too long and travis (meant for more web/script based stuff afaik) would time out your build. is this not the case anymore? last time I checked I think not even OF lib compile time fit into the permissible build duration.

[diasbruno]

it seems to be compiling well, close to get it working. the OF library is getting compiled, just needs to set up travis environment to run all tests. :)

the makefile was made with console stuff from codeblock on linux. when it works, i'll try to make it use the .../OFcompiled/project makefiles.

[bilderbuchi]

are you using OF master or develop branch? the new makefile system in develop is pretty powerful...

[bilderbuchi]

btw as you are still collecting the tests: in light of a possible future integration into OF, I think it would be better to name the test cpp files e.g. testsOfRectangle.cpp or ofRectangleTests.cpp instead of ofRectangle.cpp to avoid the duplicate filename with the "real" ofRectangle.cpp - could lead to some ugly linker/IDE autocomplete/discovery issues.

[diasbruno]

develop. i didn't use them just for simplicity (take the log from codeblocks and make it just work on linux). but i'll try make use of them.

sure. this could be a problem. maybe: test_of*.cpp?

[bilderbuchi]

test_of*.cpp: yes, sounds good.

[diasbruno]

up and running on travis-ci! https://travis-ci.org/diasbruno/openFrameworks/builds/7390042

[bilderbuchi]

great job! some strange things, though: you got linking failures, but the tests succeed and the build passes: https://travis-ci.org/diasbruno/openFrameworks/builds/7391549 something must be wrong there.

[diasbruno]

i need to change the exit status for travis, so it will invalidate the current build... must be when syncing tasks (i think)...it must copy all the libs from 'export' before it builds all the tasks...

[bilderbuchi]

@arturoc @bakercp do you see an easy way to adapt this to use the new makefile system to run the tests? Some of the stuff in here feels like it duplicates a lot of the makefile build process. Also, I'd love to be able to make tests on the OF lib to run the tests. also, @diasbruno, is the seemingly heavy dependence on ruby because it's a tool you're familiar with, or because this is necessary for travis to work correctly?

[bilderbuchi]

also, pinging @benben - I think you will find this interesting.

[bilderbuchi]

generally, I really like where this is going. Btw, is it easy to add coverage reporting later on? that will be useful when we start to add more tests.

[diasbruno]

i still have the previous implementation with make (removed just for clean up),...i tried to implement with the OF makefiles...but there are some rules that i couldn't find a way to make it work well :/

i will put the old branch back later...

i was thinking to write the apis to work with reports and analysis in ruby (for this 'study'), and travis has support for everything, so it would be nice to keep everything in one place, with ruby and rake, in this case...but, of course, any setup would work as well, i just though it would be more consistent. ideas are welcomed.

no no, i'm not a ruby dev, but it's nice to write in ruby - has good docs... :)

[diasbruno]

[update] clean up everything and now running with the OF makefiles. :) need to fix travis-ci.

[diasbruno]

lastest build: https://travis-ci.org/diasbruno/openFrameworks/builds/7969062

[bilderbuchi]

nice to see this up and running! those linking Fails are still there, though, that's curious...

[diasbruno]

maybe, it's because of that ofParameter friend typename - class, blah blah blah...but, i'll check it...

there are two things to do, and them, people who can help testing can start use it:

1 - create more logs. now it will generate logs for of.compiler.log and of.linker.log. should be:

• of*.compiler.log
• .linker.log
• .compileStr.log = g++ src/test_....
• .tests.log

2 - config.h

it's getting better... :)

[danoli3]

For Windows testing we can use http://www.appveyor.com/

[danoli3]

After seeing @bilderbuchi 's comment about automated testing.. I setup a quick test branch for this:

• Branch: https://github.com/danoli3/openFrameworks/tree/Automated/Travis-OSX
• Travis Status: https://travis-ci.org/danoli3/openFrameworks/branches
• I had to remove the documentation compiler travis code (not sure why this is a travis thing in the core).
• Testing iOS and Testing OSX.
• Using the template/emptyExample project for both

Limitations we have with Travis are the time per build (max 1 hour). So I thought for now the emptyExample is perfect. It validates the core has not been broken by a commit.

Testing this I created another branch from this, one which should soon fail.

• https://github.com/danoli3/openFrameworks/tree/Automated/Travis-CoreFail
• https://travis-ci.org/danoli3/openFrameworks/builds/68468345

[bilderbuchi]

Limitations we have with Travis are the time per build (max 1 hour).

Ah nice, when I checked Travis in the beginning, this was 10min or so, so I discarded the idea of using it for OF.

Nice work! One thing: could you leverage the make system instead of xcodebuild on the travis job? (this does work in osx already, right?) That way we could also detect errors in our makefiles, and the build code would be more homogeneous (could easily include linux, too).