Write fuzzers for asset files and network communications.

[peterclemenko]

Fuzzing is a useful tool in security that will intentionally throw bad data at a target to try to crash it. Write fuzzers for the various data file types used and the various network communication protocols and use them to check for security against malformed files/packets of data.

It would be suggested once the fuzzers are ran against the existing code base and problems are fixed to then run a fuzzer against revisions before and after merge, along with before tagging for release in order to ensure security and prevent regressions.

[jonnenauha]

This sounds pretty nice, maybe you could help out with it? I think at first it could be just a collection of assets that have been manually broken in some sense, just to get it going.

This is a bit related that we would need some kind of testing module that would run set of unit tests. I've been looking at QTest http://doc.qt.digia.com/4.7/qttest.html for some kind of thing. This could run a client + server, manipulate scene, put asset refs in and check all kinds of things on client and the server then report.

[peterclemenko]

This is what I'm learning on the side, and it should be pretty handy: http://peachfuzzer.com/

It's one of the main fuzzing tools out there, and it supports file fuzzing. For network packet based fuzzing; Scapy, and possibly Metasploit would be the tools of choice. I'll have to learn more about how to make the fuzzers, but, it may take a while because of school. I'm also a bit more focused on the build fixes right now, as my build environment on Windows is not playing nice with the build for some reason.

I'll gladly help when I get the time, and once I have some time, the first thing I'm going to do will probably be fuzz the hell out of the networking using Scapy. I'm in the process of learning peach fuzzer, so it will likely take a while before I can get any file format fuzzing done (school + other real life commitments are getting in the way of my dedicating full time).

[peterclemenko]

Also, for unit testing; Google Mock and Google Test may be useful.

http://code.google.com/p/googletest/ http://code.google.com/p/googlemock/

YUI looks like it may be useful for JS: http://developer.yahoo.com/yui/yuitest/

pytest looks like it may be useful for Python: http://pytest.org/latest/

xUnit and XmlUnit may be useful for XML: http://reflex.gforge.inria.fr/xunit.html http://xmlunit.sourceforge.net/

Wikipedia has a nice list of unit testing tools in general: http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks

[erno]

For the assets, you don't need any fancy test infrastructure to get started with the low hanging fruit. Just have a scene file with fixed asset paths and replace those between test cases. I had a go with Radamsa which is very easy to use (no configuration necessary since it's sample based) before the ogre-safe-nocrashes branch, there were lots of crashes.

[peterclemenko]

I'll take a look in to that tool, I have never heard of it before.

[erno]

Also along this theme, Coverity offers free scanning for open source projects (http://scan.coverity.com/start/). They're one of the best static analysis tools. See eg http://www.altdevblogaday.com/2011/12/24/static-code-analysis/ for a testimony from John Carmack.

[jonnenauha]

We should really get into that, include at least one guy from each company etc. to the users to begin with. Looks very interesting. Who wants to take up registering and adding people there? :)

I've ran static code analysis before to tundra repo and made few fix commits that came up from the results. Sadly none of the really good tools are free (understandably, people need to make money too), but as i understand what they are offering here is the full suite via a web interface, not the actual tool. Looks great.

[juj]

Oh erno, I did not know you actually had used radamsa on Tundra. What did you do with the results? Are the test case files available?

[erno]

I fuzzed ogre .mesh files and watched ogre just crash and burn all the time. I have the cases on the playsign laptop somewhere but it would be better to make a new run with the ogre-safe-nocrashes branch.

IIRC loading the meshes to Tundra caused (GPU driver?) lockups so I switched to just using OgreXMLConverter but that also crashed all the time.

[jonnenauha]

I guess the question there is if the data validation checks should be implemented in Ogre or in Tundra. Afaik this kind of code was the reason why we forked ogre and hence the name too :)