Write up a new Doxygen page on how to use the new unit test structure

[lossius]

Quoting a recent mail from Tim to the mailing list:

In the process of dictionary rewrite I came realize for myself something that Théo and Nathan were also concluding: that our current process of using the ruby environment to execute unit tests is to viscous. So we now also have the 'feature/unittest-rewrite' branch which is a branch off of the 'feature/dictionary-rewrite' branch. Probably this two branches should be merged at this point even though there are still a few hiccups in the new unit testing implementation.

A few notes about this:

• the new unit testing is not mutually exclusive to using ruby, the ruby stuff still works too
• in terms of a red-green-refactor approach, the new way is dramatically faster and more frictionless
• you never have to leave xcode now: just build and unit test failures show up as errors and will jump to where you need to go

_To test it (which may require patience) you can:

1. Switch to the 'feature/unittest-rewrite' branch on Core
2. Open the Foundation project in Xcode
3. Clean
4. Build
5. Observe that we currently have a couple of failing test assertions_

I was not familiar with the "Catch" project. Thanks for sharing that, it looks very well designed and I need a little time to look at it more deeply. For others on the list wanting to look into it I recommend starting with https://github.com/philsquared/Catch/blob/master/docs/tutorial.md

One aspect of Jamoma's test system that is different than catch is that the unit test is exposed as a message in the deployment code. The idea is that if there is a Jamoma installation out in the wild and it is not functioning that it should be possible to run tests on deployed code on the deployed system environment in addition to the sterile development environment on which the code was conceived.

@Tim: Would you be able to add a page to Doxygen that explains:

1. How to set up tests according to this new system. Do anything particular need to be done in the Xcode project or similar?
2. How to run a unit test
3. Do this also imply that we need a new strategy for running all of the unit tests for all of Core? If so, how do we do that?

The result of this could also well be presented as a blog post.

Thanks!

[nwolek]

Moving this assignment to me. I am working to make the build & test function in every project that has test. It would natural for me to write up directions when that work is done.

[tap]

Thanks, Nathan!

--••• •••--

• •• --

On Wed, Nov 20, 2013 at 11:22 AM, nwolek notifications@github.com wrote:

Moving this assignment to me. I am working to make the build & test function in every project that has test. It would natural for me to write up directions when that work is done.

— Reply to this email directly or view it on GitHubhttps://github.com/jamoma/JamomaCore/issues/166#issuecomment-28909332 .

[nwolek]

I started organizing my notes for this today. It's going to be a relatively long blog post. Could easily become a follow up paper to the SMC 2012 paper. http://jamoma.org/publications/attachments/smc2012-testing.pdf

[nwolek]

I am working on this in Evernote. Viewable here: https://www.evernote.com/shard/s152/sh/e66e74ce-088a-413e-8afa-cfad817712d5/943ffa6863923fbe3cfed01bb49b9af3

If others want to collaborate, we may want to find a better platform. Let me know.

[tap]

I'm happy to be a reviewer when you get to that point.

best, Tim

On Fri, Jan 3, 2014 at 3:40 PM, nwolek notifications@github.com wrote:

I am working on this in Evernote. Viewable here: https://www.evernote.com/shard/s152/sh/e66e74ce-088a-413e-8afa-cfad817712d5/943ffa6863923fbe3cfed01bb49b9af3

If others want to collaborate, we may want to find a better platform. Let me know.

— Reply to this email directly or view it on GitHubhttps://github.com/jamoma/JamomaCore/issues/166#issuecomment-31559871 .

[nwolek]

@tap: Could you take a quick look at the outline on and see if I missed anything for my narrative?

@lossius: Should I do a short blog post now and we'll talk paper later?

[tap]

Too early to quibble too much ;-), but a couple of notes...

• I think doing this, even just the outline, is way easier for collaboration if we do it as a latex doc under version control.
• 2.4: not just running a ruby script to call the test message -- you could send the test message to a Max object wrapping the code too, for example. Or Théo had the test methods for TTValue called when the Foundation initialized for a while when running on other targets / platforms (e.g. iOS)
• 4.3: It's not just Xcode that will point you to the correct spot -- Visual Studio will do it too. I haven't tried with Code Blocks or other IDEs but probably it could be expanded further.

digression #1: "makefile now looks for test.cpp file in a given project - if present, it runs" -- most of these test.cpp files are the same, right? or nearly the same? so we aren't dry and we could (should?) make this simpler and provide a default "test.cpp" if one isn't present. Guess that should be a new Github ticket.

• 7.2: We do define the tags with a preprocessor #define in 99% of cases, but this is relies on using the macro system throughout (e.g. TT_AUDIO_CONSTRUCTOR for the the constructor instead of e.g. "TTAllpass1a::TTAllpass1a"). It's subtle but for a paper it should be specific. I guess for a blog post it can be glossed over (sorry -- I was thinking paper until just now!)
• 7.4: Again, it's not just Xcode -- for a blog post its fine (maybe even preferable?) but for a paper I would say IDE.

I don't see anything explaining why you need to specify the tags when running the test, which feeds into talking about dependencies (which is the traditional thorny part of automated testing).

For a paper we should also list the shortcomings -- In Xcode for example, the number of messages from the invocation to Make is truncated at 200 lines (maybe there is a way to change that?) -- which means you might not see all of the results or you might not see any results if your code generates too many warnings (including deprecation warnings).

We should always strive for warning-free code, but at some points in the development cycle (e.g. migrating to newer APIs) having a lot of warnings is just part of the process and this trips it up a bit.

Additionally, for a paper, we should list future work. One place we try to expand is in the syntax for writing tests to make it more expressive and less tedious. We could initially try this with some fancy macros or templates. The idea being that our tests currently look like an over-decorated Christmas tree with too much tinsel which detracts from our examination of the tree itself.

Finally, for a paper, we should talk about how this system currently uses the actual Foundation code -- which is presumably minimal. This is as opposed to using "mocks" (classes which are like stubs for the real thing, but maybe do something minimal). We could also create mock classes for some testing scenarios -- the NodeLib for example where maybe we don't want a test to actually require unbounded network access -- but for most of what we are doing the basic classes are already minimal.

As hinted at earlier, maybe we could also do something in the testing context to improve logging. Maybe set a flag in Foundation that redirects most messages to a separate log file that we can examine?

Hope this helps!

best, Tim

On Tue, Jan 7, 2014 at 9:48 AM, nwolek notifications@github.com wrote:

@tap https://github.com/tap: Could you take a quick look at the outline on and see if I missed anything for my narrative?

@lossius https://github.com/lossius: Should I do a short blog post now and we'll talk paper later?

— Reply to this email directly or view it on GitHubhttps://github.com/jamoma/JamomaCore/issues/166#issuecomment-31754776 .

[lossius]

Hi, I just had a quick look, and it seems good! Depending on how we want to frame it:

• If it is to be a paper I agree that it makes sense to move it to latex and versioning
• If it is to become part of the documentation, we should probably add it to the doxygen documentation

Is there sufficient "new research" in this that it warrants a paper?

[nwolek]

@tap Spent a good part of the day learning latex. I had successfully avoided it until Tim’s challenge. So thanks!

@lossius I have seen less "new research" than this shared in the ICMC proceedings. Remember four pages constitutes a "short paper" - http://www.icmc14-smc14.net/submissions/papers.html

[tap]

Another option... It could be that a broader paper about Jamoma and/or a paper about our process could include a large section on the evolution of our testing systems.

best, Tim

On Wed, Jan 15, 2014 at 3:56 PM, nwolek notifications@github.com wrote:

@tap https://github.com/tap Spent a good part of the day learning latex. I had successfully avoided it until Tim’s challenge. So thanks!

@lossius https://github.com/lossius I have seen less "new research" than this shared in the ICMC proceedings. Remember four pages constitutes a "short paper" - http://www.icmc14-smc14.net/submissions/papers.html

— Reply to this email directly or view it on GitHubhttps://github.com/jamoma/JamomaCore/issues/166#issuecomment-32419653 .

[nwolek]

Researching and trying LaTeX some more and I think there might be a better workflow for writing projects:

1. use markdown as the preferred format under version control (http://whatismarkdown.com/)
2. use pandoc to convert between necessary formats like HTML, textile and LaTeX (http://johnmacfarlane.net/pandoc/README.html)

Side benefit: markdown is used / understood by Github

Thoughts?

[nwolek]

I am now working over here: https://github.com/jamoma/JamomaDoc/blob/dev/Drafts/buildAndTestOutline.md

[nwolek]

Blog post on this is done over here: https://github.com/jamoma/JamomaDoc/blob/dev/Drafts/buildAndTestBlogPost.md

Any chance I could get someone to read, proof and post to the blog? @lossius ?

[tap]

It looks great, Nathan!

best, Tim

On Wed, Jan 29, 2014 at 4:20 PM, nwolek notifications@github.com wrote:

Blog post on this is done over here: https://github.com/jamoma/JamomaDoc/blob/dev/Drafts/buildAndTestBlogPost.md

Any chance I could get someone to read, proof and post to the blog? @lossius https://github.com/lossius ?

— Reply to this email directly or view it on GitHubhttps://github.com/jamoma/JamomaCore/issues/166#issuecomment-33638197 .

[lossius]

Brilliant, Nathan, I've added it to the blog and pushed, so it should be online in a few minutes. Would it be an idea to also add some of this info to Doxygen so that it becomes part of the API documentation? That does maybe not need to be as historically minded, but could contain all info necessary to understand and make use of testing?

[nwolek]

Thanks for the positive feedback!

Yes, I agree some shorter format should be good in the Doxygen pages too. I will add that to my ToDo list.

Again, I think there is enough here to make a conference paper, so I was working toward that goal. Hopefully you guys can see the structure now and where to extend for that goal. April 1 is the submission deadline, so we can talk during Bergen.

@lossius - how was the transition from markdown to textile? easy enough?

[nwolek]

Trying to solve another issue in JamomaDoc in tandem with this one: https://github.com/jamoma/JamomaDoc/issues/3

[nwolek]

Finished update to "chapter" on unit testing here: https://github.com/jamoma/JamomaDoc/commit/aa9295441675e4d67111d64a762107e6c5ff5a55

[lossius]

(clap)

[nwolek]

@lossius: Right now it's finished on the dev branch. If I merge that to master, will it find it's way onto the API website? Can't remember it this has been automated.

[lossius]

Try, and see what happens. We did embark on this in Albi, but I don't recall if we finished it of. If not, we should finish it wile Henrique is in bergen, and also we need to properly document how these automations are done.

If you see that it is not being updated, please create a new ticket for me to update the API at the server. You should probably merge the other repositories as well to master, as the script is likely to make use of the master branch of all repositories when building the documentation.

[nwolek]

I merged to master in the JamomaDoc repository yesterday at 12pm eastern USA. It is not updated this morning. Or do I need to fold into master on the umbrella repository.

I agree we should nail down this automated stuff w @hems in Bergen since he is such a scripting wizard. :) There are several scripting things over here: https://github.com/jamoma/Jamoma/issues?state=open

Sounds like it should be on the agenda.