Benchmark are currently failing

[budde377]

All benchmarks and most tests are currently failing and running suspiciously fast.

[budde377]

The fast execution was caused by the standard version of PHP on ubuntu being v.5.3 which doesn't have the build-in server that we use. However after installing that the benchmark-tests either fails or hangs and a lot of the regular tests fails.

[budde377]

After updating loc and making the delta vary between 10% and 20% with respect to the difference in linecount between paper and newly counted lines; the following tests fails:

• 3D Model: Around 15 pct coverage against 74 in paper
• Pacman: Around 10 pct coverage against 44 pct in paper
• Fractal viewer: Around 50 pct coverage against 75 pct in paper

[sema]

Are these benchmarks using any Ajax? We have made a big change on that department by not enforcing synchronicity any more.

[budde377]

No, but the fractal viewer seems broken and alerts the message ">>>275 0" on keypress. Also I haven't yet been able to test ball_pool or homeostasis because of very poor execution time and extensive memory usage.

[sema]

Some of the memory usage problems will be solved when we merge the real-sites branch into master.

On 18 September 2013 15:27, Christian Budde Christensen < notifications@github.com> wrote:

No, but the fractal viewer seems broken and alerts the message ">>>275 0" on keypress. Also I haven't yet been able to test ball_pool or homeostasis because of very poor execution time and extensive memory usage.

— Reply to this email directly or view it on GitHubhttps://github.com/cs-au-dk/Artemis/issues/44#issuecomment-24663631 .

[budde377]

The merge of real-sites did decrease the memory usage, and I was able to run artemis on homeostasis and ball pool. However ball poll tests failed with a 67% coverage against ~90% in paper. I also re-downloaded the fractal viewer which corrected the errors, and the tests now succeeds.

[budde377]

In pacman it would seem that the poor coverage is caused by loading src/js/pacman10-hp.3.js in "execute sequence" instead of "fetch page".

[sema]

As in the file is loaded at some "random" point while we are executing the page, and not before the "onload" event?

Can you track down how this file is fetched (is it referenced directly in the project or is it fetched through an XMLHttpRequest).

On 25 September 2013 13:13, Christian Budde Christensen < notifications@github.com> wrote:

In pacman it would seem that the poor coverage is caused by loading src/js/pacman10-hp.3.js in "execute sequence" instead of "fetch page".

— Reply to this email directly or view it on GitHubhttps://github.com/cs-au-dk/Artemis/issues/44#issuecomment-25078133 .

[budde377]

The file is fetches by appending a script element

a = document.createElement("script");
a.type = "text/javascript";
a.src = "src/js/pacman10-hp.3.js";
document.body.appendChild(a)

This above is executed after a timer has been fired, or at a mouseclick on a submit input element. If I force the page to add the tag when the index file is loaded (independent of document load event, see index.html ) the coverage will be approx. the expected from article.

[sema]

So is the low coverage because we never trigger this load event from Artemis? Because I would think that if we did trigger it (somewhere in our event sequences) then the end result would be approximately the same, give or take.

[budde377]

That could be it. I will test for that.