Randomize order

[FilipAndersson245]

I had the idea that maybe we should randomize the order of the time measurement. instead of abababababababab

we may have abbaababbababaab

this to reduce any bias that a is run before b etc.

[bazhenov]

Actually this was the case in tango prior to version 0.3. I removed this functionality because of 2 following considerations:

• in my experience randomizing only helps when there is an underlying source of bias you want to get rid of. Otherwise randomizing just lowers the sensitivity of a test (more random results – harder to spot a difference between algorithms). When running A and B in an alternating fashion both algorithms influences each other in the same way. At least in the experiments I've done
• current version of a test scheduler in tango (dev branch) has a parallel scheduler (-p option on a cli) which runs A and B in different threads, so algorithms doesn't influence each other. Unless there are some CPU package wide issues like thermal throttling or memory bandwidth starvation etc.

[FilipAndersson245]

Okay, did not try out the -p flag as it did not have any documentation with -h. Would it not introduce a bias if the work is executed on two different thread, in theory one could be allocated on a efficiency core and one on a performance core (eg intel).

[bazhenov]

Would it not introduce a bias if the work is executed on two different thread, in theory one could be allocated on a efficiency core and one on a performance core (eg intel).

Yes and no. P/E core scheduling is not an issue in my experience. In practice scheduler priority class is usually given to a thread at startup and doesn't change after. The more important issue might be the core allocation itself. Even if both threads are on P-cores it might be the case that one thread is allocated to a core pinned for system activity (like handling IRQs). In this case it will be a bias. This is why it is opt-in behavior.