Plan for analysis section

[tcstewar]

Here's my current plan for the analysis section (which I'll hopefully have a draft for tonight):

• we can run the benchmark as it is on nengo, nengo_ocl, and nengo_spinnaker
  • bar plot, one value for each hardware, basically with all values the same
• this basically just tests the difference due to the different neuron models, since all are running with the same number of neurons
• so, let's set the number of neurons to however many that hardware can run in realtime
  • and let's vary the number of joints and see how that affects things
  • that gives a more informative plot, but it's still not controlling for power consumption
• so, let's set the number of neurons to however many that hardware can do with a fixed power budget (say 0.5W).
  • for the nengo and nengo_ocl implementations, this is tricky to do since that hardware simply can't run at that low power. So, instead, we take however many it can do in realtime and divide by the power consumption
  • this now shows SpiNNaker way better than the competitors.

After that set of data, we can start showing other comparisons:

• varying the amount of delay
• varying the number of neurons and the number of joints
• whatever else seems interesting

And then a concluding thing pointing out that we've now defined N benchmarks, each of which looks at different aspects of the hardware. If people have new hardware that is good in particular situations, they can adjust the core benchmark be specifying different parameter distributions. This lets people be explicit about what their hardware is good at, while still using the same basic benchmark framework.

[celiasmith]

Can you get power estimates for the ocl and regular backends that exclude all the stuff like running monitors, operating systems, etc.? this might be more doable with ocl... in any case, i think to do this convincingly, you have to be overly fair to the cpu/gpu so no one can complain about the comparison... and then still show spinnaker doing better... which presumably it will :)

[celiasmith]

I like the overall structure though... really drives the point home. :exclamation:

[studywolf]

awesome.

other possible things to vary could be

• delays on how long before control signal is applied (we've been only talking about feedback delay, right?)
• having a multiplicative noise on the control signal (norm(u) * value_from_Gaussian)

i'm trying to think of things that might vary between hardware,

• complexity of the function it's able to account for..?
• if learning is onboard then maybe something like how quickly the external force can be changing and still be compensated for...?

...! not sure. thoughts.

let me know if i can help!

On Thu, Aug 13, 2015 at 5:01 PM, Chris Eliasmith notifications@github.com wrote:

I like the overall structure though... really drives the point home. [image: :exclamation:]

— Reply to this email directly or view it on GitHub https://github.com/tcstewar/2015-Embodied_Benchmarks/issues/26#issuecomment-130843899 .

[celiasmith]

One thought I had is that we could possibly use the addition of the velocity information as an example of trying a 'different algorithm' on the same benchmark(s). Obviously it's not that different, but it's kinda like the small variations people do to convnets in ML... then we'd have

1. same algo simulated across diff neuro hardware
2. same algo across both sim and robot hardware
3. diff algo across ... whatever (mirroring 1 & 2, or both).

Just a possibility for demonstrating the flexibility of the method.

[studywolf]

for small variations, could also pass the sin and cos of each joint instead of the angle!

On Thu, Aug 13, 2015 at 5:48 PM, Chris Eliasmith notifications@github.com wrote:

One thought I had is that we could possibly use the addition of the velocity information as an example of trying a 'different algorithm' on the same benchmark(s). Obviously it's not that different, but it's kinda like the small variations people do to convnets in ML... then we'd have

1. same algo simulated across diff neuro hardware
2. same algo across both sim and robot hardware
3. diff algo across ... whatever (mirroring 1 & 2, or both).

Just a possibility for demonstrating the flexibility of the method.

— Reply to this email directly or view it on GitHub https://github.com/tcstewar/2015-Embodied_Benchmarks/issues/26#issuecomment-130857155 .

[tcstewar]

for small variations, could also pass the sin and cos of each joint instead of the angle!

Shoot, good point, I totally forgot about that! That'd be pretty nice to add (although possibly hard to describe to an audience that's not NEF-savvy....)