L260

[studywolf]

https://github.com/tcstewar/2015-Embodied_Benchmarks/blob/master/paper/paper.tex#L260

suggested

"Given the success of this approach for evolutionary robotics, we propose applying minimal simulation directly as one means of embodied neuromorphic benchmarking."

[tcstewar]

I think this is also a good excuse to slip in something about the extended discussion in #19... I've added a subsection break and put this in:

\subsection{Minimal Simulation as a Benchmark}

Given the success of this approach for evolutionary robotics, we propose
using a minimal simulation as a neuromorphic benchmark.  First, we note that
one important use of a benchmark is that by knowing how well particular
hardware performs on that benchmark, you can reasonably infer how well
that hardware will perform in other situations.  For example, if an image recognition
algorithm performs well on the MNIST hand-written digit recognition
benchmark, we can use that knowledge to guess that it may also perform well on
a different recognition task.  Of course, this inference will fail if that
algorithn has been specifically over-fit to exactly that one situtation.  For
that reason, it would be useful to have a benchmark that covers a large range
of variations on the task.  If the hardware performs well across that
variability, then it is more likely to also work in whatever new situation we
want to use it in.

To achieve this, we need software simulations of the environment for the task.
These simulations must be fast enough...

[studywolf]

suggested

"For example, if an image recognition algorithm performs well on the MNIST hand-written digit recognition benchmark, we can use that knowledge to guess that it may also perform well on a different recognition task. Importantly, the benchmark must have enough variation to prevent exploitation of a specific characteristic of the task and over-fitting. Such benchmarks will let us better gauge general usefulness, as hardware that performs well across such variability is more likely to also work in whatever new situation we want to use it in."