Error using PES learning rule in Direct mode

[bjkomer]

I'm wondering if it should be possible to use the PES learning rule on an ensemble running in Direct mode. I know there are no decoders to modify, but it would be nice if something functionally equivalent could be done. The use case would be debugging models with learning that take a long time to build by running them in Direct mode instead.

Right now this just produces an error

 import nengo
 import numpy as np

 model = nengo.Network()

 with model:
   error = nengo.Ensemble( 100, 1 )
   pre = nengo.Ensemble( 100, 1, neuron_type=nengo.Direct() )
   post = nengo.Ensemble( 100, 1 )
   error_conn = nengo.Connection( error, post, modulatory=True )
   nengo.Connection( pre, post, function = lambda x: np.random.random(1),
                     learning_rule_type=nengo.PES( error_conn, learning_rate=1.0 ) )

 sim = nengo.Simulator( model )
 sim.run(5)

Traceback (most recent call last):
  File "direct_mode_pes_issue.py", line 15, in <module>
    sim = nengo.Simulator( model )
  File "/home/bjkomer/nnroot/src/nengo/nengo/simulator.py", line 110, in __init__
    self.model.build(network)
  File "/home/bjkomer/nnroot/src/nengo/nengo/builder/builder.py", line 34, in build
    return Builder.build(self, *objs)
  File "/home/bjkomer/nnroot/src/nengo/nengo/builder/builder.py", line 74, in build
    cls.builders[obj_cls](model, obj, *args, **kwargs)
  File "/home/bjkomer/nnroot/src/nengo/nengo/builder/network.py", line 63, in build_network
    model.build(conn)
  File "/home/bjkomer/nnroot/src/nengo/nengo/builder/builder.py", line 34, in build
    return Builder.build(self, *objs)
  File "/home/bjkomer/nnroot/src/nengo/nengo/builder/builder.py", line 74, in build
    cls.builders[obj_cls](model, obj, *args, **kwargs)
  File "/home/bjkomer/nnroot/src/nengo/nengo/builder/connection.py", line 196, in build_connection
    modified_signal = model.sig[conn]['decoders']
KeyError: 'decoders'

[tbekolay]

This has been a TODO of mine for literally 4 years, since implementing PES in 1.4.

One approach that comes to mind at the moment is for the learning rule to set up a new signal that keeps track of a rough input to history of previous error mapping. I.e., the connection computes f(x) without learning, and f(x) + g(x) with PES applied, where g(x) = running total of error at x. There'd be some tweaking in the details of this (how are you discretizing x, how are you interpolating between nearest points in your n-dimensional space) but in principle it seems simple.

[studywolf]

This is pretty much what the slotine adaptation does, acting as the error integration term in a nonlinear system. I don't know a way to really do it without function approximation, are you talking about a lookup table approach @tbekolay ? They do it using either some specified set of functions that can capture the form of the error but the new signal to keep track of error could be done using a neural population (bias/dynamics adaptation etc).

[tbekolay]

Yeah, I'm basically talking about a lookup table with interpolation so it works in continuous space. Would be pretty memory-expensive as dimensionality goes up, but for up to, say, 3D it would be not bad. Wouldn't generalize, of course, but I sort of wouldn't expect a direct mode learning thing to generalize (which is why I avoided function approximation in the first place -- if you're doing that you probably aren't looking at direct mode but maybe something like population mode).

I could see a direct mode learning rule maybe taking in some specified set of functions, that seems pretty 'direct' to me. That might be a new learning rule type though, I dunno how that would be specified with PES.

[hunse]

In the mean time, could we tighten up that error message?

[tbekolay]

We discussed this at the last dev meeting. Still a good idea! But a pretty big job, so made https://github.com/nengo/enhancement_proposals/pull/9 for it. Then I made #1192 to tighten up the error message. Discussion can continue in either of those places!