Generate and blend between patterns

[tcstewar]

Herbert Jaeger's conceptors thing has a nice demo where it generates a bunch of 61-dimensional periodic output patterns. The patterns were gathered from motion-capture data, and are joint angles of a person doing a bunch of tasks. The model ends up being able to smoothly blend between them.

https://www.youtube.com/watch?v=DkS_Yw1ldD4 Matlab sourcecode and raw motion-capture data is available here http://minds.jacobs-university.de/conceptors (see link to matlab code at the bottom)

This looks a lot like a really simple version of dynamic motor primitives, and fits perfectly into our various ways of generating patterns (by having some generic dynamics and then decoding out the pattern). Here's a basic model:

import nengo
import numpy as np

T = 1.0
dt = 0.001

t = np.arange(1000)*dt

pattern1 = 2 * t -1
pattern2 = np.sin(t*2*np.pi)
pattern3 = np.where(t<0.5, 2*t, 2-2*t)
pattern4 = -pattern3

def make_pattern():
    return nengo.processes.sample(1000, nengo.processes.WhiteNoise(T, high=2), dt=dt, d=T)

np.random.seed(3)
patterns = [make_pattern()[:,0] for i in range(4)]
patterns = [pattern1,pattern2, pattern3, pattern4]
patterns = np.array(patterns)

model = nengo.Network()
#model.config[nengo.Ensemble].neuron_type = nengo.LIFRate()
#model.config[nengo.Connection].solver = nengo.solvers.LstsqL2(reg=0.001)
tau = 0.1
omega = 2*np.pi
with model:
    system = nengo.Ensemble(n_neurons=1000, dimensions=3, radius=1.4, label='cycle')

    def cycle(x):
        a = 1
        b = 2 * np.pi * x[2]
        r = np.sqrt(x[0]**2 + x[1]**2)
        theta = np.arctan2(x[1], x[0])
        dr = 10*(-r**3 + a*r)
        dtheta = b

        dx = dr*np.cos(theta) - r*np.sin(theta)*dtheta
        dy = dr*np.sin(theta) + r*np.cos(theta)*dtheta

        return [x[0] + tau*dx, x[1] + tau*dy]

    nengo.Connection(system, system[:2], synapse=tau, function=cycle)  

    rate = nengo.Node([1])
    nengo.Connection(rate, system[2])

    readouts = nengo.networks.EnsembleArray(n_neurons=30, n_ensembles=len(patterns))
    for i in range(len(patterns)):
        def read_func(x, index=i):
            theta_p = np.arctan2(x[1], x[0])
            r_p = np.sqrt(x[0]**2 + x[1]**2)
            if theta_p < 0: 
                theta_p += 2 * np.pi
            index_p = int(len(patterns[0]) * theta_p / (2 * np.pi))

            return patterns[index][index_p] * r_p
        nengo.Connection(system, readouts.ensembles[i], function=read_func)

    select = nengo.Node([2.5] * len(patterns))

    for i in range(len(patterns)):
        nengo.Connection(select[i], readouts.ensembles[i].neurons, transform=[[-1]]*30)

    readout = nengo.Ensemble(n_neurons=30, dimensions=1)
    nengo.Connection(readouts.output, readout, transform=[[1]*len(patterns)])

One new feature that would be needed to be added is synchronizing the patterns based on current state. That is, when you blend from one pattern to the next, it needs to adjust the phase of the core oscillator to get near the phase in the pattern that leads to the closest output pattern of the current pattern. This is tricky, but I think can be done by learning a backwards model of the pattern. Maybe.

Also, one interesting idea for these systems is that the pattern itself could also specify the frequency of the oscillator. That is, the pattern itself can control the speed of the pattern (and change it at different times in the pattern).

[Seanny123]

How difficult do you estimate implementing this might be? Is this appropriate for a small course project or is this more like a journal paper?

[tcstewar]

I think it could be a small course project. By "learning a backwards model" I actually meant doing the standard NEF solving-for-decoders to do the backwards model (mapping from the dynamical state back to the angle of the oscillator), so that should be straightforward. Maybe. I sort of get the sense that it'll either work okay or it'll be way harder than I imagine....

[Seanny123]

Resolved via my SYDE552 project, whose code is at: https://github.com/Seanny123/nef-conceptors

But I can't close this issue for some reason?