222464
commented
6 years ago Hi!
RL is still an experimental feature, so I would recommend getting the latest from the "Pre-Release" repository over here: https://github.com/222464/EOgmaNeo
I have prepared an OpenAI Gym Cart-Pole demo. This should be helpful for figuring out how to convert to a columnar SDR. It does it manually by simply rescaling the input observations into the integer range of the input columns.
Quick rundown of the input format: The user defines how many input layers they want, and each input layer is a 2D grid of columns. Each column is a one-hot vector, so it is represented by a single integer index. This makes the tensor of the shape numInputLayers X width X height, where each element is an integer representing the index into a column.
How you transform into this representation is up to you. Preferably you will want to make it locally sensitive and preserve as much information as possible, but sometimes this isn't necessary and it will work well regardless. One of the simplest encodings is to simply take e.g. a scalar and rescale it into the discrete integer range [0, inputColumnSize). This is an inefficient encoding usually but if also works quite well and is simple to implement. This is what the Cart-Pole demo uses.
For tasks with image input, you will probably want to use one of the pre-made pre-encoders, like the ImageEncoder. It is also possible to encode completely arbitrary data with the KMeansEncoder, but it often requires pre-training since it can forget.
Actions are treated the same way inputs are, you feed in the last action taken as an input each timestep. You then retrieve the next action as a prediction, using getPredictions(...).
Here is the Cart-Pole example:
# ----------------------------------------------------------------------------
# EOgmaNeo
# Copyright(c) 2017-2018 Ogma Intelligent Systems Corp. All rights reserved.
#
# This copy of EOgmaNeo is licensed to you under the terms described
# in the EOGMANEO_LICENSE.md file included in this distribution.
# ----------------------------------------------------------------------------
# -*- coding: utf-8 -*-
import eogmaneo
import numpy as np
import matplotlib.pyplot as plt
import gym
from gym import wrappers
from copy import copy
from copy import deepcopy
env = gym.make('CartPole-v0')
env = wrappers.Monitor(env, '/tmp/cartpole-experiment-1', force=True)
# Create hierarchy
cs = eogmaneo.ComputeSystem(4)
# Layer descriptors
lds = []
for l in range(6): # Layers
ld = eogmaneo.LayerDesc()
# Set some parameters
ld._width = 5
ld._height = 5
ld._columnSize = 32
ld._forwardRadius = ld._backwardRadius = 2
ld._temporalHorizon = 2
ld._ticksPerUpdate = 2
lds.append(ld)
h = eogmaneo.Hierarchy()
inputRes = 32 # Number of bins to rescale inputs into. This is the resolution of the input columns (input column size)
# 4 input scalars, 1 binary action
h.create([ (2, 2), (1, 1) ], [ inputRes, 2 ], [ False, True ], lds, 123)
# Set parameters
for i in range(len(lds)):
l = h.getLayer(i)
l._alpha = 0.01
l._beta = 0.01
l._gamma = 0.9
l._maxReplaySamples = 32 # Equivalent to a finite length eligibility trace
l._codeIters = 4
# Bounds for rescaling inputs into discrete columns
obsMin = [ -0.25, -0.25, -0.25, -0.25 ]
obsMax = [ 0.25, 0.25, 0.25, 0.25 ]
useReward = 0.0
action = 0
for ep in range(5000):
observation = env.reset()
for t in range(1000):
#env.render()
# Rescale and discretize inputs into seperate IO layers
obsSDR = []
for i in range(4):
obsSDR.append(int((min(obsMax[i], max(obsMin[i], observation[i])) - obsMin[i]) / (obsMax[i] - obsMin[i]) * (inputRes - 1) + 0.5)) # Rescale from [obsMax, obsMin] range to discrete range [0, inputRes)
h.step(cs, [ obsSDR, [ action ] ], useReward, True)
# Retrieve action (2nd IO layer)
action = h.getPredictions(1)[0] # First and only entry
# Exploration
if np.random.rand() < 0.01: # Epsilon-greedy
action = np.random.randint(0, 2)
observation, reward, done, info = env.step(action)
# Reward is simply a negative one for when the episode ends (task is to maximize episode length, which balances the pole)
useReward = -1.0 * float(done)
if done:
print("Episode {} finished after {} timesteps.".format(ep, t + 1))
break
dsimba
Hi there,
i would like to use EOgmaNeo and apply it to a reinforcement learning problem. I have inputs as a vector at each time step. But I don't know how to pass these inputs to the learning agent, i.e EOgmaNeo and get the action.
Please any help appretiated.
Thanks