Change network architecture

[schrum2]

In DQN_cartpole.py, the architecture of the network is represented in QValueNetwork, specifically in the step function.

I think that there is a single hidden layer with 10 neurons, and that is it. Each tf.layers.dense command represents a single layer. The first is input into the hidden layer, and the second is hidden to the output layer (I believe).

Try tweaking this architecture. Add/remove layers and/or hidden neurons from within the layer, just to get a feel for how to manipulate the architecture.

[nazaruka]

Tensorflow's tf.layers module allows for the creation of convolutional, pooling, and dense (fully connected) layers. Because we are not dealing with frames just yet, the only layers we will be using are dense layers, in which every node in a given layer is connected to every node in the preceding layer.

For a layer to flow into a second layer, the second layer must take the first one as an input parameter tensor in its declaration. In other words, if there is a declared layer1 and we are about to declare a layer2, we do so by writing layer2 = tf.layers.dense(inputs=layer1, ...) [NB: inputs is optional]. The first layer takes the observation space, or obs, as an input. Changing the number of neurons in a layer is also easy; there is a parameter that represents how many neurons you may assign to a layer.

Creating four layers, three of numbers of neurons going up ascending powers of two and one an output layer, would be written like so:

h1 = tf.layers.dense(obs, 2, tf.nn.tanh,
                     kernel_initializer=tf.random_normal_initializer(mean=0, stddev=0.2))
h2 = tf.layers.dense(h1, 4, tf.nn.tanh,
                     kernel_initializer=tf.random_normal_initializer(mean=0, stddev=0.2))
h3 = tf.layers.dense(h2, 8, tf.nn.tanh,
                     kernel_initializer=tf.random_normal_initializer(mean=0, stddev=0.2))
value = tf.layers.dense(h3, self.act_dim,
                     kernel_initializer=tf.random_normal_initializer(mean=0, stddev=0.2))

Of course, this code ended up being much less effective than the original (which had only one hidden layer with ten neurons), potentially because of the numerous connections between the dense layers' nodes. Still, it proved to be a fascinating experiment.