Great work!! - Githubissues

Congratulations, @Mostafa-Samir.

This implementation of the DeepMind DNC works like a charm. I've tested it with a new task for the model learning and generalizing a pseudo-code like this:

function(x): if (x == 1) return (near) 1.0 (float values) else return (near) 0.0 (float values)

I did it in this way (the model learns this function in less than 1000 iterations!!):

Common Settings

The model is trained on 2-layer feedforward controller (with hidden sizes 128 and 256 respectively) with the following set of hyperparameters:

RMSProp Optimizer with learning rate of 10⁻⁴, momentum of 0.9.
Memory word size of 10, with a single read head.
A batch size of 1.
input_size = 3.
output_size = 1.
sequence_max_length = 100.
words_count = 15.
word_size = 10.
read_heads = 1.

A square loss function of the form: (y - y)**2 is used. Where both 'y' and 'y' are scalar numbers.

The input is a (1, random_length, 3) tensor, where the 3 is for a one-hot encoding vector of size 3, where:

010 is a '0' 100 is a '1' 001 is the end mark

So, and example of an input of length 10 will be the next 3D-tensor:

[[[ 0. 1. 0.] [ 0. 1. 0.] [ 0. 1. 0.] [ 1. 0. 0.] [ 0. 1. 0.] [ 0. 1. 0.] [ 1. 0. 0.] [ 0. 1. 0.] [ 0. 1. 0.] [ 0. 0. 1.]]]

This input is a represenation of a sequence of adding 0 or 1 values in the form of:

0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + (end_mark)

The target outoput is a 3D-tensor with the result of this adding task. In the example above:

[[[2.0]]]

The DNC output is a 3D-tensor of shape (1, random_length, 1). For example:

[[[ 0.45] [ -0.11] [ 1.3] [ 5.0] [ 0.5] [ 0.1] [ 1.0] [ -0.5] [ 0.33] [ 0.12]]]

The target output and the DNC output are both then reduced with tf.reduce_sum() so we end up with two scalar values. For example:

Target_output: 2.0 DNC_output: 5.89

And we apply then the square loss function:

loss = (Target_o - DNC_o)**2

and finally the gradient update.

Results

The model is going to recieve as input a random length sequence of 0 or 1 values like:

Input: 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1

Then it will return a scalar value for this input adding proccess. For example, the DNC will output something like: 3.98824. This value will be the predicted result for the input adding sequence (we are going to truncate the integer part of the result):

DNC prediction: 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 = 3 [3.98824]

Once we train the model with:

$python tasks/copy/train.py --iterations=50000

we can see that the model learns in less than 1000 iterations to compute this adding function, and the loss drop from:

Iteration 0/1000 Avg. Logistic Loss: 24.9968

to:

Iteration 1000/1000 Avg. Logistic Loss: 0.0076

It seems like the DNC model is able to learn this pseudo-code:

function(x): if (x == [ 1. 0. 0.]) return (near) 1.0 (float values) else return (near) 0.0 (float values)

Generalization test

We use for the model a sequence_max_length = 100, but in the training proccess we use just random length sequences up to 10 (sequence_max_length/10). Once the train is finished, we let the trained model to generalize to random length sequences up to 100 (sequence_max_length).

Results show that the model successfully generalize the adding task even with sequence 10 times larger than the training ones.

These are real data outputs:

Building Computational Graph ... Done! Initializing Variables ... Done!

Iteration 0/1000 Avg. Logistic Loss: 24.9968 Real value: 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 = 5 Predicted: 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 = 0 [0.000319847]

Iteration 100/1000 Avg. Logistic Loss: 5.8042 Real value: 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 = 5 Predicted: 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 = 6 [6.1732]

Iteration 200/1000 Avg. Logistic Loss: 0.7492 Real value: 1 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 = 9 Predicted: 1 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 = 8 [8.91952]

Iteration 300/1000 Avg. Logistic Loss: 0.0253 Real value: 0 + 1 + 1 = 2 Predicted: 0 + 1 + 1 = 2 [2.0231]

Iteration 400/1000 Avg. Logistic Loss: 0.0089 Real value: 0 + 1 + 0 + 0 + 0 + 1 + 1 = 3 Predicted: 0 + 1 + 0 + 0 + 0 + 1 + 1 = 2 [2.83419]

Iteration 500/1000 Avg. Logistic Loss: 0.0444 Real value: 1 + 0 + 1 + 1 = 3 Predicted: 1 + 0 + 1 + 1 = 2 [2.95937]

Iteration 600/1000 Avg. Logistic Loss: 0.0093 Real value: 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 = 4 Predicted: 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 = 3 [3.98824]

Iteration 700/1000 Avg. Logistic Loss: 0.0224 Real value: 0 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 = 6 Predicted: 0 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 = 5 [5.93554]

Iteration 800/1000 Avg. Logistic Loss: 0.0115 Real value: 0 + 0 = 0 Predicted: 0 + 0 = -1 [-0.0118587]

Iteration 900/1000 Avg. Logistic Loss: 0.0023 Real value: 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 = 5 Predicted: 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 = 4 [4.97147]

Iteration 1000/1000 Avg. Logistic Loss: 0.0076 Real value: 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 = 4Done!

Testing generalization...

Iteration 0/1000 Predicted: 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 = 4 [4.123]

Saving Checkpoint ... Real value: 1 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 = 6 Predicted: 1 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 = 6 [6.24339]

Iteration 1/1000 Real value: 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 = 11 Predicted: 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 = 11 [11.1931]

Iteration 2/1000 Real value: 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 = 33 Predicted: 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 = 32 [32.9866]

Iteration 3/1000 Real value: 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 = 16 Predicted: 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 = 16 [16.1541]

Iteration 4/1000 Real value: 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 = 44 Predicted: 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 = 43 [43.5211]

Iteration 5/1000 Real value: 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 = 33 Predicted: 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 = 33 [33.081]

Iteration 6/1000 Real value: 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 = 12 Predicted: 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 = 12 [12.2167]

Iteration 7/1000 Real value: 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 = 10 Predicted: 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 = 10 [10.1519]

Iteration 8/1000 Real value: 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 = 33 Predicted: 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 = 33 [33.1624]

Iteration 9/1000 Real value: 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 = 17 Predicted: 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 = 17 [17.2243]

Iteration 10/1000 Real value: 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 = 25 Predicted: 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 = 25 [25.0643]

Iteration 11/1000 Real value: 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 = 42 Predicted: 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 = 41 [41.6496]

Iteration 12/1000 Real value: 1 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 1 = 42 Predicted: 1 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 1 = 41 [41.4914]

Iteration 13/1000 Real value: 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 = 19 Predicted: 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 = 19 [19.2566]

Iteration 14/1000 Real value: 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 = 25 Predicted: 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 = 25 [25.0976]

Iteration 15/1000 Real value: 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 0 = 48 Predicted: 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 0 = 47 [47.1815]

Iteration 16/1000 Real value: 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 = 8 Predicted: 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 = 8 [8.11703]

Iteration 17/1000 Real value: 0 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 = 45 Predicted: 0 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 = 44 [44.3639]

Iteration 18/1000 Real value: 1 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 = 15 Predicted: 1 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 = 15 [15.2815]

Iteration 19/1000 Real value: 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 = 40 Predicted: 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 = 39 [39.928]

Iteration 20/1000 Real value: 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 0 = 19 Predicted: 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 0 = 19 [19.2498]

Iteration 21/1000 Real value: 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 = 38 Predicted: 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 1 = 37 [37.7267]

Iteration 22/1000 Real value: 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 0 = 21 Predicted: 0 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 0 = 21 [21.2265]

Iteration 23/1000 Real value: 1 + 1 = 2 Predicted: 1 + 1 = 2 [2.01672]

Iteration 24/1000 Real value: 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 = 45 Predicted: 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 = 44 [44.5173]

Iteration 25/1000 Real value: 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 = 40 Predicted: 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 = 39 [39.6734]

Iteration 26/1000 Real value: 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 = 9 Predicted: 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 = 9 [9.10768]

Iteration 27/1000 Real value: 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 = 20 Predicted: 1 + 0 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 = 20 [20.1105]

Iteration 28/1000 Real value: 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 = 44 Predicted: 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 0 + 1 + 0 = 43 [43.5083]

Iteration 29/1000 Real value: 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 = 20 Predicted: 1 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 0 + 0 = 20 [20.1701]

Iteration 30/1000 Real value: 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 0 = 31 Predicted: 1 + 0 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 1 + 0 + 1 + 0 = 30 [30.8908]

Iteration 31/1000 Real value: 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 = 38 Predicted: 0 + 1 + 0 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 1 + 1 + 0 + 1 + 1 + 0 + 0 + 0 + 1 + 1 + 0 + 0 + 1 + 1 + 1 + 0 + 1 + 1 + 1 + 1 + 0 + 1 + 0 + 0 + 1 + 1 + 0 + 0 + 0 + 0 + 1 + 0 + 0 + 1 + 0 + 1 = 37 [37.9008] ... ...

Mostafa-Samir / DNC-tensorflow

Great work!! #5

Common Settings

Results

Generalization test