Add logging support for: MAPE (and MAE, MSE, RMSE, NRMSE)

[xinsuinizhuan]

You are so greate, i set epoch to 200 and 1000, the performace is better, and no -nans. But i guess no the more epoch the better, so we should have MAPE (and MAE, MSE, RMSE, NRMSE) information, so when arrive the expected accuracy then stop train auto. Then i expecte print the MAPE (and MAE, MSE, RMSE, NRMSE) information every epoch!

[josephjaspers]

Added logging as of: https://github.com/josephjaspers/blackcat_tensors/commit/e6fd6d546c44dbd388b25320b66dcbd2ea163a71

(The mnist_test.h has been modified to use it)

    auto network = neuralnetwork(
        feedforward(system_tag, 784, 256),
        tanh(system_tag, 256),
        feedforward(system_tag, 256, 10),
        softmax(system_tag, 10),
        logging_output_layer(system_tag, 10, BC::nn::RMSE).skip_every(100)
    );

Currently untested on windows.

mnist_test.h (example) output

./mnist_test_exec mnist_train.csv 
Dataset location: mnist_train.csv
Neural Network architecture: 
FeedForward:
    inputs: 784
    outputs: 256
Tanh:
    inputs: 256
    outputs: 256
FeedForward:
    inputs: 256
    outputs: 10
SoftMax:
    inputs: 10
    outputs: 10
OutputLayer:
    inputs: 10
    outputs: 10

 training...
 current epoch: 0
Batch index: 100 loss: 0.39624
Batch index: 200 loss: 0.357391
 current epoch: 1
Batch index: 300 loss: 0.351267
Batch index: 400 loss: 0.338473
Batch index: 500 loss: 0.316958
 current epoch: 2
Batch index: 600 loss: 0.300975
Batch index: 700 loss: 0.306344
 current epoch: 3
Batch index: 800 loss: 0.28626
Batch index: 900 loss: 0.287176
Batch index: 1000 loss: 0.266664
 current epoch: 4
Batch index: 1100 loss: 0.264154
Batch index: 1200 loss: 0.272472
 current epoch: 5
Batch index: 1300 loss: 0.256432
Batch index: 1400 loss: 0.271314
Batch index: 1500 loss: 0.263249
 current epoch: 6
Batch index: 1600 loss: 0.247778
Batch index: 1700 loss: 0.260698
 current epoch: 7
Batch index: 1800 loss: 0.245016
Batch index: 1900 loss: 0.239343
Batch index: 2000 loss: 0.221002
 current epoch: 8
Batch index: 2100 loss: 0.245184
Batch index: 2200 loss: 0.211524
Batch index: 2300 loss: 0.204806
 current epoch: 9
Batch index: 2400 loss: 0.212802
Batch index: 2500 loss: 0.224889
 training time: 16.5934
 testing... 
[[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.7, 1.0, 0.3,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.7, 0.9, 0.9, 0.3,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.4, 0.9, 0.9, 0.6,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.3, 0.9, 0.9, 0.8,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.1, 0.8, 0.9, 0.9, 0.3,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.2, 0.8, 0.9, 0.9, 0.3,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.3, 0.9, 0.9, 0.9, 0.6,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.8, 0.9, 0.9, 0.6,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.8, 0.9, 0.9, 0.9, 0.3,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.1, 0.9, 0.9, 0.9,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.8, 0.9, 0.9, 0.7,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.6, 0.9, 0.9, 0.7,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.7, 0.9, 0.9, 0.2,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    , 0.7, 0.9, 0.9, 0.3,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    , 0.4, 0.9, 0.9, 0.7,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    , 0.3, 0.9, 0.9, 0.7,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    , 0.8, 0.9, 0.9, 0.3,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    , 0.3, 0.9, 0.9, 0.9, 0.3,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    , 0.3, 0.9, 0.9, 0.9, 0.5,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    , 0.8, 0.8, 0.3,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]
[   ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ,    ]]
[0.000000, 0.999987, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000013, 0.000000]
------------------------------------

[xinsuinizhuan]

when i replace the "outputlayer(system_tag, 10)" to "logging_output_layer(system_tag, 10, BC::nn::RMSE).skip_every(100)" in mnist_test_recurrent example, then compile it, it errors. But it works in mnist_test example.

[xinsuinizhuan]

In addition, this should be Host_Convolution2d.h

[josephjaspers]

Yes I fixed that, shouldn't have pushed that in. Currently building (successfully) now https://travis-ci.org/josephjaspers/BlackCat_Tensors

[xinsuinizhuan]

you could try to change the way to log! exmaple, to translate the x and y to Mean_Absolute_Error, then calculate or others!

[xinsuinizhuan]

How about it now?

[josephjaspers]

I want to try solve the issue you are having but I will think of an alternate way to implement logging if we cannot find the solution.

[josephjaspers]

See: https://github.com/josephjaspers/blackcat_tensors/commit/e6fd6d546c44dbd388b25320b66dcbd2ea163a71

Problem was an issue with Visual Studio Preview (16.3) edition. The stable releases (16.1 and 16.2) work correctly.

[xinsuinizhuan]

I am so sorry, a lot of time was wasted of you to solve this problem, becouse of my Visual Studio Preview error edition! Then focuse on the others improtance function of the sulotion.

[josephjaspers]

No problem, Its good to know for the future anyway!

[xinsuinizhuan]

Could add the MAPE log?

[josephjaspers]

Added MAPE as of : https://github.com/josephjaspers/blackcat_tensors/commit/cf43de380da13cbc97e8f67c17f76f385695b1da

[xinsuinizhuan]

I log MPAE、 MAE and RMSE， but they are so big and not decrease progressively, so they could not reflect the accuracy of my model, then how do I rate the accuracy of my model?I divided the data into training data and test data. After the training, I used the test data to do the test, and then calculated the average MPAE of the whole test data. I'm not familiar with your template class operations, so i don't konw how should i do? I write it, but break:

[josephjaspers]

The img variable is reshabed to a cube. (It is simply for stdout) and it is not actually fed into the network. The inputs are fed into the network.

The "testing..." section is a batched operation. To calculate the testing output error simply do:

    double error = BC::nn::MAPE(hyps, outputs[0]);
    double average_error = error / batch_size;
    std::cout << "Average loss: " << error << std::endl;

Which template class operations are you referring to? (If there is something you need to know/confused I will write documentation for it if it doesn't exist yet).

[josephjaspers]

I am unsure what you mean by are so big and not decrease progressively

https://travis-ci.org/josephjaspers/blackcat_tensors as the loss (from the travis build) is shown to be decreasing according to the example output.

[xinsuinizhuan]

I am unsure what you mean by are so big and not decrease progressively

https://travis-ci.org/josephjaspers/blackcat_tensors as the loss (from the travis build) is shown to be decreasing according to the example output.

I log the MAPE in mnist_test_recurrent, that log loss is big than actual, because so namy loss data greater than 1. MAPE：value_type epsilon_001.txt

In addtion, i attention to in MAPE, value_type epsilon = .001， when i make the input to [0, 1], it is so big, so i attept to set value_type epsilon = 1e-6, but the MAPE log data so bigger. MAPE：value_type epsilon_1e-6.txt

In addtion, i don't know why, the log loss is so big, but the test data set's average loss is so good performance?

[xinsuinizhuan]

The img variable is reshabed to a cube. (It is simply for stdout) and it is not actually fed into the network. The inputs are fed into the network.

The "testing..." section is a batched operation. To calculate the testing output error simply do:

  double error = BC::nn::MAPE(hyps, outputs[0]);
  double average_error = error / batch_size;
  std::cout << "Average loss: " << error << std::endl;

Which template class operations are you referring to? (If there is something you need to know/confused I will write documentation for it if it doesn't exist yet).

It is my falut to operate the inputs, it should be operate hyps, to clculate the test data set's avarage loss.

[josephjaspers]

the MAPE's error appears large when it predicts '0' when the correct answer is '1'

IE When expected == 1 and epsilon is .0001

Error is: 1 / .0001 == 10000

So it is expected that MAPES may appear to have very large error at times.

A better way to log error would be categorical cross entropy loss. Though I have not implemented it yet.

[xinsuinizhuan]

the MAPE's error appears large when it predicts '0' when the correct answer is '1'

IE When expected == 1 and epsilon is .0001

Error is: 1 / .0001 == 10000

So it is expected that MAPES may appear to have very large error at times.

A better way to log error would be categorical cross entropy loss. Though I have not implemented it yet.

oh ,i know.But i think it shoul be most of the MAPE is not so big, only few of them big.

[xinsuinizhuan]

In addtion, i don't know why, the log loss is so big, but the test data set's average loss is so good performance?

[josephjaspers]

MAPE is calculated via... sum(abs((actual - expected)/(actual))) / batch_size

Is this what you expect or a different function?

[xinsuinizhuan]

MAPE is calculated via... sum(abs((actual - expected)/(actual))) / batch_size

Is this what you expect or a different function?

Yes, it is my expect function, but i am only puzzle, why the log loss is so big, but the test data set's average loss is so good performance.

[xinsuinizhuan]

Now, when i print the average loss, to conver the BC::nn::MAPE(hyps, outputs[0]) to double is error:

    double error = BC::nn::MAPE(hyps, outputs[0]);
double average_error = error / batch_size;
std::cout << "Average loss: " << error << std::endl;

compile it, it errors.

[josephjaspers]

Now, when i print the average loss, to conver the BC::nn::MAPE(hyps, outputs[0]) to double is error:

    double error = BC::nn::MAPE(hyps, outputs[0]);
double average_error = error / batch_size;
std::cout << "Average loss: " << error << std::endl;

compile it, it errors.

Yes, that is expected I changed the implementation of sum to something faster. You can value_sum(tensor) to use the old implementation. (which calls the older sum method)

[josephjaspers]

Closing as this feature has been added.