Neural Network: Save/Load should preserve intermediate values

[josephjaspers]

Currently when when saving a model only the weights are preserved, however the state should be preserved as well.

[xinsuinizhuan]

Are you optimizing your network structure?

[josephjaspers]

Hello!

Since talking, I have added Convolution, Maxpooling (cpu and gpu) with optimized versions for convolution.

LSTM is not optimized yet (though it already runs at reasonable speed, I believe it can be even faster).

I have been adding a large amount of error checking and static_asserts which should make using the library easier (as it should more often give easy to understand errors).

The save/load differences are limited to the LSTM, but I can try to fix them soon. (It currently works as expected for non LSTMs)

---

I can try to solve the save/load issues soon as they shouldn't be too hard to fix.

[xinsuinizhuan]

Then i hope solved this issues。

[xinsuinizhuan]

How about this issues?

[josephjaspers]

I'm working on changing the LSTM to an optimzied version so I'm holding off on fixing this issue till than. (Hopefully will be done by end of this weekend).

[xinsuinizhuan]

thank you very much!

在 2019-12-12 06:34:00，"Joseph Jaspers" notifications@github.com 写道：

I'm working on changing the LSTM to an optimzied version so I'm holding off on fixing this issue till than. (Hopefully will be done by end of this weekend).

I am also trying to change the interface of Layers to make the easier to use. (It will look more like keras)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[xinsuinizhuan]

when i save the trained model parameters, my screen black and flicker,how to avoid it?

[josephjaspers]

Are you using the GPU implementation?

It most likely means that you are close to running out of (gpu) ram, if this happens with the CPU that is very strange, and I am not sure if this code could even cause such an issue. (Thought maybe a ram issue as well)

[josephjaspers]

Branch for the optimized LSTM https://github.com/josephjaspers/blackcat_tensors/tree/optimize_lstm

Oddly enough it is faster in CPU mode but slower on the GPU (due to strided memory access).

I might need to still work on it as I would prefer not to have different versions for each architecture

[josephjaspers]

when i save the trained model parameters, my screen black and flicker,how to avoid it?

I just added a change to save/load (really the "to_string" implementation) that avoids a gpu-allocation when the memory is continuous.

This doesn't effect the CPU version at all.

https://github.com/josephjaspers/blackcat_tensors/commit/cff63513d1fce25d418a49bb287b5c6a1f3f20f7

---

Additionally I have just added "clear_recycler" BC::allocators::Recycle_Allocator_Globals::clear_recycler(system_tag);

Which will clear all currently unused memory in the Recycler_Allocator (The Neural Network doesn't deallocate its memory after use, and instead stores it in a global-dictionary for reuse).

This may help if you are close to running out of ram, however I would try if the latest code fixes your problem before trying this.

https://github.com/josephjaspers/blackcat_tensors/commit/10a6f4398e837baa896caca53739e25c677c63ee edit** minor fix: https://github.com/josephjaspers/blackcat_tensors/commit/e7249607b0b58738dc0c81700390a448ab9e1c23

[josephjaspers]

Load/saving in LSTM has been fixed: https://github.com/josephjaspers/blackcat_tensors/commit/279581f289331f6df13d05bfdc512107531a235d

Caused by a bug with an incorrectly named method "load_from_cache" -> "load_to_cache"

[josephjaspers]

You can test that saving/loading works by replacing the mnist_test_recurrent/mnist_test.h with:

#include "../../include/BlackCat_Tensors.h"
#include "../../include/BlackCat_NeuralNetworks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=BC::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
        int epochs=5, int batch_size=32, int samples=32*1024) {

    using value_type     = typename System::default_floating_point_type;
    using allocator_type = BC::Allocator<System, value_type>;
    using cube           = BC::Cube<value_type, allocator_type>;
    using mat            = BC::Matrix<value_type, allocator_type>;
    using clock          = std::chrono::duration<double>;

    auto network = BC::nn::neuralnetwork(
        BC::nn::lstm(system_tag, 784/4, 128),
        BC::nn::lstm(system_tag, 128, 64),
        BC::nn::feedforward(system_tag, 64, 10),
        BC::nn::softmax(system_tag, 10),
        BC::nn::logging_output_layer(system_tag, 10, BC::nn::RMSE).skip_every(100)
    );

    BC::print("Neural Network architecture:");
    BC::print(network.get_string_architecture());

    network.set_batch_size(batch_size);

    std::pair<cube, cube> data = load_mnist(
            system_tag, mnist_dataset, batch_size, samples);

    cube& inputs = data.first;
    cube& outputs = data.second;

    BC::print("training...");
    auto start = std::chrono::system_clock::now();

    int img_partitions = 4;
    for (int i = 0; i < epochs; ++i){
        BC::print("current epoch:  ", i);

        for (int j = 0; j < samples/batch_size; j++) {
            for (int p = 0; p < img_partitions; ++p) {

                auto batch = inputs[j];
                auto index = BC::index(0,784 * (p/(float)img_partitions));
                auto shape = BC::shape(784/4, batch_size);

                network.forward_propagation(batch[{index, shape}]);
            }
                //Apply backprop on the last two images (images 3/4 and 4/4)
                network.back_propagation(outputs[j]);
                network.back_propagation(outputs[j]);

                network.update_weights();
        }
    }

    auto end = std::chrono::system_clock::now();
    BC::print("training time:", clock(end - start).count());

    BC::print("testing...");

    network.save("mnist_test");

    auto batch = inputs[0];
    auto shape = BC::shape(784/4, batch_size);
    {
        for (int p = 0; p < img_partitions-1; ++p) {
            auto index = BC::index(0, 784*(p/(float)img_partitions));
            network.predict(batch[{index, shape}]);
        }

        auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
        mat hyps =network.predict(batch[{last_index, shape}]);

        BC::size_t test_images = 10;
        cube img = cube(inputs[0].reshaped(28,28, batch_size));
        for (int i = 0; i < test_images; ++i) {
            outputs[0][i];
            hyps[i].print();
            BC::print("------------------------------------");
        }
    }

    network.load("mnist_test");
    {
        for (int p = 0; p < img_partitions-1; ++p) {
            auto index = BC::index(0, 784*(p/(float)img_partitions));
            network.predict(batch[{index, shape}]);
        }

        auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
        mat hyps =network.predict(batch[{last_index, shape}]);

        BC::size_t test_images = 10;
        cube img = cube(inputs[0].reshaped(28,28, batch_size));
        for (int i = 0; i < test_images; ++i) {
            outputs[0][i];
            hyps[i].print();
            BC::print("------------------------------------");
        }
    }

    BC::print("success");
    return 0;
}

[xinsuinizhuan]

Are you using the GPU implementation?

It most likely means that you are close to running out of (gpu) ram, if this happens with the CPU that is very strange, and I am not sure if this code could even cause such an issue. (Thought maybe a ram issue as well)

No, i am so sorry, it seems that the console popup, and output something.

[xinsuinizhuan]

I only run it CPU，because using the GPU， the time is nearly equal to the cpu.

[xinsuinizhuan]

incorrectly named method

OK. Thank you very much, let me have a try!

[xinsuinizhuan]

You can test that saving/loading works by replacing the mnist_test_recurrent/mnist_test.h with:

#include "../../include/BlackCat_Tensors.h"
#include "../../include/BlackCat_NeuralNetworks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=BC::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
      int epochs=5, int batch_size=32, int samples=32*1024) {

  using value_type     = typename System::default_floating_point_type;
  using allocator_type = BC::Allocator<System, value_type>;
  using cube           = BC::Cube<value_type, allocator_type>;
  using mat            = BC::Matrix<value_type, allocator_type>;
  using clock          = std::chrono::duration<double>;

  auto network = BC::nn::neuralnetwork(
      BC::nn::lstm(system_tag, 784/4, 128),
      BC::nn::lstm(system_tag, 128, 64),
      BC::nn::feedforward(system_tag, 64, 10),
      BC::nn::softmax(system_tag, 10),
      BC::nn::logging_output_layer(system_tag, 10, BC::nn::RMSE).skip_every(100)
  );

  BC::print("Neural Network architecture:");
  BC::print(network.get_string_architecture());

  network.set_batch_size(batch_size);

  std::pair<cube, cube> data = load_mnist(
          system_tag, mnist_dataset, batch_size, samples);

  cube& inputs = data.first;
  cube& outputs = data.second;

  BC::print("training...");
  auto start = std::chrono::system_clock::now();

  int img_partitions = 4;
  for (int i = 0; i < epochs; ++i){
      BC::print("current epoch:  ", i);

      for (int j = 0; j < samples/batch_size; j++) {
          for (int p = 0; p < img_partitions; ++p) {

              auto batch = inputs[j];
              auto index = BC::index(0,784 * (p/(float)img_partitions));
              auto shape = BC::shape(784/4, batch_size);

              network.forward_propagation(batch[{index, shape}]);
          }
              //Apply backprop on the last two images (images 3/4 and 4/4)
              network.back_propagation(outputs[j]);
              network.back_propagation(outputs[j]);

              network.update_weights();
      }
  }

  auto end = std::chrono::system_clock::now();
  BC::print("training time:", clock(end - start).count());

  BC::print("testing...");

  network.save("mnist_test");

  auto batch = inputs[0];
  auto shape = BC::shape(784/4, batch_size);
  {
      for (int p = 0; p < img_partitions-1; ++p) {
          auto index = BC::index(0, 784*(p/(float)img_partitions));
          network.predict(batch[{index, shape}]);
      }

      auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
      mat hyps =network.predict(batch[{last_index, shape}]);

      BC::size_t test_images = 10;
      cube img = cube(inputs[0].reshaped(28,28, batch_size));
      for (int i = 0; i < test_images; ++i) {
          outputs[0][i];
          hyps[i].print();
          BC::print("------------------------------------");
      }
  }

  network.load("mnist_test");
  {
      for (int p = 0; p < img_partitions-1; ++p) {
          auto index = BC::index(0, 784*(p/(float)img_partitions));
          network.predict(batch[{index, shape}]);
      }

      auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
      mat hyps =network.predict(batch[{last_index, shape}]);

      BC::size_t test_images = 10;
      cube img = cube(inputs[0].reshaped(28,28, batch_size));
      for (int i = 0; i < test_images; ++i) {
          outputs[0][i];
          hyps[i].print();
          BC::print("------------------------------------");
      }
  }

  BC::print("success");
  return 0;
}

Did not to need network.copy_training_data_to_single_predict(0)？？

[xinsuinizhuan]

Neural Network: Load problem. when i trained, i use the model to single_predict, the same input the same output, time after time. But when i load the model, then to single_predict, the same input, output is different, the different time. When i load the model, whether need to network.copy_training_data_to_single_predict(0)???

[xinsuinizhuan]

When i save the model, how to forbid the console popup to print something? Because i did not the console project, but when i save the model, the console popup and screen flicker.

[xinsuinizhuan]

mnist_test_example: network.save("mnist_test"); auto batch = inputs[0]; BC::print("predict ------------------------------------"); auto shape = BC::shape(784 / 4, batch_size); { for (int p = 0; p < img_partitions - 1; ++p) { auto index = BC::index(0, 784 * (p / (float)img_partitions)); network.predict(batch[{index, shape}]); }

    auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
    mat hyps = network.predict(batch[{last_index, shape}]);

    BC::size_t test_images = 10;
    cube img = cube(inputs[0].reshaped(28, 28, batch_size));
    for (int i = 0; i < test_images; ++i) {
        outputs[0][i];
        hyps[i].print();
        BC::print("------------------------------------");
    }
}
    network.load("mnist_test");
BC::print("load fist ------------------------------------");
{
    for (int p = 0; p < img_partitions - 1; ++p) {
        auto index = BC::index(0, 784 * (p / (float)img_partitions));
        network.predict(batch[{index, shape}]);
    }

    auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
    mat hyps = network.predict(batch[{last_index, shape}]);

    BC::size_t test_images = 10;
    cube img = cube(inputs[0].reshaped(28, 28, batch_size));
    for (int i = 0; i < test_images; ++i) {
        outputs[0][i];
        hyps[i].print();
        BC::print("------------------------------------");
    }
}

BC::print("load second ------------------------------------");
{
    for (int p = 0; p < img_partitions - 1; ++p) {
        auto index = BC::index(0, 784 * (p / (float)img_partitions));
        network.predict(batch[{index, shape}]);
    }

    auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
    mat hyps = network.predict(batch[{last_index, shape}]);

    BC::size_t test_images = 10;
    cube img = cube(inputs[0].reshaped(28, 28, batch_size));
    for (int i = 0; i < test_images; ++i) {
        outputs[0][i];
        hyps[i].print();
        BC::print("------------------------------------");
    }
}

the reuslt: when i load to predict, the result is not same as the load fist, and the load seconrd is not as all of them

[josephjaspers]

mnist_test_example: network.save("mnist_test"); auto batch = inputs[0]; BC::print("predict ------------------------------------"); auto shape = BC::shape(784 / 4, batch_size); { for (int p = 0; p < img_partitions - 1; ++p) { auto index = BC::index(0, 784 * (p / (float)img_partitions)); network.predict(batch[{index, shape}]); }

  auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
  mat hyps = network.predict(batch[{last_index, shape}]);

  BC::size_t test_images = 10;
  cube img = cube(inputs[0].reshaped(28, 28, batch_size));
  for (int i = 0; i < test_images; ++i) {
      outputs[0][i];
      hyps[i].print();
      BC::print("------------------------------------");
  }
}
    network.load("mnist_test");
BC::print("load fist ------------------------------------");
{
  for (int p = 0; p < img_partitions - 1; ++p) {
      auto index = BC::index(0, 784 * (p / (float)img_partitions));
      network.predict(batch[{index, shape}]);
  }

  auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
  mat hyps = network.predict(batch[{last_index, shape}]);

  BC::size_t test_images = 10;
  cube img = cube(inputs[0].reshaped(28, 28, batch_size));
  for (int i = 0; i < test_images; ++i) {
      outputs[0][i];
      hyps[i].print();
      BC::print("------------------------------------");
  }
}

BC::print("load second ------------------------------------");
{
  for (int p = 0; p < img_partitions - 1; ++p) {
      auto index = BC::index(0, 784 * (p / (float)img_partitions));
      network.predict(batch[{index, shape}]);
  }

  auto last_index = BC::index(0, 784 * ((img_partitions - 1) / (float)img_partitions));
  mat hyps = network.predict(batch[{last_index, shape}]);

  BC::size_t test_images = 10;
  cube img = cube(inputs[0].reshaped(28, 28, batch_size));
  for (int i = 0; i < test_images; ++i) {
      outputs[0][i];
      hyps[i].print();
      BC::print("------------------------------------");
  }
}

the reuslt: when i load to predict, the result is not same as the load fist, and the load seconrd is not as all of them

I will have to check that tomorrow! I just added some code which will hopefully deal with the console pop up.

[xinsuinizhuan]

In my project, when i load the trained model, then to single_predict, the first time the output is worse, the sencord, the third, the fourth and fifth, the output got better and better, and eventually stabilized。

[josephjaspers]

In my project, when i load the trained model, then to single_predict, the first time the output is worse, the sencord, the third, the fourth and fifth, the output got better and better, and eventually stabilized。

--

Could you show me the outputs you receiving? I can't seem to emulate the problem. (Are you using copy_training_data_to_single_predict?)

[josephjaspers]

When i save the model, how to forbid the console popup to print something? Because i did not the console project, but when i save the model, the console popup and screen flicker.

Also is this still an issue? Sorry I have not been able to test on Windows yet.

[josephjaspers]

(Not related to this issue) I have renamed the directories and changed the BC namespace to bc

the directories should be included as such:

#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h" 

I did this to be more consistent with most C++ style guides

[xinsuinizhuan]

(Not related to this issue) I have renamed the directories and changed the BC namespace to bc

the directories should be included as such:

#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h" 

I did this to be more consistent with most C++ style guides

Oh, No. when i use the new code, so many compile error:

[xinsuinizhuan]

When i save the model, how to forbid the console popup to print something? Because i did not the console project, but when i save the model, the console popup and screen flicker.

Also is this still an issue? Sorry I have not been able to test on Windows yet.

My net struct: auto make_lstm_network() { return BC::nn::neuralnetwork( BC::nn::lstm(BC::host_tag(), 96 * 10, 1024, BC::nn::adam), BC::nn::lstm(BC::host_tag(), 1024, 512, BC::nn::adam), BC::nn::lstm(BC::host_tag(), 512, 216, BC::nn::adam), BC::nn::feedforward(BC::host_tag(), 216, 192, BC::nn::adam), BC::nn::logistic(BC::host_tag(), 192), BC::nn::logging_output_layer(BC::host_tag(), 192, BC::nn::RMSE).skip_every(100) ); }

my code: //start train LstmPredictTask* lstmpredicttask = new LstmPredictTask(); if (lstmpredicttask == NULL) { return -2; }

//LstmPredictTask lstmpredicttask;
std::cout << "Neural Network architecture: \n" << lstmpredicttask->m_pnetwork.get_string_architecture() << std::endl;
lstmpredicttask->m_pnetwork.set_learning_rate(lstmpredicttask->m_learning_rate);
lstmpredicttask->m_pnetwork.set_batch_size(lstmpredicttask->m_batch_size);

int training_sets;
std::pair<cube, cube> data = load_train_data(system_tag, datafilepath, lstmpredicttask, &training_sets);
cube& inputs = data.first;
cube& outputs = data.second;

std::cout <<" training..." << std::endl;
auto start = std::chrono::system_clock::now();

std::cout << "imagesinput real data:------------------------------------" << std::endl;
cube imagesinput = cube(inputs[0].reshaped(96, 10, lstmpredicttask->m_batch_size));
imagesinput[0].t().print_sparse();

std::cout << "imagesoutput real data:------------------------------------" << std::endl;
//auto imagesoutput = reshape(outputs[0], BC::shape(96, 2, lstmpredicttask->m_batch_size));
cube imagesoutput = cube(outputs[0].reshaped(96, 2, lstmpredicttask->m_batch_size));
imagesoutput[0].t().print_sparse();

    lstmpredicttask->m_pnetwork.load(_trainparamsavefile);

   std::cout << "process_predict_data first------------------------------------" << std::endl;
std::vector<double> forcast_outputvec;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec);
std::cout << "process_predict_data second------------------------------------" << std::endl;
std::vector<double> forcast_outputvec1;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec1);
std::cout << "process_predict_data third------------------------------------" << std::endl;
std::vector<double> forcast_outputvec2;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec2);

the single_predict result:

error.txt

i use the laod data to single_predict, use the same input, test three times, the first, second and the third time, the output is different. the result got better and better, the third time output close to the not load net single_predict.

[josephjaspers]

Looking into it now

[josephjaspers]

(Not related to this issue) I have renamed the directories and changed the BC namespace to bc the directories should be included as such:

#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h" 

I did this to be more consistent with most C++ style guides

Oh, No. when i use the new code, so many compile error:

Hmm. I was able to compile without issues. If you could send me the first errors I might be able to debug the issue. It seems you are getting errors in cstdint which is strange as it is not my code.

[xinsuinizhuan]

(Not related to this issue) I have renamed the directories and changed the BC namespace to bc the directories should be included as such:

#include "blackcat/tensors.h"
#include "blackcat/neural_networks.h" 

I did this to be more consistent with most C++ style guides

Oh, No. when i use the new code, so many compile error:

Hmm. I was able to compile without issues. If you could send me the first errors I might be able to debug the issue. It seems you are getting errors in cstdint which is strange as it is not my code.

Yes, not your code, but old version code no problem, error: 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(27,14): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(27,14): error C2143: syntax error: missing ';' before '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(27,14): error C2059: syntax error: '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(29,25): error C2143: syntax error: missing ';' before '{' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(29,25): error C2447: '{': missing function header (old-style formal list?) 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(70,14): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(70,14): error C2143: syntax error: missing ';' before '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(70,14): error C2059: syntax error: '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(71,49): error C2143: syntax error: missing ';' before '{' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(71,49): error C2447: '{': missing function header (old-style formal list?) 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(76,14): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(76,14): error C2143: syntax error: missing ';' before '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(76,14): error C2059: syntax error: '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(77,25): error C2143: syntax error: missing ';' before '{' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(77,25): error C2447: '{': missing function header (old-style formal list?) 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(82,14): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(82,14): error C2143: syntax error: missing ';' before '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(82,14): error C2059: syntax error: '<' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(83,49): error C2143: syntax error: missing ';' before '{' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(83,49): error C2447: '{': missing function header (old-style formal list?) 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(89,29): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(89,29): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(89,29): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(92,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(95,43): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(95,50): error C2065: '_Base': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(97,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(98,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(102,9): error C3861: '_Xout_of_range': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(105,9): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(106,10): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(106,37): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(112,30): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(112,30): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(112,30): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(115,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(118,43): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(118,50): error C2065: '_Base': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(120,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(121,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(125,9): error C3861: '_Xout_of_range': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(128,9): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(129,10): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(129,37): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(135,40): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(135,40): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(135,40): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(138,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(141,53): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(141,60): error C2065: '_Base': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(143,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(144,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(148,9): error C3861: '_Xout_of_range': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(151,9): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(152,10): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(152,37): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(158,36): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(158,36): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(158,36): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(161,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(164,49): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(164,56): error C2065: '_Base': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(166,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(167,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(171,9): error C3861: '_Xout_of_range': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(174,9): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(175,10): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(175,37): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(181,46): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(181,46): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(181,46): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(184,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(187,59): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(187,66): error C2065: '_Base': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(189,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(190,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(194,9): error C3861: '_Xout_of_range': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(197,9): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(198,10): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(198,37): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(204,31): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(204,31): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(204,31): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(206,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(209,44): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(211,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(212,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(216,9): error C3861: '_Xout_of_range': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(219,9): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(220,10): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(220,37): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(226,32): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(226,32): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(226,32): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(228,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(231,45): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(233,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(234,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(238,9): error C3861: '_Xout_of_range': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(241,9): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(242,10): error C2065: '_Idx': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(242,37): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(248,38): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(248,38): error C2988: unrecognizable template declaration/definition 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(248,38): error C2143: syntax error: missing ',' before '&' 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(250,24): error C2065: '_Str': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(253,51): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(255,17): error C2065: '_Eptr': undeclared identifier 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(256,9): error C3861: '_Xinvalid_argument': identifier not found 1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.24.28314\include\string(256,52): fatal error C1003: error count exceeds 100; stopping compilation 1>Done building project "MNIST_Test_Recurrent.vcxproj" -- FAILED.

[josephjaspers]

When I run this: predict and single_predict give me the exact same results.

So I am not sure why your results aren't the same. I think I would have to see how you are using single_predict to know what the issue is.

#include "../../blackcat/tensors.h"
#include "../../blackcat/neural_networks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=bc::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
        int epochs=5, int batch_size=32, int samples=32*1024) {

    using value_type     = typename System::default_floating_point_type;
    using allocator_type = bc::Allocator<System, value_type>;
    using cube           = bc::Cube<value_type, allocator_type>;
    using mat            = bc::Matrix<value_type, allocator_type>;
    using clock          = std::chrono::duration<double>;

    auto network = bc::nn::neuralnetwork(
        bc::nn::lstm(system_tag, 784/4, 128),
        bc::nn::lstm(system_tag, 128, 64),
        bc::nn::feedforward(system_tag, 64, 10),
        bc::nn::softmax(system_tag, 10),
        bc::nn::logging_output_layer(system_tag, 10, bc::nn::RMSE).skip_every(100)
    );

    bc::print("Neural Network architecture:");
    bc::print(network.get_string_architecture());

    network.set_batch_size(batch_size);

    std::pair<cube, cube> data = load_mnist(
            system_tag, mnist_dataset, batch_size, samples);

    cube& inputs = data.first;
    cube& outputs = data.second;

    bc::print("training...");
    auto start = std::chrono::system_clock::now();

    int img_partitions = 4;
    for (int i = 0; i < epochs; ++i){
        bc::print("current epoch:  ", i);

        for (int j = 0; j < samples/batch_size; j++) {
            for (int p = 0; p < img_partitions; ++p) {

                auto batch = inputs[j];
                auto index = bc::index(0,784 * (p/(float)img_partitions));
                auto shape = bc::shape(784/4, batch_size);

                network.forward_propagation(batch[{index, shape}]);
            }
                //Apply backprop on the last two images (images 3/4 and 4/4)
                network.back_propagation(outputs[j]);
                network.back_propagation(outputs[j]);

                network.update_weights();
        }
    }

    auto end = std::chrono::system_clock::now();
    bc::print("training time:", clock(end - start).count());

    bc::print("testing...");

    bc::print("saving..............");
    network.save("mnist_test");

    auto shape = bc::shape(784/4, batch_size);
    {

        bc::size_t test_images = 10;
        cube img = cube(inputs[0].reshaped(28,28, batch_size));
        for (int i = 0; i < test_images; ++i) {
            auto batch = inputs[i];
            for (int p = 0; p < img_partitions-1; ++p) {
                        auto index = bc::index(0, 784*(p/(float)img_partitions));
                        network.predict(batch[{index, shape}]);
                    }

            auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
            mat hyps =network.predict(batch[{last_index, shape}]);

            outputs[i][0];
            hyps[0].print();
            bc::print("------------------------------------");
        }
    }

    bc::print("load1");
    network.load("mnist_test");
    network.copy_training_data_to_single_predict(0);
    {

        bc::size_t test_images = 10;
        cube img = cube(inputs[0].reshaped(28,28, batch_size));
        for (int i = 0; i < test_images; ++i) {
            auto batch = inputs[i];
            for (int p = 0; p < img_partitions-1; ++p) {
                        auto index = bc::index(0, 784*(p/(float)img_partitions));
                        network.single_predict(batch[{index, shape}][0]);
                    }

            auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
            auto hyps = network.single_predict(batch[{last_index, shape}][0]);

            outputs[i][0];
            hyps.print();
            bc::print("------------------------------------");
        }
    }

    bc::print("load2");
    network.load("mnist_test");
    network.copy_training_data_to_single_predict(0);
    {

        bc::size_t test_images = 10;
        cube img = cube(inputs[0].reshaped(28,28, batch_size));
        for (int i = 0; i < test_images; ++i) {
            auto batch = inputs[i];
            for (int p = 0; p < img_partitions-1; ++p) {
                        auto index = bc::index(0, 784*(p/(float)img_partitions));
                        network.single_predict(batch[{index, shape}][0]);
                    }

            auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
            auto hyps = network.single_predict(batch[{last_index, shape}][0]);

            outputs[i][0];
            hyps.print();
            bc::print("------------------------------------");
        }
    }

    bc::print("success");
    return 0;
}

[xinsuinizhuan]

When I run this: predict and single_predict give me the exact same results.

So I am not sure why your results aren't the same. I think I would have to see how you are using single_predict to know what the issue is.

#include "../../blackcat/tensors.h"
#include "../../blackcat/neural_networks.h"
#include "../datasets/mnist_loader.h"
#include <chrono>
#include <string>

template<class System=bc::host_tag>
int percept_MNIST(System system_tag, std::string mnist_dataset,
      int epochs=5, int batch_size=32, int samples=32*1024) {

  using value_type     = typename System::default_floating_point_type;
  using allocator_type = bc::Allocator<System, value_type>;
  using cube           = bc::Cube<value_type, allocator_type>;
  using mat            = bc::Matrix<value_type, allocator_type>;
  using clock          = std::chrono::duration<double>;

  auto network = bc::nn::neuralnetwork(
      bc::nn::lstm(system_tag, 784/4, 128),
      bc::nn::lstm(system_tag, 128, 64),
      bc::nn::feedforward(system_tag, 64, 10),
      bc::nn::softmax(system_tag, 10),
      bc::nn::logging_output_layer(system_tag, 10, bc::nn::RMSE).skip_every(100)
  );

  bc::print("Neural Network architecture:");
  bc::print(network.get_string_architecture());

  network.set_batch_size(batch_size);

  std::pair<cube, cube> data = load_mnist(
          system_tag, mnist_dataset, batch_size, samples);

  cube& inputs = data.first;
  cube& outputs = data.second;

  bc::print("training...");
  auto start = std::chrono::system_clock::now();

  int img_partitions = 4;
  for (int i = 0; i < epochs; ++i){
      bc::print("current epoch:  ", i);

      for (int j = 0; j < samples/batch_size; j++) {
          for (int p = 0; p < img_partitions; ++p) {

              auto batch = inputs[j];
              auto index = bc::index(0,784 * (p/(float)img_partitions));
              auto shape = bc::shape(784/4, batch_size);

              network.forward_propagation(batch[{index, shape}]);
          }
              //Apply backprop on the last two images (images 3/4 and 4/4)
              network.back_propagation(outputs[j]);
              network.back_propagation(outputs[j]);

              network.update_weights();
      }
  }

  auto end = std::chrono::system_clock::now();
  bc::print("training time:", clock(end - start).count());

  bc::print("testing...");

  bc::print("saving..............");
  network.save("mnist_test");

  auto shape = bc::shape(784/4, batch_size);
  {

      bc::size_t test_images = 10;
      cube img = cube(inputs[0].reshaped(28,28, batch_size));
      for (int i = 0; i < test_images; ++i) {
          auto batch = inputs[i];
          for (int p = 0; p < img_partitions-1; ++p) {
                      auto index = bc::index(0, 784*(p/(float)img_partitions));
                      network.predict(batch[{index, shape}]);
                  }

          auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
          mat hyps =network.predict(batch[{last_index, shape}]);

          outputs[i][0];
          hyps[0].print();
          bc::print("------------------------------------");
      }
  }

  bc::print("load1");
  network.load("mnist_test");
  network.copy_training_data_to_single_predict(0);
  {

      bc::size_t test_images = 10;
      cube img = cube(inputs[0].reshaped(28,28, batch_size));
      for (int i = 0; i < test_images; ++i) {
          auto batch = inputs[i];
          for (int p = 0; p < img_partitions-1; ++p) {
                      auto index = bc::index(0, 784*(p/(float)img_partitions));
                      network.single_predict(batch[{index, shape}][0]);
                  }

          auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
          auto hyps = network.single_predict(batch[{last_index, shape}][0]);

          outputs[i][0];
          hyps.print();
          bc::print("------------------------------------");
      }
  }

  bc::print("load2");
  network.load("mnist_test");
  network.copy_training_data_to_single_predict(0);
  {

      bc::size_t test_images = 10;
      cube img = cube(inputs[0].reshaped(28,28, batch_size));
      for (int i = 0; i < test_images; ++i) {
          auto batch = inputs[i];
          for (int p = 0; p < img_partitions-1; ++p) {
                      auto index = bc::index(0, 784*(p/(float)img_partitions));
                      network.single_predict(batch[{index, shape}][0]);
                  }

          auto last_index = bc::index(0,784*((img_partitions-1)/(float)img_partitions));
          auto hyps = network.single_predict(batch[{last_index, shape}][0]);

          outputs[i][0];
          hyps.print();
          bc::print("------------------------------------");
      }
  }

  bc::print("success");
  return 0;
}

No, i aslo use the your mnist_text_recurrent example, the same isssue: https://github.com/josephjaspers/blackcat_tensors/issues/53#issuecomment-566343046

[josephjaspers]

Could you send me what process_predict_data is doing?

[xinsuinizhuan]

process_predict_data

this is process_predict_data function:

template bool process_predict_data(System system_tag, LstmPredictTask network, char datafilepath, int _predictdate, std::vector& _outputvec) { using value_type = typename System::default_floating_point_type; using allocator_type = BC::Allocator<System, value_type>; using cube = BC::Cube<value_type, allocator_type>; using mat = BC::Matrix<value_type, allocator_type>; using vec = BC::Vector<value_type, allocator_type>; using clock = std::chrono::duration;

//std::vector<pair<int, std::vector<double>>> normaliInputTimeSeries;
//std::vector<std::vector<double>> maxminSeries;
//std::vector<pair<int, std::vector<double>>> inputTimeSeries;
//vector<vector<double>> matricsInputSeries;
//readwholeline(datafilepath, inputTimeSeries, maxminSeries);
//wholeline_Normalization_maxmin(inputTimeSeries, maxminSeries, normaliInputTimeSeries);
////get_convertMatrixData(normaliInputTimeSeries, _predictdate, network->m_sequence_length, matricsInputSeries);
//get_convertBatchMatrixData(normaliInputTimeSeries, _predictdate, network->m_batch_size, network->m_sequence_length, matricsInputSeries);

std::vector<pair<int, std::vector<double>>> normaliInputTimeSeries;
double maxvalue = 0.0, minvalue = 0.0;
std::vector<pair<int, std::vector<double>>> inputTimeSeries;
vector<vector<double>> matricsInputSeries, matricsOutputSeries;
readwholeline_ex(datafilepath, inputTimeSeries, maxvalue, minvalue);
wholeline_Normalization_maxmin_ex(inputTimeSeries, maxvalue, minvalue, normaliInputTimeSeries);
get_convertMatrix_PredictData(normaliInputTimeSeries, _predictdate, network->m_sequence_length, matricsInputSeries);

if (matricsInputSeries.size() != 1)
{
    return false;
}

int img_sz = network->m_inputs_number * network->m_sequence_length;
if (matricsInputSeries[0].size() != img_sz)
{
    return false;
}

cube inputs(img_sz, 1, 1);
//tensor's memory is uninitialized
inputs.zero();
//inputs[0][0].data_format_convers(matricsInputSeries[0]);
BC::copy(inputs[0][0].get_stream(),matricsInputSeries[0].begin(), matricsInputSeries[0].end(), inputs[0][0].cw_begin());

vec hyps = network->m_pnetwork.single_predict(inputs[0][0]);

std::cout << "single_predict inputdata------------------------------------" << std::endl;
cube imagesinput0 = cube(inputs[0].reshaped(96, 10, 1));
imagesinput0[0].t().print_sparse();

std::cout << "single_predict output predict data------------------------------------" << std::endl;
hyps.print();

//copy data to vector
std::vector<double> predict_vec(hyps.size());
BC::copy(hyps.get_stream(), hyps.cw_begin(), hyps.cw_end(), predict_vec.begin());

//Assign 192 records data to two vector, every 96 record
std::vector<std::vector<double>> outTimeSeries;
for (int i = 0; i < predict_vec.size();)
{
    vector<double> output;
    for (int j = 0; j < network->m_inputs_number; j++)
    {
        output.push_back(predict_vec[i + j]);
    }
    outTimeSeries.push_back(output);
    i += network->m_inputs_number;
}

//data recove
std::vector<std::vector<double>> vec_recovedata;
//wholeline_recove_maxmin(outTimeSeries, maxminSeries, vec_recovedata);
wholeline_recove_maxmin_ex(outTimeSeries, maxvalue, minvalue, vec_recovedata);

//after recover, the merge the two vector data(every 96 record ) to one vector(192 data)
for (int k = 0; k < vec_recovedata.size(); k++)
{
    for (int h = 0; h < vec_recovedata[k].size(); h++)
    {
        _outputvec.push_back(vec_recovedata[k][h]);
    }
}

return true;

}

my whole example file: blackcat_tensors_consoledemo.txt

[josephjaspers]

After

    lstmpredicttask->m_pnetwork.load(_trainparamsavefile);

you should call

    lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0);

---

Here: network.load("mnist_test"); { for (int p = 0; p < img_partitions-1; ++p) { auto index = BC::index(0, 784*(p/(float)img_partitions)); network.predict(batch[{index, shape}]); }

    auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
    mat hyps =network.predict(batch[{last_index, shape}]);

    BC::size_t test_images = 10;
    cube img = cube(inputs[0].reshaped(28,28, batch_size));
    for (int i = 0; i < test_images; ++i) {
        outputs[0][i];
        hyps[i].print();
        BC::print("------------------------------------");
    }
}

BC::print("success");
return 0;

} Did not to need network.copy_training_data_to_single_predict(0)？？

---

I am using regular (batched) predict, so I do not need to copy the data over.

[xinsuinizhuan]

After

    lstmpredicttask->m_pnetwork.load(_trainparamsavefile);

you should call

    lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0);

Here: network.load("mnist_test"); { for (int p = 0; p < img_partitions-1; ++p) { auto index = BC::index(0, 784*(p/(float)img_partitions)); network.predict(batch[{index, shape}]); }

  auto last_index = BC::index(0,784*((img_partitions-1)/(float)img_partitions));
  mat hyps =network.predict(batch[{last_index, shape}]);

  BC::size_t test_images = 10;
  cube img = cube(inputs[0].reshaped(28,28, batch_size));
  for (int i = 0; i < test_images; ++i) {
      outputs[0][i];
      hyps[i].print();
      BC::print("------------------------------------");
  }
}

BC::print("success");
return 0;

} Did not to need network.copy_training_data_to_single_predict(0)？？

I am using regular (batched) predict, so I do not need to copy the data over.

No, there are something wrong with it, it was still a bug. I add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0) after load, but in my project, after load, the same input, output three times still different.

//start train
LstmPredictTask* lstmpredicttask = new LstmPredictTask();
if (lstmpredicttask == NULL) {
    return -2;
}

//LstmPredictTask lstmpredicttask;
std::cout << "Neural Network architecture: \n" <<          lstmpredicttask->m_pnetwork.get_string_architecture() << std::endl;
lstmpredicttask->m_pnetwork.set_learning_rate(lstmpredicttask->m_learning_rate);
lstmpredicttask->m_pnetwork.set_batch_size(lstmpredicttask->m_batch_size);

int training_sets;
std::pair<cube, cube> data = load_train_data(system_tag, datafilepath, lstmpredicttask, &training_sets);
cube& inputs = data.first;
cube& outputs = data.second;

std::cout <<" training..." << std::endl;
auto start = std::chrono::system_clock::now();

std::cout << "imagesinput real data:------------------------------------" << std::endl;
cube imagesinput = cube(inputs[0].reshaped(96, 10, lstmpredicttask->m_batch_size));
imagesinput[0].t().print_sparse();

std::cout << "imagesoutput real data:------------------------------------" << std::endl;
cube imagesoutput = cube(outputs[0].reshaped(96, 2, lstmpredicttask->m_batch_size));
imagesoutput[0].t().print_sparse();

lstmpredicttask->m_pnetwork.load(_trainparamsavefile);
lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0);

std::cout << "process_predict_data first------------------------------------" << std::endl;
std::vector<double> forcast_outputvec;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec);
std::cout << "process_predict_data second------------------------------------" << std::endl;
std::vector<double> forcast_outputvec1;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec1);
std::cout << "process_predict_data third------------------------------------" << std::endl;
std::vector<double> forcast_outputvec2;
process_predict_data(BC::host_tag(), lstmpredicttask, datafilepath, 18080, forcast_outputvec2);

result:

[xinsuinizhuan]

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

[xinsuinizhuan]

when i to single_predict time by time, i think same input, and same output, it should only be use the trained parameters and not to update and change the parameters. I fell it still a bug, it it close to the no-load output, but time by time, the output is different.

[josephjaspers]

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works.

I will try fix the console pop up bug.

The new code compiles on Windows for me so I will have to look into that more.

[xinsuinizhuan]

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works.

I will try fix the console pop up bug.

The new code compiles on Windows for me so I will have to look into that more.

You have the blackcat\string.h file, and the struct name is same as the MVS's string file and class name:

struct string: std::string {

so i think they are confict.

so when i rename the blackcat\string.h to blackcat\bcstring.h, and rename the struct string in blackcat\string.h to struct bcstring. and change the string name to bcstring in the io.h and filesystem.h file, it works.

[josephjaspers]

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works. I will try fix the console pop up bug. The new code compiles on Windows for me so I will have to look into that more.

You have the blackcat\string.h file, and the struct name is same as the MVS's string file and class name:

struct string: std::string {

so i think they are confict.

so when i rename the blackcat\string.h to blackcat\bcstring.h, and rename the struct string in blackcat\string.h to struct bcstring. and change the string name to bcstring in the io.h and filesystem.h file, it works.

My string class is in the bc namespace so it should not have any issues.

Are you directly including the blackcat directory? As that could cause a an inclusion issue with the std:: library.

Try to include the folder containing blackcat opposed to including the blackcat folder directly.

Correct:

include "blackcat/tensors.h"

Incorrect:

include "tensors.h"

[xinsuinizhuan]

I am so sorry.After i load the net , then add the lstmpredicttask->m_pnetwork.copy_training_data_to_single_predict(0), to single_predict, the result is so closely to the no-load single_predict, so this is not still a bug. But when i use the new code, so many compile error, it still the bug, it seems confilict whit the system string, somewheres. And when i save the net, console still pop to print somethings.

Glad to hear it works. I will try fix the console pop up bug. The new code compiles on Windows for me so I will have to look into that more.

You have the blackcat\string.h file, and the struct name is same as the MVS's string file and class name: struct string: std::string { so i think they are confict. so when i rename the blackcat\string.h to blackcat\bcstring.h, and rename the struct string in blackcat\string.h to struct bcstring. and change the string name to bcstring in the io.h and filesystem.h file, it works.

My string class is in the bc namespace so it should not have any issues.

Are you directly including the blackcat directory? As that could cause a an inclusion issue with the std:: library.

Try to include the folder containing blackcat opposed to including the blackcat folder directly.

Correct:

include "blackcat/tensors.h"

Incorrect:

include "tensors.h"

thank you very much. when i use #include "blackcat/tensors.h", it works.

[josephjaspers]

Awesome, glad it worked!

[xinsuinizhuan]

What's next plan? attention-lstm, gpu optimize?

[josephjaspers]

I want to write a few tests and examples first, than I will do attention-lstm