search

mlpack / models

models built with mlpack
https://models.mlpack.org/docs
BSD 3-Clause "New" or "Revised" License
35 stars 41 forks source link

Add DarkNet models, Ensmallen Callbacks and ChannelFirst Preprocessor. #20

Closed kartikdutt18 closed 4 years ago

kartikdutt18 commented 4 years ago

Hey everyone, This is is a WIP of DarkNet model for this repo.

TO DO:

Kindly let me know what you think. Thanks a lot.

kartikdutt18 commented 4 years ago

Added the Darknet 19 model today, will add Darknet 53 soon.

kartikdutt18 commented 4 years ago

One more thing, we would probably need to increase input width and input height of cifar 10 since the input width / heights trickles down to 1 in the last few layers. I'll set up my linux machine today that I'll use for training.

kartikdutt18 commented 4 years ago

Hey @KimSangYeon-DGU, @saksham189, @zoq, When you get a chance could you take a look at the implementation of the Darknet models here. It gives out of memory error but the number of parameters are even less than AlexNet which was implemented earlier.

kartikdutt18 commented 4 years ago

This is the output size of Conv layers for input of 64, 64, 3. (Darknet 19)

(64, 64, 3) ----> (64, 64, 32)
(32, 32, 32) ----> (32, 32, 64)
(16, 16, 64) ----> (16, 16, 128)
(16, 16, 128) ----> (16, 16, 64)
(16, 16, 64) ----> (16, 16, 128)
(8, 8, 128) ----> (8, 8, 256)
(8, 8, 256) ----> (8, 8, 128)
(8, 8, 128) ----> (8, 8, 256)
(4, 4, 256) ----> (4, 4, 512)
(4, 4, 512) ----> (4, 4, 256)
(4, 4, 256) ----> (4, 4, 512)
(4, 4, 512) ----> (4, 4, 256)
(4, 4, 256) ----> (4, 4, 512)
(2, 2, 1024) ----> (2, 2, 2048)
(2, 2, 2048) ----> (2, 2, 1024)
(2, 2, 1024) ----> (2, 2, 2048)
(2, 2, 1024) ----> (2, 2, 512)
(2, 2, 512) ----> (2, 2, 1024)
(2, 2, 1024) ----> (2, 2, 1000)

Reference. Adding one for 256. For 256, 256, 3. For Conv layers only.

(256, 256, 3) ----> (256, 256, 32)
(128, 128, 32) ----> (128, 128, 64)
(64, 64, 64) ----> (64, 64, 128)
(64, 64, 128) ----> (64, 64, 64)
(64, 64, 64) ----> (64, 64, 128)
(32, 32, 128) ----> (32, 32, 256)
(32, 32, 256) ----> (32, 32, 128)
(32, 32, 128) ----> (32, 32, 256)
(16, 16, 256) ----> (16, 16, 512)
(16, 16, 512) ----> (16, 16, 256)
(16, 16, 256) ----> (16, 16, 512)
(16, 16, 512) ----> (16, 16, 256)
(16, 16, 256) ----> (16, 16, 512)
(8, 8, 1024) ----> (8, 8, 2048)
(8, 8, 2048) ----> (8, 8, 1024)
(8, 8, 1024) ----> (8, 8, 2048)
(8, 8, 1024) ----> (8, 8, 512)
(8, 8, 512) ----> (8, 8, 1024)
(8, 8, 1024) ----> (8, 8, 1000)
zoq commented 4 years ago

This is the output size of Conv layers for input of 64, 64, 3. (Darknet 19)

(64, 64, 3) ----> (64, 64, 32)
(32, 32, 32) ----> (32, 32, 64)
(16, 16, 64) ----> (16, 16, 128)
(16, 16, 128) ----> (16, 16, 64)
(16, 16, 64) ----> (16, 16, 128)
(8, 8, 128) ----> (8, 8, 256)
(8, 8, 256) ----> (8, 8, 128)
(8, 8, 128) ----> (8, 8, 256)
(4, 4, 256) ----> (4, 4, 512)
(4, 4, 512) ----> (4, 4, 256)
(4, 4, 256) ----> (4, 4, 512)
(4, 4, 512) ----> (4, 4, 256)
(4, 4, 256) ----> (4, 4, 512)
(2, 2, 1024) ----> (2, 2, 2048)
(2, 2, 2048) ----> (2, 2, 1024)
(2, 2, 1024) ----> (2, 2, 2048)
(2, 2, 1024) ----> (2, 2, 512)
(2, 2, 512) ----> (2, 2, 1024)
(2, 2, 1024) ----> (2, 2, 1000)

Reference. Adding one for 256. For 256, 256, 3. For Conv layers only.

(256, 256, 3) ----> (256, 256, 32)
(128, 128, 32) ----> (128, 128, 64)
(64, 64, 64) ----> (64, 64, 128)
(64, 64, 128) ----> (64, 64, 64)
(64, 64, 64) ----> (64, 64, 128)
(32, 32, 128) ----> (32, 32, 256)
(32, 32, 256) ----> (32, 32, 128)
(32, 32, 128) ----> (32, 32, 256)
(16, 16, 256) ----> (16, 16, 512)
(16, 16, 512) ----> (16, 16, 256)
(16, 16, 256) ----> (16, 16, 512)
(16, 16, 512) ----> (16, 16, 256)
(16, 16, 256) ----> (16, 16, 512)
(8, 8, 1024) ----> (8, 8, 2048)
(8, 8, 2048) ----> (8, 8, 1024)
(8, 8, 1024) ----> (8, 8, 2048)
(8, 8, 1024) ----> (8, 8, 512)
(8, 8, 512) ----> (8, 8, 1024)
(8, 8, 1024) ----> (8, 8, 1000)

Maybe it's time to push forward with the model summary, would be handy in such a situation.

kartikdutt18 commented 4 years ago

Maybe it's time to push forward with the model summary, would be handy in such a situation.

Agreed. Does this make sense, Iterate over the archived / serialized model and call the layer name for each layer.

zoq commented 4 years ago

Maybe it's time to push forward with the model summary, would be handy in such a situation.

Agreed. Does this make sense, Iterate over the archived / serialized model and call the layer name for each layer.

Sure, in the past I had the idea to extract the information from the serialized model (xml), but it turned out that would not be the best option.

kartikdutt18 commented 4 years ago

Sure, in the past I had the idea to extract the information from the serialized model (xml), but it turned out that would not be the best option.

Sure, I'll try to work on it in my spare time. Any other way you suggest that make sense / are better ?

kartikdutt18 commented 4 years ago

Another update, The model segfaults in both training / prediction. I tried the following things :

  1. Add copy constructors from Himanshu's PR.
  2. Changed dimensions.
  3. Indexing labels from 1 instead of 0.
  4. Removed global average pooling layer.
  5. Wrote test in mlpack to see if any input dimensions caused failure.

With a single layer : Output of GDB,

Thread 2 received signal SIGSEGV, Segmentation fault.
0x00000001004277c8 in _ZNR5boost7variantIPN6mlpack3ann3AddIN4arma3MatIdEES6_EEJPNS2_8AddMergeIS6_S6_JEEEPNS2_12AlphaDropoutIS6_S6_EEPNS2_17AtrousConvolutionINS2_16NaiveConvolutionINS2_16ValidConvolutionEEENSG_INS2_15FullConvolutionEEESI_S6_S6_EEPNS2_9BaseLayerINS2_16LogisticFunctionES6_S6_EEPNSN_INS2_16IdentityFunctionES6_S6_EEPNSN_INS2_12TanhFunctionES6_S6_EEPNSN_INS2_17RectifierFunctionES6_S6_EEPNSN_INS2_16SoftplusFunctionES6_S6_EEPNS2_9BatchNormIS6_S6_EEPNS2_21BilinearInterpolationIS6_S6_EEPNS2_4CELUIS6_S6_EEPNS2_6ConcatIS6_S6_JEEEPNS2_11ConcatenateIS6_S6_EEPNS2_17ConcatPerformanceINS2_21NegativeLogLikelihoodIS6_S6_EES6_S6_EEPNS2_8ConstantIS6_S6_EEPNS2_11ConvolutionISI_SK_SI_S6_S6_EEPNS2_5CReLUIS6_S6_EEPNS2_11DropConnectIS6_S6_JEEEPNS2_7DropoutIS6_S6_EEPNS2_3ELUIS6_S6_EEPNS2_8FastLSTMIS6_S6_EEPNS2_12FlexibleReLUIS6_S6_EEPNS2_7GlimpseIS6_S6_EEPNS2_3GRUIS6_S6_JEEEPNS2_8HardTanHIS6_S6_EEPNS2_7HighwayIS6_S6_JEEEPNS2_4JoinIS6_S6_EEPNS2_9LayerNormIS6_S6_EEPNS2_9LeakyReLUIS6_S6_EEPNS2_6LinearIS6_S6_NS2_13NoRegularizerEEEPNS2_12LinearNoBiasIS6_S6_S2X_EEPNS2_10LogSoftMaxIS6_S6_EEPNS2_6LookupIS6_S6_EEPNS2_4LSTMIS6_S6_EEPNS2_10MaxPoolingIS6_S6_EEPNS2_11MeanPoolingIS6_S6_EEPNS2_23MiniBatchDiscriminationIS6_S6_EEPNS2_16MultiplyConstantIS6_S6_EEPNS2_13MultiplyMergeIS6_S6_JEEEPS1K_PNS2_7PaddingIS6_S6_EEPNS2_5PReLUIS6_S6_EEPNS2_7SoftmaxIS6_S6_EEPNS2_21TransposedConvolutionISI_SI_SI_S6_S6_EEPNS2_10WeightNormIS6_S6_JEEENS0_IPNS2_9RecurrentIS6_S6_JEEEJPNS2_18RecurrentAttentionIS6_S6_JEEEPNS2_15ReinforceNormalIS6_S6_EEPNS2_17ReparametrizationIS6_S6_EEPNS2_6SelectIS6_S6_EEPNS2_10SequentialIS6_S6_Lb0EJEEEPNS4M_IS6_S6_Lb1EJEEEPNS2_7SubviewIS6_S6_EEPNS2_13VRClassRewardIS6_S6_EEPNS2_16VirtualBatchNormIS6_S6_EEEEEEE13apply_visitorINS2_22OutputParameterVisitorEEENT_11result_typeERS54_ ()

Now I'm breaking the network to single layer and see if a subnetwork works.

kartikdutt18 commented 4 years ago

Valgrind gives an invalid read of size 4,

==60954== Memcheck, a memory error detector
==60954== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==60954== Using Valgrind-3.16.0.GIT and LibVEX; rerun with -h for copyright info
==60954== Command: ./bin/object_classification
==60954== 
Loading Dataset!
Dataset Loaded!
Conv Layer.  (32, 32, 3) ----> (32, 32, 32)
Model Compiled
Optimizer Created, Starting Training!
26 1
26 3072
Model loaded!
==60954== Invalid read of size 4
==60954==    at 0x1004277C8: mlpack::ann::OutputParameterVisitor::result_type boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::apply_visitor<mlpack::ann::OutputParameterVisitor>(mlpack::ann::OutputParameterVisitor&) & (in ./bin/object_classification)
==60954==    by 0x100396174: mlpack::ann::OutputParameterVisitor::result_type boost::apply_visitor<mlpack::ann::OutputParameterVisitor, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&>(mlpack::ann::OutputParameterVisitor&, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&) (in ./bin/object_classification)
==60954==    by 0x100405BEF: void mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>::Forward<double>(arma::Mat<double> const&, arma::Mat<double>&) (in ./bin/object_classification)
==60954==    by 0x100405B96: void mlpack::ann::ForwardVisitor::operator()<mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false> >(mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*) const (in ./bin/object_classification)
==60954==    by 0x100405B65: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRPNS4_10SequentialIN4arma3MatIdEESC_Lb0EJEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_SH_EE5valueEvE4typeEOSH_i (in ./bin/object_classification)
==60954==    by 0x100405B2B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, mpl_::bool_<true>) (in ./bin/object_classification)
==60954==    by 0x1003FC23E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_, int) (in ./bin/object_classification)
==60954==    by 0x1003FBA2C: mlpack::ann::ForwardVisitor const::result_type boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::apply_visitor<mlpack::ann::ForwardVisitor const>(mlpack::ann::ForwardVisitor const&) & (in ./bin/object_classification)
==60954==    by 0x1003FB80F: mlpack::ann::ForwardVisitor::operator()(boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>) const (in ./bin/object_classification)
==60954==    by 0x1003FB79C: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRNS_7variantIPNS4_9RecurrentIN4arma3MatIdEESD_JEEEJPNS4_18RecurrentAttentionISD_SD_JEEEPNS4_15ReinforceNormalISD_SD_EEPNS4_17ReparametrizationISD_SD_EEPNS4_6SelectISD_SD_EEPNS4_10SequentialISD_SD_Lb0EJEEEPNSS_ISD_SD_Lb1EJEEEPNS4_7SubviewISD_SD_EEPNS4_13VRClassRewardISD_SD_EEPNS4_16VirtualBatchNormISD_SD_EEEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_S19_EE5valueEvE4typeEOS19_i (in ./bin/object_classification)
==60954==    by 0x1003FB73B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, mpl_::bool_<true>) (in ./bin/object_classification)
==60954==    by 0x10039818E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_, int) (in ./bin/object_classification)
==60954==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==60954== 
==60954== 
==60954== Process terminating with default action of signal 11 (SIGSEGV)
==60954==  Access not within mapped region at address 0x0
==60954==    at 0x1004277C8: mlpack::ann::OutputParameterVisitor::result_type boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::apply_visitor<mlpack::ann::OutputParameterVisitor>(mlpack::ann::OutputParameterVisitor&) & (in ./bin/object_classification)
==60954==    by 0x100396174: mlpack::ann::OutputParameterVisitor::result_type boost::apply_visitor<mlpack::ann::OutputParameterVisitor, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&>(mlpack::ann::OutputParameterVisitor&, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&) (in ./bin/object_classification)
==60954==    by 0x100405BEF: void mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>::Forward<double>(arma::Mat<double> const&, arma::Mat<double>&) (in ./bin/object_classification)
==60954==    by 0x100405B96: void mlpack::ann::ForwardVisitor::operator()<mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false> >(mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*) const (in ./bin/object_classification)
==60954==    by 0x100405B65: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRPNS4_10SequentialIN4arma3MatIdEESC_Lb0EJEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_SH_EE5valueEvE4typeEOSH_i (in ./bin/object_classification)
==60954==    by 0x100405B2B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, mpl_::bool_<true>) (in ./bin/object_classification)
==60954==    by 0x1003FC23E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_, int) (in ./bin/object_classification)
==60954==    by 0x1003FBA2C: mlpack::ann::ForwardVisitor const::result_type boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::apply_visitor<mlpack::ann::ForwardVisitor const>(mlpack::ann::ForwardVisitor const&) & (in ./bin/object_classification)
==60954==    by 0x1003FB80F: mlpack::ann::ForwardVisitor::operator()(boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>) const (in ./bin/object_classification)
==60954==    by 0x1003FB79C: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRNS_7variantIPNS4_9RecurrentIN4arma3MatIdEESD_JEEEJPNS4_18RecurrentAttentionISD_SD_JEEEPNS4_15ReinforceNormalISD_SD_EEPNS4_17ReparametrizationISD_SD_EEPNS4_6SelectISD_SD_EEPNS4_10SequentialISD_SD_Lb0EJEEEPNSS_ISD_SD_Lb1EJEEEPNS4_7SubviewISD_SD_EEPNS4_13VRClassRewardISD_SD_EEPNS4_16VirtualBatchNormISD_SD_EEEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_S19_EE5valueEvE4typeEOS19_i (in ./bin/object_classification)
==60954==    by 0x1003FB73B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, mpl_::bool_<true>) (in ./bin/object_classification)
==60954==    by 0x10039818E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_, int) (in ./bin/object_classification)
==60954==  If you believe this happened as a result of a stack
==60954==  overflow in your program's main thread (unlikely but
==60954==  possible), you can try to increase the size of the
==60954==  main thread stack using the --main-stacksize= flag.
==60954==  The main thread stack size used in this run was 8388608.
==60954== 
==60954== HEAP SUMMARY:
==60954==     in use at exit: 15,450,812 bytes in 4,038 blocks
==60954==   total heap usage: 4,735 allocs, 697 frees, 44,127,712 bytes allocated
==60954== 
==60954== LEAK SUMMARY:
==60954==    definitely lost: 1,800 bytes in 10 blocks
==60954==    indirectly lost: 0 bytes in 0 blocks
==60954==      possibly lost: 153,920 bytes in 187 blocks
==60954==    still reachable: 15,295,092 bytes in 3,841 blocks
==60954==         suppressed: 0 bytes in 0 blocks
==60954== Rerun with --leak-check=full to see details of leaked memory
==60954== 
==60954== For lists of detected and suppressed errors, rerun with: -s
==60954== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 42 from 4)
[1]    60953 segmentation fault  sudo valgrind ./bin/object_classification
KimSangYeon-DGU commented 4 years ago

@kartikdutt18 Can you let me know how to reproduce the same issue as yours? Let me check as well tomorrow morning. It seems we try accessing a variable that already released

kartikdutt18 commented 4 years ago

Sure, Steps to run:

mkdir build && cd build
sudo make -j2
./bin/object_classification

I am currently trying valgrind + gdb, to get some more insight.

KimSangYeon-DGU commented 4 years ago

Ok, reproduced.

kartikdutt18 commented 4 years ago

16th Log : I'll keep updating it every couple of hours. I'll stop after one epoch and add a periodic save as well. Also Kindly let me know if I can implement a graph for the logs while the model trains.

593/998 [===========================================================>........................................] 59% - ETA: 55s - loss: 1.44491
kartikdutt18 commented 4 years ago

Graph for the first epoch (loss vs iteration): 1st_epoch

zoq commented 4 years ago

Graph for the first epoch (loss vs iteration): 1st_epoch

Looks good, let me know if you need a remote machine to train the model on.

kartikdutt18 commented 4 years ago

On subset with 12.5k images with 80% training, One epoch with openmp (12 threads) takes about 9 Hrs (I guess). The training started at roughly 1 PM today, assuming roughly 20 min to load data the model is now at 70 % of its first epoch. 2_log_epoch. I am guessing the loss should start decreasing after the first epoch. On average, we should have roughly 3 epochs a day.

zoq commented 4 years ago

I am guessing the loss should start decreasing after the first epoch. On average, we should have roughly 3 epochs a day.

What system are you using?

kartikdutt18 commented 4 years ago

I am using MacBook Pro 2019 with the following specs :

2.6 GHz 6-Core Intel Core i7 16 GB 2667 MHz DDR4

kartikdutt18 commented 4 years ago

Current Params:

constexpr size_t EPOCHS = 10;
constexpr double STEP_SIZE = 1.2e-3;
constexpr int BATCH_SIZE = 32;

Results: 2_log_epoch

Train accuracy : 10.084384
Validation accuracy : 11.254689

The loss has decreased from 2.9x to 2.8x. With this as reference, I was thinking of increasing learning rate to 0.1 and decreasing batch size to 16 (in case model is trying to generalize).

Since we have the weights of first and soon the second epoch, we can resume training from there. I think Adam is still the best optimizer for this. Kindly let me know your thoughts.

CC: @saksham189, @KimSangYeon-DGU, @zoq

KimSangYeon-DGU commented 4 years ago

@kartikdutt18 Thanks for the update.

With this graph, I guess you saved the model when 1 epoch was done and re-trained with the model again. The training progress seems slow even almost 2 epochs, so I think increasing the learning rate to 0.1 with Adam optimizer is a good trial with 16 batch size as you said. Let's give it a try. Also if you need a remote machine to train the model, please let @zoq know. This way will make your laptop available for another work :)

kartikdutt18 commented 4 years ago

With this graph, I guess you saved the model when 1 epoch was done and re-trained with the model again. The training progress seems slow even almost 2 epochs, so I think increasing the learning rate to 0.1 with Adam optimizer is a good trial with 16 batch size as you said. Let's give it a try.

Great, I will make the change.

Also if you need a remote machine to train the model, please let @zoq know. This way will make your laptop available for another work :)

I think my laptop can handle this much for now, incase we increase this size or when we start training version 53 maybe we would have to trouble him. I also started working on YOLO preprocessor while the model is training, I should have a PR in the next couple of days.

zoq commented 4 years ago

Also if you need a remote machine to train the model, please let @zoq know. This way will make your laptop available for another work :)

I think my laptop can handle this much for now, incase we increase this size or when we start training version 53 maybe we would have to trouble him. I also started working on YOLO preprocessor while the model is training, I should have a PR in the next couple of days.

Okay, just let me know; easy for me to give you access to a remote machine if that would make things easier.

zoq commented 4 years ago

Since we have the weights of first and soon the second epoch, we can resume training from there. I think Adam is still the best optimizer for this. Kindly let me know your thoughts.

Agreed, we could test out some other optimizers first but I don't think at this stage it makes a huge difference.

kartikdutt18 commented 4 years ago

4_mean_log_epoch 4_log_epoch I think I found the right set of training parameters for the model. The problem was with using Xavier Initialization or He Initialization, The loss didn't converge for any learning rate. I also added batch norm layer after the last layer. The learning rate is 1e-3. The training and valid accuracy after first epoch was about 10%. I'm letting this complete 5 epochs and then we can start it again. I think this is good for now. I also found a nice reference Keras implementation of YOLO preprocessor. Link. I am working on generalizing the approach.

saksham189 commented 4 years ago

I think I found the right set of training parameters for the model. The problem was with using Xavier Initialization or He Initialization, The loss didn't converge for any learning rate. I also added batch norm layer after the last layer. The learning rate is 1e-3. The training and valid accuracy after first epoch was about 10%. I'm letting this complete 5 epochs and then we can start it again. I think this is good for now. I also found a nice reference Keras implementation of YOLO preprocessor. Link. I am working generalizing the approach.

10% accuracy after one epoch would mean just as good as a random algorithm right?

kartikdutt18 commented 4 years ago

10% accuracy after one epoch would mean just as good as a random algorithm right?

Right. I think it should increase every epoch since the loss is decreasing. Kindly let me know your thoughts.

saksham189 commented 4 years ago

Alright sounds good. Then we should wait for the training to complete.

kartikdutt18 commented 4 years ago

Great, Will keep updating the graph above every couple of hours.

kartikdutt18 commented 4 years ago

4_log_epoch The loss is still decreasing, Accuracies for both epochs are given below : 

Train accuracy : 9.313470
Validation accuracy : 9.754064

Train accuracy : 12.730180
Validation accuracy : 10.129221

There is a very slow rise in accuracy. We would require a lot of epochs for training if this trend is followed. maybe after this training we can train again with a bit higher learning rate.

kartikdutt18 commented 4 years ago

4_mean_log_epoch 4_log_epoch Graph for nearly the three epochs.

kartikdutt18 commented 4 years ago

One more thing, The weights are available for FFN in mlpack. So does something like this make sense : According to this, we can calculate offset in the whole matrix for each layer and then set parameters of ffn (subview according to offset) according to the pretrained weights.

kartikdutt18 commented 4 years ago

Yes, I think that'll work. The parameter count is similar on both darknet (mlpack & pytorch) hence I think we could do that.

KimSangYeon-DGU commented 4 years ago

Hey @kartikdutt18. Since PR for BatchNorm got merged, can we work on the last item on the TODO list?

saksham189 commented 4 years ago

Hey @kartikdutt18 ,

Have you been able to find the difference with eval mode?

kartikdutt18 commented 4 years ago

I have found the difference in the output keeps increasing and becomes prominent especially around batch norm and leaky relu layer. For example, Consider the these 5 layers and their output through the network :

Output Diff of Conv layer (mlpack | PyTorch): -39,863.2  | -39,838.8086
Output Diff of BatchNorm (mlpack | PyTorch):  -32,60,423.129 | -32,12,411.9022
Output Diff of LeakyReLu Layer (mlpack | PyTorch): 8,16,870.236 | 7,51,650.3750
Output Diff of MaxPool Layer (mlpack | PyTorch) : 2,16,609.8613 | 2,00,402.9688
Output Diff of Conv Layer (mlpack | PyTorch) : -2,48,555.4017 | -2,35,535.7188

I've looked through the implementation of both of the above layers and referenced them with PyTorch, they seem correct. I repeated the above for around 10 layers to get an idea. Since the difference keeps increasing the output in eval model isn't correct. For example after the 10th layer the output for mlpack and PyTorch are 97,983.6707 | 56,498.9453 respectively. I'm looking into why the difference is there. Also, for some reason, We need to manually set deterministic for batchnorm and pooling layers, if I don't the output is way off. Example for the 9th layer the output of mlpack model is -930673.4098.

kartikdutt18 commented 4 years ago

Increasing the precision of weights to 128 reduced the difference for the BatchNorm Layer from 1.5% to 0.464% but the output after the leakyReLu layer is still the nearly same.

Output Diff of Conv layer (mlpack | PyTorch): -39,843.2  | -39,838.8086
Output Diff of BatchNorm (mlpack | PyTorch):  -32,27,851.915 | -32,12,411.9022
Output Diff of  LeakyReLu Layer (mlpack | PyTorch): 8,14,050.0138 | 7,51,650.3750
Output Diff of Conv Layer (mlpack | PyTorch) : -2,47,966.5294 | -2,35,535.7188
KimSangYeon-DGU commented 4 years ago

@kartikdutt18 Can you save the weight matrices after the first layer for both of mlpack and PyTorch and compre them numerically?

I think we get some insights from that way

kartikdutt18 commented 4 years ago

@kartikdutt18 Can you save the weight matrices after the first layer for both of mlpack and PyTorch and compre them numerically?

Sure, I can try that. Let me check that now.

kartikdutt18 commented 4 years ago

@KimSangYeon-DGU, The mlpack weights match PyTorch's weight to a precision of 32 digits.

kartikdutt18 commented 4 years ago

I think instead of checking on a single input, Let me run it on multiple inputs.

saksham189 commented 4 years ago

So this difference only shows up during eval mode? and during the training mode there is no difference at all? can you just check this once as well.

kartikdutt18 commented 4 years ago

So this difference only shows up during eval mode? and during the training mode there is no difference at all? can you just check this once as well.

The final output is the same and in between layers, I tested the difference of output after each 7 - 8 layers, the difference is negligible.

Output Diff of Conv layer (mlpack | PyTorch): -39,843.2  | -39,838.8086
Output Diff of BatchNorm (mlpack | PyTorch):  -23,74,227.263 | -23,74,597.7500
Output Diff of  LeakyReLu Layer (mlpack | PyTorch): 6,13,343.0759 | 6,13,349.0625
Output Diff of Maxpool Layer (mlpack | PyTorch) : 1,97,138.8427 | 1,97,134.3125
Output Diff of Conv Layer (mlpack | PyTorch) : -218228.6864 | -218223.5156

The difference in training mode in above case is less than 0.002%.

kartikdutt18 commented 4 years ago

Also, only BatchNorm Layer and MaxPool layer have deterministic option. Each of these layers has unit testing where they verify output from PyTorch.

kartikdutt18 commented 4 years ago

I set eps = 1e-5 for batchnorm and set deterministic for maxpool manually and got the following results in eval (They are much closer, 0.001 % error) :

(PyTorch) | (mlpack)
Output of layer  1  :  tensor(-39838.8086, grad_fn=<SumBackward0>) | -39,838.8086
Output of layer  2  :  tensor(-3260192., grad_fn=<SumBackward0>) | -3260423.129
Output of layer  3  :  tensor(816813.1250, grad_fn=<SumBackward0>) | 816870.236
Output of layer  4  :  tensor(216611.4688, grad_fn=<SumBackward0>) | 216609.8613
Output of layer  5  :  tensor(-248550.6250, grad_fn=<SumBackward0>) | -248555.4017
Output of layer  6  :  tensor(147896.6094, grad_fn=<SumBackward0>) | 147892.867
Output of layer  7  :  tensor(619941., grad_fn=<SumBackward0>) | 619931.0677
Output of layer  8  :  tensor(164179.2031, grad_fn=<SumBackward0>) | 164181.2969

I defined the model as follows : 

  FFN<mlpack::ann::CrossEntropyError<>> model;
  model.Add<IdentityLayer<>>();
  model.Add<Convolution<>>(3, 32, 3, 3, 1, 1, 1, 1, 224, 224);
  model.Add<BatchNorm<>>(32, 1e-5, false);
  model.Add<LeakyReLU<>>(0.1);
  model.Add<MaxPooling<>>(2, 2, 2, 2);
  model.Add<Convolution<>>(32, 64, 3, 3, 1, 1, 1, 1, 112, 112);
  model.Add<BatchNorm<>>(64, 1e-5, false);
  model.Add<LeakyReLU<>>(0.1);
  model.Add<MaxPooling<>>(2, 2, 2, 2);

The final output of Darknet class still differs, I'll go into more layers where the output starts to differ. The only difference in this model and DarkNet class model is that it has sequential layers that wraps (Convolutional, BatchNorm, LeakyReLu and Maxpool). Let me try build the whole Darknet network defining single layers and seeing if the output still differs or finding out where it starts to differ.

saksham189 commented 4 years ago

Also, only BatchNorm Layer and MaxPool layer have deterministic option. Each of these layers has unit testing where they verify output from PyTorch.

Those unit tests might be missing an edge case. You can try to find the input to these layers in eval mode and try to debug from there.

saksham189 commented 4 years ago

Maybe the runningMean or runningVariance is slightly different during eval mode. You could look into that.

kartikdutt18 commented 4 years ago

Maybe the runningMean or runningVariance is slightly different during eval mode. You could look into that.

Agreed. The other thing that I want to try is that if I can declare the model as shown below, does the model work because if the first 10 layers (in both eval and train) give nearly the correct output then so should the whole network.

  FFN<mlpack::ann::CrossEntropyError<>> model;
  model.Add<IdentityLayer<>>();
  model.Add<Convolution<>>(3, 32, 3, 3, 1, 1, 1, 1, 224, 224);
  model.Add<BatchNorm<>>(32, 1e-5, false);
  model.Add<LeakyReLU<>>(0.1);
  model.Add<MaxPooling<>>(2, 2, 2, 2);
  model.Add<Convolution<>>(32, 64, 3, 3, 1, 1, 1, 1, 112, 112);
  model.Add<BatchNorm<>>(64, 1e-5, false);
  model.Add<LeakyReLU<>>(0.1);
  model.Add<MaxPooling<>>(2, 2, 2, 2);
saksham189 commented 4 years ago

yup you could try the smaller network if it is easier to test.

saksham189 commented 4 years ago

Let me know once you are able to isolate the problem.