Closed kartikdutt18 closed 4 years ago
Added the Darknet 19 model today, will add Darknet 53 soon.
One more thing, we would probably need to increase input width and input height of cifar 10 since the input width / heights trickles down to 1 in the last few layers. I'll set up my linux machine today that I'll use for training.
Hey @KimSangYeon-DGU, @saksham189, @zoq, When you get a chance could you take a look at the implementation of the Darknet models here. It gives out of memory error but the number of parameters are even less than AlexNet which was implemented earlier.
This is the output size of Conv layers for input of 64, 64, 3. (Darknet 19)
(64, 64, 3) ----> (64, 64, 32)
(32, 32, 32) ----> (32, 32, 64)
(16, 16, 64) ----> (16, 16, 128)
(16, 16, 128) ----> (16, 16, 64)
(16, 16, 64) ----> (16, 16, 128)
(8, 8, 128) ----> (8, 8, 256)
(8, 8, 256) ----> (8, 8, 128)
(8, 8, 128) ----> (8, 8, 256)
(4, 4, 256) ----> (4, 4, 512)
(4, 4, 512) ----> (4, 4, 256)
(4, 4, 256) ----> (4, 4, 512)
(4, 4, 512) ----> (4, 4, 256)
(4, 4, 256) ----> (4, 4, 512)
(2, 2, 1024) ----> (2, 2, 2048)
(2, 2, 2048) ----> (2, 2, 1024)
(2, 2, 1024) ----> (2, 2, 2048)
(2, 2, 1024) ----> (2, 2, 512)
(2, 2, 512) ----> (2, 2, 1024)
(2, 2, 1024) ----> (2, 2, 1000)
Reference. Adding one for 256. For 256, 256, 3. For Conv layers only.
(256, 256, 3) ----> (256, 256, 32)
(128, 128, 32) ----> (128, 128, 64)
(64, 64, 64) ----> (64, 64, 128)
(64, 64, 128) ----> (64, 64, 64)
(64, 64, 64) ----> (64, 64, 128)
(32, 32, 128) ----> (32, 32, 256)
(32, 32, 256) ----> (32, 32, 128)
(32, 32, 128) ----> (32, 32, 256)
(16, 16, 256) ----> (16, 16, 512)
(16, 16, 512) ----> (16, 16, 256)
(16, 16, 256) ----> (16, 16, 512)
(16, 16, 512) ----> (16, 16, 256)
(16, 16, 256) ----> (16, 16, 512)
(8, 8, 1024) ----> (8, 8, 2048)
(8, 8, 2048) ----> (8, 8, 1024)
(8, 8, 1024) ----> (8, 8, 2048)
(8, 8, 1024) ----> (8, 8, 512)
(8, 8, 512) ----> (8, 8, 1024)
(8, 8, 1024) ----> (8, 8, 1000)
This is the output size of Conv layers for input of 64, 64, 3. (Darknet 19)
(64, 64, 3) ----> (64, 64, 32) (32, 32, 32) ----> (32, 32, 64) (16, 16, 64) ----> (16, 16, 128) (16, 16, 128) ----> (16, 16, 64) (16, 16, 64) ----> (16, 16, 128) (8, 8, 128) ----> (8, 8, 256) (8, 8, 256) ----> (8, 8, 128) (8, 8, 128) ----> (8, 8, 256) (4, 4, 256) ----> (4, 4, 512) (4, 4, 512) ----> (4, 4, 256) (4, 4, 256) ----> (4, 4, 512) (4, 4, 512) ----> (4, 4, 256) (4, 4, 256) ----> (4, 4, 512) (2, 2, 1024) ----> (2, 2, 2048) (2, 2, 2048) ----> (2, 2, 1024) (2, 2, 1024) ----> (2, 2, 2048) (2, 2, 1024) ----> (2, 2, 512) (2, 2, 512) ----> (2, 2, 1024) (2, 2, 1024) ----> (2, 2, 1000)
Reference. Adding one for 256. For 256, 256, 3. For Conv layers only.
(256, 256, 3) ----> (256, 256, 32) (128, 128, 32) ----> (128, 128, 64) (64, 64, 64) ----> (64, 64, 128) (64, 64, 128) ----> (64, 64, 64) (64, 64, 64) ----> (64, 64, 128) (32, 32, 128) ----> (32, 32, 256) (32, 32, 256) ----> (32, 32, 128) (32, 32, 128) ----> (32, 32, 256) (16, 16, 256) ----> (16, 16, 512) (16, 16, 512) ----> (16, 16, 256) (16, 16, 256) ----> (16, 16, 512) (16, 16, 512) ----> (16, 16, 256) (16, 16, 256) ----> (16, 16, 512) (8, 8, 1024) ----> (8, 8, 2048) (8, 8, 2048) ----> (8, 8, 1024) (8, 8, 1024) ----> (8, 8, 2048) (8, 8, 1024) ----> (8, 8, 512) (8, 8, 512) ----> (8, 8, 1024) (8, 8, 1024) ----> (8, 8, 1000)
Maybe it's time to push forward with the model summary, would be handy in such a situation.
Maybe it's time to push forward with the model summary, would be handy in such a situation.
Agreed. Does this make sense, Iterate over the archived / serialized model and call the layer name for each layer.
Maybe it's time to push forward with the model summary, would be handy in such a situation.
Agreed. Does this make sense, Iterate over the archived / serialized model and call the layer name for each layer.
Sure, in the past I had the idea to extract the information from the serialized model (xml), but it turned out that would not be the best option.
Sure, in the past I had the idea to extract the information from the serialized model (xml), but it turned out that would not be the best option.
Sure, I'll try to work on it in my spare time. Any other way you suggest that make sense / are better ?
Another update, The model segfaults in both training / prediction. I tried the following things :
With a single layer : Output of GDB,
Thread 2 received signal SIGSEGV, Segmentation fault.
0x00000001004277c8 in _ZNR5boost7variantIPN6mlpack3ann3AddIN4arma3MatIdEES6_EEJPNS2_8AddMergeIS6_S6_JEEEPNS2_12AlphaDropoutIS6_S6_EEPNS2_17AtrousConvolutionINS2_16NaiveConvolutionINS2_16ValidConvolutionEEENSG_INS2_15FullConvolutionEEESI_S6_S6_EEPNS2_9BaseLayerINS2_16LogisticFunctionES6_S6_EEPNSN_INS2_16IdentityFunctionES6_S6_EEPNSN_INS2_12TanhFunctionES6_S6_EEPNSN_INS2_17RectifierFunctionES6_S6_EEPNSN_INS2_16SoftplusFunctionES6_S6_EEPNS2_9BatchNormIS6_S6_EEPNS2_21BilinearInterpolationIS6_S6_EEPNS2_4CELUIS6_S6_EEPNS2_6ConcatIS6_S6_JEEEPNS2_11ConcatenateIS6_S6_EEPNS2_17ConcatPerformanceINS2_21NegativeLogLikelihoodIS6_S6_EES6_S6_EEPNS2_8ConstantIS6_S6_EEPNS2_11ConvolutionISI_SK_SI_S6_S6_EEPNS2_5CReLUIS6_S6_EEPNS2_11DropConnectIS6_S6_JEEEPNS2_7DropoutIS6_S6_EEPNS2_3ELUIS6_S6_EEPNS2_8FastLSTMIS6_S6_EEPNS2_12FlexibleReLUIS6_S6_EEPNS2_7GlimpseIS6_S6_EEPNS2_3GRUIS6_S6_JEEEPNS2_8HardTanHIS6_S6_EEPNS2_7HighwayIS6_S6_JEEEPNS2_4JoinIS6_S6_EEPNS2_9LayerNormIS6_S6_EEPNS2_9LeakyReLUIS6_S6_EEPNS2_6LinearIS6_S6_NS2_13NoRegularizerEEEPNS2_12LinearNoBiasIS6_S6_S2X_EEPNS2_10LogSoftMaxIS6_S6_EEPNS2_6LookupIS6_S6_EEPNS2_4LSTMIS6_S6_EEPNS2_10MaxPoolingIS6_S6_EEPNS2_11MeanPoolingIS6_S6_EEPNS2_23MiniBatchDiscriminationIS6_S6_EEPNS2_16MultiplyConstantIS6_S6_EEPNS2_13MultiplyMergeIS6_S6_JEEEPS1K_PNS2_7PaddingIS6_S6_EEPNS2_5PReLUIS6_S6_EEPNS2_7SoftmaxIS6_S6_EEPNS2_21TransposedConvolutionISI_SI_SI_S6_S6_EEPNS2_10WeightNormIS6_S6_JEEENS0_IPNS2_9RecurrentIS6_S6_JEEEJPNS2_18RecurrentAttentionIS6_S6_JEEEPNS2_15ReinforceNormalIS6_S6_EEPNS2_17ReparametrizationIS6_S6_EEPNS2_6SelectIS6_S6_EEPNS2_10SequentialIS6_S6_Lb0EJEEEPNS4M_IS6_S6_Lb1EJEEEPNS2_7SubviewIS6_S6_EEPNS2_13VRClassRewardIS6_S6_EEPNS2_16VirtualBatchNormIS6_S6_EEEEEEE13apply_visitorINS2_22OutputParameterVisitorEEENT_11result_typeERS54_ ()
Now I'm breaking the network to single layer and see if a subnetwork works.
Valgrind gives an invalid read of size 4,
==60954== Memcheck, a memory error detector
==60954== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==60954== Using Valgrind-3.16.0.GIT and LibVEX; rerun with -h for copyright info
==60954== Command: ./bin/object_classification
==60954==
Loading Dataset!
Dataset Loaded!
Conv Layer. (32, 32, 3) ----> (32, 32, 32)
Model Compiled
Optimizer Created, Starting Training!
26 1
26 3072
Model loaded!
==60954== Invalid read of size 4
==60954== at 0x1004277C8: mlpack::ann::OutputParameterVisitor::result_type boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::apply_visitor<mlpack::ann::OutputParameterVisitor>(mlpack::ann::OutputParameterVisitor&) & (in ./bin/object_classification)
==60954== by 0x100396174: mlpack::ann::OutputParameterVisitor::result_type boost::apply_visitor<mlpack::ann::OutputParameterVisitor, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&>(mlpack::ann::OutputParameterVisitor&, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&) (in ./bin/object_classification)
==60954== by 0x100405BEF: void mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>::Forward<double>(arma::Mat<double> const&, arma::Mat<double>&) (in ./bin/object_classification)
==60954== by 0x100405B96: void mlpack::ann::ForwardVisitor::operator()<mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false> >(mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*) const (in ./bin/object_classification)
==60954== by 0x100405B65: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRPNS4_10SequentialIN4arma3MatIdEESC_Lb0EJEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_SH_EE5valueEvE4typeEOSH_i (in ./bin/object_classification)
==60954== by 0x100405B2B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, mpl_::bool_<true>) (in ./bin/object_classification)
==60954== by 0x1003FC23E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_, int) (in ./bin/object_classification)
==60954== by 0x1003FBA2C: mlpack::ann::ForwardVisitor const::result_type boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::apply_visitor<mlpack::ann::ForwardVisitor const>(mlpack::ann::ForwardVisitor const&) & (in ./bin/object_classification)
==60954== by 0x1003FB80F: mlpack::ann::ForwardVisitor::operator()(boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>) const (in ./bin/object_classification)
==60954== by 0x1003FB79C: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRNS_7variantIPNS4_9RecurrentIN4arma3MatIdEESD_JEEEJPNS4_18RecurrentAttentionISD_SD_JEEEPNS4_15ReinforceNormalISD_SD_EEPNS4_17ReparametrizationISD_SD_EEPNS4_6SelectISD_SD_EEPNS4_10SequentialISD_SD_Lb0EJEEEPNSS_ISD_SD_Lb1EJEEEPNS4_7SubviewISD_SD_EEPNS4_13VRClassRewardISD_SD_EEPNS4_16VirtualBatchNormISD_SD_EEEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_S19_EE5valueEvE4typeEOS19_i (in ./bin/object_classification)
==60954== by 0x1003FB73B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, mpl_::bool_<true>) (in ./bin/object_classification)
==60954== by 0x10039818E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_, int) (in ./bin/object_classification)
==60954== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==60954==
==60954==
==60954== Process terminating with default action of signal 11 (SIGSEGV)
==60954== Access not within mapped region at address 0x0
==60954== at 0x1004277C8: mlpack::ann::OutputParameterVisitor::result_type boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::apply_visitor<mlpack::ann::OutputParameterVisitor>(mlpack::ann::OutputParameterVisitor&) & (in ./bin/object_classification)
==60954== by 0x100396174: mlpack::ann::OutputParameterVisitor::result_type boost::apply_visitor<mlpack::ann::OutputParameterVisitor, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&>(mlpack::ann::OutputParameterVisitor&, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >&) (in ./bin/object_classification)
==60954== by 0x100405BEF: void mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>::Forward<double>(arma::Mat<double> const&, arma::Mat<double>&) (in ./bin/object_classification)
==60954== by 0x100405B96: void mlpack::ann::ForwardVisitor::operator()<mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false> >(mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*) const (in ./bin/object_classification)
==60954== by 0x100405B65: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRPNS4_10SequentialIN4arma3MatIdEESC_Lb0EJEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_SH_EE5valueEvE4typeEOSH_i (in ./bin/object_classification)
==60954== by 0x100405B2B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, mpl_::bool_<true>) (in ./bin/object_classification)
==60954== by 0x1003FC23E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>**, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::has_fallback_type_, int) (in ./bin/object_classification)
==60954== by 0x1003FBA2C: mlpack::ann::ForwardVisitor const::result_type boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>::apply_visitor<mlpack::ann::ForwardVisitor const>(mlpack::ann::ForwardVisitor const&) & (in ./bin/object_classification)
==60954== by 0x1003FB80F: mlpack::ann::ForwardVisitor::operator()(boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>) const (in ./bin/object_classification)
==60954== by 0x1003FB79C: _ZN5boost6detail7variant14invoke_visitorIKN6mlpack3ann14ForwardVisitorELb0EE14internal_visitIRNS_7variantIPNS4_9RecurrentIN4arma3MatIdEESD_JEEEJPNS4_18RecurrentAttentionISD_SD_JEEEPNS4_15ReinforceNormalISD_SD_EEPNS4_17ReparametrizationISD_SD_EEPNS4_6SelectISD_SD_EEPNS4_10SequentialISD_SD_Lb0EJEEEPNSS_ISD_SD_Lb1EJEEEPNS4_7SubviewISD_SD_EEPNS4_13VRClassRewardISD_SD_EEPNS4_16VirtualBatchNormISD_SD_EEEEEEENS_12disable_if_cIXaaLb0Esr7is_sameIT_S19_EE5valueEvE4typeEOS19_i (in ./bin/object_classification)
==60954== by 0x1003FB73B: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, mpl_::bool_<true>) (in ./bin/object_classification)
==60954== by 0x10039818E: boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<mlpack::ann::ForwardVisitor const, false>&, void*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*>*, boost::variant<mlpack::ann::Add<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AddMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::AlphaDropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::AtrousConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::LogisticFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::TanhFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BaseLayer<mlpack::ann::SoftplusFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::BilinearInterpolation<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Concat<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Concatenate<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ConcatPerformance<mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Constant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::CReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::DropConnect<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Dropout<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ELU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FastLSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::FlexibleReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::GRU<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::HardTanH<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::Join<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LayerNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LeakyReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LinearNoBias<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LogSoftMax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Lookup<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::LSTM<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MeanPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MiniBatchDiscrimination<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyConstant<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::MultiplyMerge<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::NegativeLogLikelihood<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PReLU<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Softmax<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::TransposedConvolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::WeightNorm<arma::Mat<double>, arma::Mat<double>>*, boost::variant<mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*> >::has_fallback_type_, int) (in ./bin/object_classification)
==60954== If you believe this happened as a result of a stack
==60954== overflow in your program's main thread (unlikely but
==60954== possible), you can try to increase the size of the
==60954== main thread stack using the --main-stacksize= flag.
==60954== The main thread stack size used in this run was 8388608.
==60954==
==60954== HEAP SUMMARY:
==60954== in use at exit: 15,450,812 bytes in 4,038 blocks
==60954== total heap usage: 4,735 allocs, 697 frees, 44,127,712 bytes allocated
==60954==
==60954== LEAK SUMMARY:
==60954== definitely lost: 1,800 bytes in 10 blocks
==60954== indirectly lost: 0 bytes in 0 blocks
==60954== possibly lost: 153,920 bytes in 187 blocks
==60954== still reachable: 15,295,092 bytes in 3,841 blocks
==60954== suppressed: 0 bytes in 0 blocks
==60954== Rerun with --leak-check=full to see details of leaked memory
==60954==
==60954== For lists of detected and suppressed errors, rerun with: -s
==60954== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 42 from 4)
[1] 60953 segmentation fault sudo valgrind ./bin/object_classification
@kartikdutt18 Can you let me know how to reproduce the same issue as yours? Let me check as well tomorrow morning. It seems we try accessing a variable that already released
Sure, Steps to run:
mkdir build && cd build
sudo make -j2
./bin/object_classification
I am currently trying valgrind + gdb, to get some more insight.
Ok, reproduced.
16th Log : I'll keep updating it every couple of hours. I'll stop after one epoch and add a periodic save as well. Also Kindly let me know if I can implement a graph for the logs while the model trains.
593/998 [===========================================================>........................................] 59% - ETA: 55s - loss: 1.44491
Graph for the first epoch (loss vs iteration):
Graph for the first epoch (loss vs iteration):
Looks good, let me know if you need a remote machine to train the model on.
On subset with 12.5k images with 80% training, One epoch with openmp (12 threads) takes about 9 Hrs (I guess). The training started at roughly 1 PM today, assuming roughly 20 min to load data the model is now at 70 % of its first epoch.
.
I am guessing the loss should start decreasing after the first epoch. On average, we should have roughly 3 epochs a day.
I am guessing the loss should start decreasing after the first epoch. On average, we should have roughly 3 epochs a day.
What system are you using?
I am using MacBook Pro 2019 with the following specs :
2.6 GHz 6-Core Intel Core i7 16 GB 2667 MHz DDR4
Current Params:
constexpr size_t EPOCHS = 10;
constexpr double STEP_SIZE = 1.2e-3;
constexpr int BATCH_SIZE = 32;
Results:
Train accuracy : 10.084384
Validation accuracy : 11.254689
The loss has decreased from 2.9x to 2.8x. With this as reference, I was thinking of increasing learning rate to 0.1 and decreasing batch size to 16 (in case model is trying to generalize).
Since we have the weights of first and soon the second epoch, we can resume training from there. I think Adam is still the best optimizer for this. Kindly let me know your thoughts.
CC: @saksham189, @KimSangYeon-DGU, @zoq
@kartikdutt18 Thanks for the update.
With this graph, I guess you saved the model when 1 epoch was done and re-trained with the model again. The training progress seems slow even almost 2 epochs, so I think increasing the learning rate to 0.1 with Adam optimizer is a good trial with 16 batch size as you said. Let's give it a try. Also if you need a remote machine to train the model, please let @zoq know. This way will make your laptop available for another work :)
With this graph, I guess you saved the model when 1 epoch was done and re-trained with the model again. The training progress seems slow even almost 2 epochs, so I think increasing the learning rate to 0.1 with Adam optimizer is a good trial with 16 batch size as you said. Let's give it a try.
Great, I will make the change.
Also if you need a remote machine to train the model, please let @zoq know. This way will make your laptop available for another work :)
I think my laptop can handle this much for now, incase we increase this size or when we start training version 53 maybe we would have to trouble him. I also started working on YOLO preprocessor while the model is training, I should have a PR in the next couple of days.
Also if you need a remote machine to train the model, please let @zoq know. This way will make your laptop available for another work :)
I think my laptop can handle this much for now, incase we increase this size or when we start training version 53 maybe we would have to trouble him. I also started working on YOLO preprocessor while the model is training, I should have a PR in the next couple of days.
Okay, just let me know; easy for me to give you access to a remote machine if that would make things easier.
Since we have the weights of first and soon the second epoch, we can resume training from there. I think Adam is still the best optimizer for this. Kindly let me know your thoughts.
Agreed, we could test out some other optimizers first but I don't think at this stage it makes a huge difference.
I think I found the right set of training parameters for the model. The problem was with using Xavier Initialization or He Initialization, The loss didn't converge for any learning rate. I also added batch norm layer after the last layer. The learning rate is 1e-3. The training and valid accuracy after first epoch was about 10%. I'm letting this complete 5 epochs and then we can start it again. I think this is good for now. I also found a nice reference Keras implementation of YOLO preprocessor. Link. I am working on generalizing the approach.
I think I found the right set of training parameters for the model. The problem was with using Xavier Initialization or He Initialization, The loss didn't converge for any learning rate. I also added batch norm layer after the last layer. The learning rate is 1e-3. The training and valid accuracy after first epoch was about 10%. I'm letting this complete 5 epochs and then we can start it again. I think this is good for now. I also found a nice reference Keras implementation of YOLO preprocessor. Link. I am working generalizing the approach.
10% accuracy after one epoch would mean just as good as a random algorithm right?
10% accuracy after one epoch would mean just as good as a random algorithm right?
Right. I think it should increase every epoch since the loss is decreasing. Kindly let me know your thoughts.
Alright sounds good. Then we should wait for the training to complete.
Great, Will keep updating the graph above every couple of hours.
The loss is still decreasing, Accuracies for both epochs are given below :
Train accuracy : 9.313470
Validation accuracy : 9.754064
Train accuracy : 12.730180
Validation accuracy : 10.129221
There is a very slow rise in accuracy. We would require a lot of epochs for training if this trend is followed. maybe after this training we can train again with a bit higher learning rate.
Graph for nearly the three epochs.
One more thing, The weights are available for FFN in mlpack. So does something like this make sense : According to this, we can calculate offset in the whole matrix for each layer and then set parameters of ffn (subview according to offset) according to the pretrained weights.
Yes, I think that'll work. The parameter count is similar on both darknet (mlpack & pytorch) hence I think we could do that.
Hey @kartikdutt18. Since PR for BatchNorm got merged, can we work on the last item on the TODO list?
Hey @kartikdutt18 ,
Have you been able to find the difference with eval mode?
I have found the difference in the output keeps increasing and becomes prominent especially around batch norm and leaky relu layer. For example, Consider the these 5 layers and their output through the network :
Output Diff of Conv layer (mlpack | PyTorch): -39,863.2 | -39,838.8086
Output Diff of BatchNorm (mlpack | PyTorch): -32,60,423.129 | -32,12,411.9022
Output Diff of LeakyReLu Layer (mlpack | PyTorch): 8,16,870.236 | 7,51,650.3750
Output Diff of MaxPool Layer (mlpack | PyTorch) : 2,16,609.8613 | 2,00,402.9688
Output Diff of Conv Layer (mlpack | PyTorch) : -2,48,555.4017 | -2,35,535.7188
I've looked through the implementation of both of the above layers and referenced them with PyTorch, they seem correct. I repeated the above for around 10 layers to get an idea. Since the difference keeps increasing the output in eval model isn't correct. For example after the 10th layer the output for mlpack and PyTorch are 97,983.6707 | 56,498.9453
respectively. I'm looking into why the difference is there.
Also, for some reason, We need to manually set deterministic for batchnorm and pooling layers, if I don't the output is way off. Example for the 9th layer the output of mlpack model is -930673.4098
.
Increasing the precision of weights to 128 reduced the difference for the BatchNorm Layer from 1.5% to 0.464% but the output after the leakyReLu layer is still the nearly same.
Output Diff of Conv layer (mlpack | PyTorch): -39,843.2 | -39,838.8086
Output Diff of BatchNorm (mlpack | PyTorch): -32,27,851.915 | -32,12,411.9022
Output Diff of LeakyReLu Layer (mlpack | PyTorch): 8,14,050.0138 | 7,51,650.3750
Output Diff of Conv Layer (mlpack | PyTorch) : -2,47,966.5294 | -2,35,535.7188
@kartikdutt18 Can you save the weight matrices after the first layer for both of mlpack and PyTorch and compre them numerically?
I think we get some insights from that way
@kartikdutt18 Can you save the weight matrices after the first layer for both of mlpack and PyTorch and compre them numerically?
Sure, I can try that. Let me check that now.
@KimSangYeon-DGU, The mlpack weights match PyTorch's weight to a precision of 32 digits.
I think instead of checking on a single input, Let me run it on multiple inputs.
So this difference only shows up during eval mode? and during the training mode there is no difference at all? can you just check this once as well.
So this difference only shows up during eval mode? and during the training mode there is no difference at all? can you just check this once as well.
The final output is the same and in between layers, I tested the difference of output after each 7 - 8 layers, the difference is negligible.
Output Diff of Conv layer (mlpack | PyTorch): -39,843.2 | -39,838.8086
Output Diff of BatchNorm (mlpack | PyTorch): -23,74,227.263 | -23,74,597.7500
Output Diff of LeakyReLu Layer (mlpack | PyTorch): 6,13,343.0759 | 6,13,349.0625
Output Diff of Maxpool Layer (mlpack | PyTorch) : 1,97,138.8427 | 1,97,134.3125
Output Diff of Conv Layer (mlpack | PyTorch) : -218228.6864 | -218223.5156
The difference in training mode in above case is less than 0.002%.
Also, only BatchNorm Layer and MaxPool layer have deterministic option. Each of these layers has unit testing where they verify output from PyTorch.
I set eps = 1e-5 for batchnorm and set deterministic for maxpool manually and got the following results in eval (They are much closer, 0.001 % error) :
(PyTorch) | (mlpack)
Output of layer 1 : tensor(-39838.8086, grad_fn=<SumBackward0>) | -39,838.8086
Output of layer 2 : tensor(-3260192., grad_fn=<SumBackward0>) | -3260423.129
Output of layer 3 : tensor(816813.1250, grad_fn=<SumBackward0>) | 816870.236
Output of layer 4 : tensor(216611.4688, grad_fn=<SumBackward0>) | 216609.8613
Output of layer 5 : tensor(-248550.6250, grad_fn=<SumBackward0>) | -248555.4017
Output of layer 6 : tensor(147896.6094, grad_fn=<SumBackward0>) | 147892.867
Output of layer 7 : tensor(619941., grad_fn=<SumBackward0>) | 619931.0677
Output of layer 8 : tensor(164179.2031, grad_fn=<SumBackward0>) | 164181.2969
I defined the model as follows :
FFN<mlpack::ann::CrossEntropyError<>> model;
model.Add<IdentityLayer<>>();
model.Add<Convolution<>>(3, 32, 3, 3, 1, 1, 1, 1, 224, 224);
model.Add<BatchNorm<>>(32, 1e-5, false);
model.Add<LeakyReLU<>>(0.1);
model.Add<MaxPooling<>>(2, 2, 2, 2);
model.Add<Convolution<>>(32, 64, 3, 3, 1, 1, 1, 1, 112, 112);
model.Add<BatchNorm<>>(64, 1e-5, false);
model.Add<LeakyReLU<>>(0.1);
model.Add<MaxPooling<>>(2, 2, 2, 2);
The final output of Darknet class still differs, I'll go into more layers where the output starts to differ. The only difference in this model and DarkNet class model is that it has sequential layers that wraps (Convolutional, BatchNorm, LeakyReLu and Maxpool). Let me try build the whole Darknet network defining single layers and seeing if the output still differs or finding out where it starts to differ.
Also, only BatchNorm Layer and MaxPool layer have deterministic option. Each of these layers has unit testing where they verify output from PyTorch.
Those unit tests might be missing an edge case. You can try to find the input to these layers in eval mode and try to debug from there.
Maybe the runningMean or runningVariance is slightly different during eval mode. You could look into that.
Maybe the runningMean or runningVariance is slightly different during eval mode. You could look into that.
Agreed. The other thing that I want to try is that if I can declare the model as shown below, does the model work because if the first 10 layers (in both eval and train) give nearly the correct output then so should the whole network.
FFN<mlpack::ann::CrossEntropyError<>> model;
model.Add<IdentityLayer<>>();
model.Add<Convolution<>>(3, 32, 3, 3, 1, 1, 1, 1, 224, 224);
model.Add<BatchNorm<>>(32, 1e-5, false);
model.Add<LeakyReLU<>>(0.1);
model.Add<MaxPooling<>>(2, 2, 2, 2);
model.Add<Convolution<>>(32, 64, 3, 3, 1, 1, 1, 1, 112, 112);
model.Add<BatchNorm<>>(64, 1e-5, false);
model.Add<LeakyReLU<>>(0.1);
model.Add<MaxPooling<>>(2, 2, 2, 2);
yup you could try the smaller network if it is easier to test.
Let me know once you are able to isolate the problem.
Hey everyone, This is is a WIP of DarkNet model for this repo.
TO DO:
[x] Add Basic class definitions.
[x] Add basic definition for tests.
[x] Add Darknet 19 architecture.
[x] Add Darknet 53 architecture.
[ ] Add weights for CIFAR-10 for both models. Add Transfer Learning Function in Each model.
Kindly let me know what you think. Thanks a lot.