Addition of resnet module.

[Aakash-kaushik]

Extension of #61 and a PR for review of the Additon of resnet module.

[Aakash-kaushik]

Btw i just realised right now that the load model or data load functions in mlpack they don't require an architecture and when you load the weight or the bin file it also creates an architecture before this is just thought it was similar to just saving weights in other framework rather than just the complete architecture with weights. Do correct me if i am wrong.

[Aakash-kaushik]

And a thing i did with this was just have a bool preTrained that loads imageNet weights and not have a string for weights as that can be loaded with the Resnet object by using LoadModel, i tried making it a bit similar to other frameworks, let me know what you guys think about it?

[Aakash-kaushik]

Also i wanted to know if our conv layer by default has the bias param added ? and is there any way to turn that off ?

[Aakash-kaushik]

Left is pytroch's resnet18 params and on the right the one i implemented, I guess one more things i would need to rectify now.

[kartikdutt18]

The difference could be because of bias params, Have you checked that?

[Aakash-kaushik]

The difference could be because of bias params, Have you checked that?

Yeah you were right they were the extra bias params, no work for me. :p

[zoq]

In the call today we said it might be a good idea to check the parameter size for each layer as well:

https://github.com/mlpack/mlpack/blob/68ff28175d1103c092d4c3f0bc495c450ecd40d2/src/mlpack/methods/ann/init_rules/network_init.hpp#L64-L65

is a simple example that uses the WeightSizeVisitor visitor.

[Aakash-kaushik]

In the call today we said it might be a good idea to check the parameter size for each layer as well:

https://github.com/mlpack/mlpack/blob/68ff28175d1103c092d4c3f0bc495c450ecd40d2/src/mlpack/methods/ann/init_rules/network_init.hpp#L64-L65

is a simple example that uses the WeightSizeVisitor visitor.

Yup i can do this too, Should be more reinforcing to have a lyer wise check

[Aakash-kaushik]

Hey @kartikdutt18 can you take me through the weight_converter.cpp I have converted weights for resnets and have a file that can do that for every resent i mean it's very hacky but does the job. so now only the cpp part is left and i can't really get that.

or maybe give me sometime, I will let you know if i need help.

[Aakash-kaushik]

Hey @kartikdutt18 can you take me through the weight_converter.cpp I have converted weights for resnets and have a file that can do that for every resent i mean it's very hacky but does the job. so now only the cpp part is left and i can't really get that.

or maybe give me sometime, I will let you know if i need help.

Hey do you know where would this be situated ? i get this when i run the converter from c++ side.

[Aakash-kaushik]

And I am also confused on how this accounts for the shorcut connection and the indentity layer. so if you are free today can we talk ? modification to your translator code: https://github.com/Aakash-kaushik/mlpack-PyTorch-Weight-Translator though i would like to add again it is only done to make it work and the code looks real ugly but again does the job and sequentially extracts layers not sure about skip connections and other things we did to make resnet work.

@kartikdutt18 @zoq

[kartikdutt18]

I won't be able to have a meet today, Can we have one tomorrow (around 730 PM IST).

[Aakash-kaushik]

Works with me. Marcus, would you want to join?

[zoq]

Works with me. Marcus, would you want to join?

Yes, works for me as well.

[Aakash-kaushik]

the boost::get thing is a bit confusing, I am trying to go over it but can't understand much of it.

reference: https://github.com/Aakash-kaushik/mlpack-PyTorch-Weight-Translator/blob/master/sample_examples/test_models/resnet.cpp#L160-L192

[kartikdutt18]

This portion loads BN running mean and variance. This is model dependent. Feel free to write hardcoded version for loading running mean and variance for Resnet.

[Aakash-kaushik]

This portion loads BN running mean and variance. This is model dependent. Feel free to write hardcoded version for loading running mean and variance for Resnet.

Thanks i was just confused as to how those boost::get workes i will have to read the ducmentation or something if that's available for them.

[Aakash-kaushik]

Hey @kartikdutt18 I edited the files for resnet18 but i get an error while calling for the sequential layer through boost::get it basically says that the variant is not available, even though i can see the exact same variant being available in the error list.

here is the line: https://github.com/Aakash-kaushik/mlpack-PyTorch-Weight-Translator/blob/master/sample_examples/test_models/resnet.cpp#L175

And honestly sorry to ask this on a sunday but would you be able to have a meet on sunday(27th) to resolve this ?

[Aakash-kaushik]

here is the error: https://gist.github.com/Aakash-kaushik/e66e37396ed86785716df850bd0afb7d

[zoq]

I haven't test it yet, but what happens if you put:

https://github.com/mlpack/mlpack/blob/0d9de4b541de12413dcd77d184390a3a59b2cb59/src/mlpack/methods/ann/layer/layer_types.hpp#L237-L238

into LayerTypes instead of MoreTypes, you might have to move some layers from LayerTypes into MoreTypes, in this case.

[kartikdutt18]

Lets have a short call around 5 PM IST

[kartikdutt18]

I haven't test it yet, but what happens if you put:

https://github.com/mlpack/mlpack/blob/0d9de4b541de12413dcd77d184390a3a59b2cb59/src/mlpack/methods/ann/layer/layer_types.hpp#L237-L238

into LayerTypes instead of MoreTypes, you might have to move some layers from LayerTypes into MoreTypes, in this case.

Agreed, this should fix it.

[Aakash-kaushik]

I haven't test it yet, but what happens if you put:

https://github.com/mlpack/mlpack/blob/0d9de4b541de12413dcd77d184390a3a59b2cb59/src/mlpack/methods/ann/layer/layer_types.hpp#L237-L238

into LayerTypes instead of MoreTypes, you might have to move some layers from LayerTypes into MoreTypes, in this case.

Just woke up, I will try this and let you know the results

[Aakash-kaushik]

Lets have a short call around 5 PM IST

Btw if you can still make it, that would be awesome.

[kartikdutt18]

Sure, will be there.

[Aakash-kaushik]

Sure, will be there.

Thanks :D

[Aakash-kaushik]

I haven't test it yet, but what happens if you put:

https://github.com/mlpack/mlpack/blob/0d9de4b541de12413dcd77d184390a3a59b2cb59/src/mlpack/methods/ann/layer/layer_types.hpp#L237-L238

into LayerTypes instead of MoreTypes, you might have to move some layers from LayerTypes into MoreTypes, in this case.

I am not 100% sure if this is the case because i wrote some code to get the type of data that is currently stored in the model and darkent also has the same types as resnet in sequential or just differ in the last template param at places but they both are listed in the data type list so i am not sure what the problem. let me show you what i mean.

1. For resnet

ResNet18 resnet(3, 224, 224);
int res;
for (size_t idx = 0; idx < resnet.GetModel().Model().size(); ++idx)
    std::cout << idx << ": "<< abi::__cxa_demangle((resnet.GetModel().Model()[idx]).type().name(), 0, 0, &res) << std::endl;

output:

0: mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*
1: mlpack::ann::BatchNorm<arma::Mat<double>, arma::Mat<double> >*
2: mlpack::ann::BaseLayer<mlpack::ann::RectifierFunction, arma::Mat<double>, arma::Mat<double> >*
3: mlpack::ann::Padding<arma::Mat<double>, arma::Mat<double> >*
4: mlpack::ann::MaxPooling<arma::Mat<double>, arma::Mat<double> >*
5: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
6: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
7: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
8: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
9: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
10: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
11: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
12: mlpack::ann::AdaptiveMeanPooling<arma::Mat<double>, arma::Mat<double> >*
13: mlpack::ann::Linear<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*

2. For darknet, uses the same code just replaces with darknet 19

   
   0: mlpack::ann::BaseLayer<mlpack::ann::IdentityFunction, arma::Mat<double>, arma::Mat<double> >*
   1: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   2: mlpack::ann::AdaptiveMaxPooling<arma::Mat<double>, arma::Mat<double> >*
   3: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   4: mlpack::ann::AdaptiveMaxPooling<arma::Mat<double>, arma::Mat<double> >*
   5: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   6: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   7: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   8: mlpack::ann::AdaptiveMaxPooling<arma::Mat<double>, arma::Mat<double> >*
   9: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   10: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   11: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   12: mlpack::ann::AdaptiveMaxPooling<arma::Mat<double>, arma::Mat<double> >*
   13: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   14: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   15: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   16: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   17: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   18: mlpack::ann::AdaptiveMaxPooling<arma::Mat<double>, arma::Mat<double> >*
   19: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   20: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   21: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   22: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   23: boost::variant<mlpack::ann::Linear3D<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::LpPooling<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PixelShuffle<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Glimpse<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Highway<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::MultiheadAttention<arma::Mat<double>, arma::Mat<double>, mlpack::ann::NoRegularizer>*, mlpack::ann::Recurrent<arma::Mat<double>, arma::Mat<double>>*, mlpack::ann::RecurrentAttention<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ReinforceNormal<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Reparametrization<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Select<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, false>*, mlpack::ann::Sequential<arma::Mat<double>, arma::Mat<double>, true>*, mlpack::ann::Subview<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VRClassReward<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::VirtualBatchNorm<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::RBF<arma::Mat<double>, arma::Mat<double>, mlpack::ann::GaussianFunction>*, mlpack::ann::BaseLayer<mlpack::ann::GaussianFunction, arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::PositionalEncoding<arma::Mat<double>, arma::Mat<double> >*, mlpack::ann::ISRLU<arma::Mat<double>, arma::Mat<double> >*>
   24: mlpack::ann::Convolution<mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::FullConvolution>, mlpack::ann::NaiveConvolution<mlpack::ann::ValidConvolution>, arma::Mat<double>, arma::Mat<double> >*
   25: mlpack::ann::AdaptiveMeanPooling<arma::Mat<double>, arma::Mat<double> >*

[Aakash-kaushik]

I haven't test it yet, but what happens if you put:

https://github.com/mlpack/mlpack/blob/0d9de4b541de12413dcd77d184390a3a59b2cb59/src/mlpack/methods/ann/layer/layer_types.hpp#L237-L238

into LayerTypes instead of MoreTypes, you might have to move some layers from LayerTypes into MoreTypes, in this case.

btw i did try just shifting those two layer but seems like it requires more than just shifting those two, i was greeted with more errors on that side.

[Aakash-kaushik]

Sure, will be there.

I created a meet link because for some reason whenever i use zoom my wifi crashes.

[Aakash-kaushik]

I was able to transfer the weights but the values don't seem to match for now :(

[Aakash-kaushik]

I just checked and every layer has the same weights as the pytroch model so the only thing that comes to my mind right now is that the flow of weights might be wrong. is there a way we could trace that ?

[kartikdutt18]

Have you also matched running mean and variance for batchnorm layers (if they don't match), output also won't match.

[Aakash-kaushik]

Have you also matched running mean and variance for batchnorm layers (if they don't match), output also won't match.

I did and that matches too.

[kartikdutt18]

Can you compare layer wise output?

[Aakash-kaushik]

I basically added a cout statement in the forward function of conv2d, bn, and linear layer to print weights, bias, running mean and running variance.

[Aakash-kaushik]

Can you compare layer wise output?

That's a good idea, i am going to try that next.

[Aakash-kaushik]

Hey @kartikdutt18, @zoq does it make sense to download the pretrained weights and then have some tests for them in ctest ? doesn't seems a like a thing that will break because even if the archtiecture changes they are complete models downloaded from internet they are going to produce the same output.

[kartikdutt18]

Yes, the idea is to create a small dataset (refer voc dataset on mlpack.org), and download weights then load them into the model and predictions should match with pytorch.

[Aakash-kaushik]

Yes, the idea is to create a small dataset (refer voc dataset on mlpack.org), and download weights then load them into the model and predictions should match with pytorch.

But even for that i can just do that one time for each model and that should verify that. i don't see the benifit of putting that in the tests because tests are run everytime there is a change and if we just download the models and data from the internet which won't change and just carry out the prediction that will also not change.

[Aakash-kaushik]

Btw you were also talking about making the converter more robust and and better, i suggest that we use onnx support that helps us with.

1. We can use a lot more frameworks to get models, here is a list of them.
2. We would just have a huge library of models instantly and anyone who sees a benifit in mlpack instead of those libraries that person should be able to use mlpack without much hassle.
3. and all the other benifits of having to convert models from onnx to mlpack. For this we would only need to implement a converted from onnx to mlpack and if we want to have support for converting mlpack models to other frameworks we might have to write a mlpack to onnx converter.

PS: I wanted to propose this as a project but this was somewhat explored here and was added to the topics that can't be applied for this year but this isin't in a complete state.

[kartikdutt18]

Yes, the idea is to create a small dataset (refer voc dataset on mlpack.org), and download weights then load them into the model and predictions should match with pytorch.

But even for that i can just do that one time for each model and that should verify that. i don't see the benifit of putting that in the tests because tests are run everytime there is a change and if we just download the models and data from the internet which won't change and just carry out the prediction that will also not change.

Changes in mlpack might break something, tests will help in determining that (Like a function was renamed but wasn't updaged in models repo). Some serialization fix is made and so on. Secondly, Devs find it easier if everything is tested end to end so that they can use the test case as example. Third, user can see that if model is correct by running tests locally (issue during download removed). Just saying the weights are correct doesn't have much merit. Having tests that show the weights are correct require no further verification.

To reduce bandwidth usage, we can have builds that only run certain tests for certain file changes. Like for example, if a change is made in augmentation class then only augmentation and datasets are tested.

[Aakash-kaushik]

Yes, the idea is to create a small dataset (refer voc dataset on mlpack.org), and download weights then load them into the model and predictions should match with pytorch.

But even for that i can just do that one time for each model and that should verify that. i don't see the benifit of putting that in the tests because tests are run everytime there is a change and if we just download the models and data from the internet which won't change and just carry out the prediction that will also not change.

Changes in mlpack might break something, tests will help in determining that (Like a function was renamed but wasn't updaged in models repo). Some serialization fix is made and so on. Secondly, Devs find it easier if everything is tested end to end so that they can use the test case as example. Third, user can see that if model is correct by running tests locally (issue during download removed). Just saying the weights are correct doesn't have much merit. Having tests that show the weights are correct require no further verification.

To reduce bandwidth usage, we can have builds that only run certain tests for certain file changes. Like for example, if a change is made in augmentation class then only augmentation and datasets are tested.

Makes a lot of sense, so i can setup the predict function and things and as soon as we post the models on the domain i can see how to download them.

[kartikdutt18]

We already have download function in utils, you can use that and pass the path to the constructor.

[Aakash-kaushik]

We already have download function in utils, you can use that and pass the path to the constructor.

Thanks, will write tests tomorrow morning. good night btw.

[zoq]

About the ONNX idea, I really like that one, and I think the current converter tool goes into a similar direction but instead of supporting PyTorch we support ONNX. That could definitely an independent project.

[zoq]

We already have download function in utils, you can use that and pass the path to the constructor.

Thanks, will write tests tomorrow morning. good night btw.

I think we are going to support both things, we have the tests that checks that the conversion is correct with changes in the different codebases and we provide the converted model as well, so you could also directly download the mlpack model and use that for predictions.

[Aakash-kaushik]

About the ONNX idea, I really like that one, and I think the current converter tool goes into a similar direction but instead of supporting PyTorch we support ONNX. That could definitely an independent project.

Yup this was an interesting project but will take a lot of time and probably not something i can do in this period while also completing mobilnet. Btw for our conversion thing i found something that might help, take a look here: https://github.com/ysh329/deep-learning-model-convertor

Anyway i would love to work on the the onnx to mlpack converter if i get time.

[Aakash-kaushik]

We already have download function in utils, you can use that and pass the path to the constructor.

Thanks, will write tests tomorrow morning. good night btw.

I think we are going to support both things, we have the tests that checks that the conversion is correct with changes in the different codebases and we provide the converted model as well, so you could also directly download the mlpack model and use that for predictions.

Makes sense, i will write some tests tomorrow and then we can iterate on them.

[Aakash-kaushik]

The path set right now download models in the same dir inside weights/resnet we need to either have a way to download them in a folder in home dir which can be have a . before it's name so it's hidden because we can't download it inside the models repo until we know where the models repo is so i suggest a common path, i don't know how that will turn out in windows :/

what pytorch does is it takes the .cache directory in unix like systems to download the models and load them from there by default.

[Aakash-kaushik]

just checked for windows there is a folder names .cache in c://users/<your username> where pytorch saves our model and i don't know where mac does has this folder, but pytorch has a var named pytorch home which tells it the path on any os can we do something like that when we download the models ?