quantiztion and non-idealities factor

[MySHworks]

Hello, I am a college student. I've been using the Memtorch you developed, recently. I have to say that this is a very good simulation library, which implements a lot of functions. It provides great convenience for the research of memristor - based network. However, I encountered some problems in the process of using it. I hope you can give me some advice. At First, The accuracy of the model didn't change when I added the non-idealities factor using apply_nonidealities.And the accuracy doesn't change much when you run the example you provided. May I ask what the reason is? In addition, when I was in use memtorch. Bh. Quantize.quantize function to quantify my model, it seems doesn't work.But my code is a one-to-one correspondence to the example.

[coreylammie]

Hi @MySHworks,

Thank you for reaching out! Could you please attach your simulation routines or minimal working examples so I can investigate further? The accuracy degradation observed will depend on what non-ideal characteristics you are introducing to your device model/peripheral circuitry, and the exact parameters in which you have used.

If you could provide a specific error message for memtorch.bh.Quantize.quantize and more information, it would be appreciated. In addition, could you please specify:

1. The version of MemTorch you are using (CPU/GPU, Operating System, Instillation Source);
2. The Python version that you are using.

These will enable me to better identify any specific issues that you may be encountering.

Kind Regards,

Corey.

[MySHworks]

Hello, I have used the Memtorch library you released to simulate forward reasoning in networks with memristors, recently. I have to say it's a very handy library. However, one problem I have found in my patched use is that if the network is built using torch.nn.Sequential container, patched does not seem to change the associated Conv2d layer. As a result, the precision of patched_model remains unchanged no matter how the relevant parameters of patched_model are adjusted. For complex models, the code will becomes complex if you do not build it using torch.nn.Sequential container. So I am wondering if you have any other suggestions? Thank you very much!

[coreylammie]

Hi @MySHworks,

Apologies for my late response! I have been focused on implementing support for comprehensive modeling of source and line resistance. I can confirm that your original issue was caused by using torch.nn.Sequential containers.

It appears that the patching procedures defined in memtorch.mn.Module.patch_model and memtorch.bh.nonideality.apply_nonidealities are currently incompatible with torch.nn.Sequential containers. I have created a separate issue (#87) detailing this. Thankfully it should be a fairly easy fix- I will try attend to this later today, and will update this issue when #87 is closed and the functionality is merged to the master branch.

[coreylammie]

Hi @MySHworks,

This fix has been merged to master in #88. I have tested it using torch.nn.Sequential containers with and without explicitly named layers (using OrderedDict), and with and without CUDA support.

This functionality was verified using the following code snippet:

import torch
from torch.autograd import Variable
import memtorch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import copy
from memtorch.mn.Module import patch_model
from memtorch.map.Input import naive_scale
from memtorch.map.Parameter import naive_map
from memtorch.bh.nonideality.NonIdeality import apply_nonidealities
from collections import OrderedDict

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.convs = nn.Sequential(nn.Conv2d(1, 20, 5, 1), nn.MaxPool2d(2, 2), nn.Conv2d(20, 50, 5, 1), nn.MaxPool2d(2, 2), nn.ReLU())
        self.fc1 = nn.Sequential(OrderedDict([('fc1', nn.Linear(4*4*50, 500))]))
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, 4*4*50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

device = torch.device('cpu' if 'cpu' in memtorch.__version__ else 'cuda')
model = Net().to(device)
reference_memristor = memtorch.bh.memristor.VTEAM
reference_memristor_params = {'time_series_resolution': 1e-10}
patched_model = patch_model(copy.deepcopy(model),
                          memristor_model=reference_memristor,
                          memristor_model_params=reference_memristor_params,
                          module_parameters_to_patch=[torch.nn.Conv2d, torch.nn.Linear],
                          mapping_routine=naive_map,
                          transistor=True,
                          programming_routine=None,
                          tile_shape=(128, 128),
                          max_input_voltage=0.3,
                          scaling_routine=naive_scale,
                          ADC_resolution=8,
                          ADC_overflow_rate=0.,
                          quant_method='linear')
patched_model.tune_()
patched_model_ = apply_nonidealities(copy.deepcopy(patched_model),
                                  non_idealities=[memtorch.bh.nonideality.NonIdeality.DeviceFaults],
                                  lrs_proportion=0.5,
                                  hrs_proportion=0.5,
                                  electroform_proportion=0)
patched_model_.tune_()

Could you please confirm that this works for your specific use-case? Thanks again for raising the issue!

[MySHworks]

Thank you for providing such a simple solution to this problem. With your help, I have successfully solved the previous problem. Thank you again for your help. However, I think the specific form of many networks is not the same, or to make some small modifications on this basis to be applicable. For example, in my model, each Convolution and LeakReLU module is placed in a torch.nn.Sequential container and then module_list, so if we convert according to the source code, we will lose the LeakyReLU module. I wonder if there's a better way to make it work for more different models. Looking forward to your reply. Thank you very much!

[coreylammie]

Hi @MySHworks,

No problem at all! Looking at the most recent PyTorch documentation, it appears that both torch.nn.ModuleList and torch.nn.ModuleDict can be used to store sub-modules and torch.nn.Sequential containers. It is unclear to what extent these can be nested, but it should be possible to iterate through all contains to flatten/extract all layers sequentially.

I have created a separate issue (#91) detailing this. I will update this thread once the issue has been fixed- my current intuition is to iterate through each element of each container and to check if any elements are containers recursively.

[MySHworks]

Hello, I looked at your quantization code before when I was debugging, but I didn't find the real quantization part memtorch_binding.quantize(). May I ask where the code of this part is? I have encountered some problems in model quantification recently, and I really want to learn it. Thank you very much!

[coreylammie]

Hi @MySHworks,

Methods defined in memtorch.bh.Quantize.py and memtorch_binding.quantize() call C++ functions defined in https://github.com/coreylammie/MemTorch/blob/master/memtorch/cpp/quantize.cpp using pybind11 bindings.

Previous versions of MemTorch have used quantization routines from the pytorch-playground, which should be functionally equivalent to the above C++ routines. pytorch-playground was previously integrated using a sub-module, and was called directly from memtorch.bh.Quantize.py in 27b1880aee.

If you would like, in the next release, I can add backwards compatibility, so that both C++/CUDA bindings and Python routines are accessible for memtorch.bh.Quantize.py. You are welcome to extend/modify this code as you see fit.

[MySHworks]

Hello, I would like to ask you some questions in the process of using memtorch.In the process of simulation of the finite conductivity state I found it didn't really quantify, the reason is that the "memtorch/bh/nonideality FiniteConductanceStates py" file try quantitative function of Max and min appeared error.Error: Expected Scalar type Float but found Double. I don't know what forms of Max and min should be passed in correctly. In addition, the two models seem to be the same after the problems occurred in the simulation of the finite conductance state, but the results obtained after tune() adjustment are completely different, the latter Tune shows a high score, but the accuracy is close to 0.If the latter is not tune() adjusted, the accuracy is the same as the former.I am very confused about this question and looking forward to your reply.

[MySHworks]

Hello, while using the Memtorch library, I found a problem with the quantization operation in the FiniteConductanceStates file. Because the data type in the crossbar[i].conductance_matrix tensor is float64, the error "Expected Scalar type Float but found Double" will always occur in the quantification function section. However, if the conductance_matrix data type is changed to FLOAT32, an error occurs: "Segmentation fault (core DUMPED)". Do you have a good solution to this problem? I will be glad to hear from you. Thank you very much!

[coreylammie]

Hi @MySHworks,

Apologies for the delayed response! I have looked into this further, and both issues should now be fixed in #110.

[24367452]

Hi, I am still facing the problem of changing the state of finite conductance but accuracy remains the same, how do I use quantification correctly, thank you for your answer! @coreylammie @MySHworks