search

sandialabs / cross-sim

CrossSim: accuracy simulation of analog in-memory computing
Other
117 stars 26 forks source link

What is this baseline_mat in training_net.py? #10

Closed TejaswanthMaram closed 8 months ago

TejaswanthMaram commented 1 year ago

image

This snippet is from training_net.py So, What is this baseline_mat?

How it is calibrated to these values and how do I do it if I want to add my own dataset?

ptxiao commented 1 year ago

For training only, baseline_mat sets the limits on the weights. The length of the vector is equal to the # layers that have a weight matrix. During training, any conductance update that pushes the weight value in a given layer beyond the range [-baseline_mat, +baseline_mat] will be clipped to the limits. This is needed to assign a device conductance range to a weight range. In that training code block, each set of baseline_mat parameters was tuned not just for the dataset but for a specific MLP topology that is set in the MLP_training file.

One way to set this is to first train the network with ideal numeric settings, with baseline_mat set very high. Then look at the numerical range of weight values in the trained matrices. Then set baseline_mat to between 1-2X the maximum absolute weight value in each layer when running training with lookup tables.