Back-Propagation Concept

[49V]

Basic concept for how Back-Propagation will be implemented. Overview of input and output handling and basic design flow. You should also consider a basic timeline for getting everything finished.

[sdwpearson]

Ok here is the general plan: Inputs

• a matrix
• y (expected value vector)
• z matrix
• weights
• clk
• rst_n
• go (start back propagation)

Outputs

• bias gradient matrix
• weight gradient matrix
• ready (ready for back propagation)
• busy (back propagating)

General algorithm controlled by an FSM

1. Once go is asserted (provided that ready is as well) latch inputs and assert busy
2. Once busy is asserted inputs are ignored (so don't try to change them)
3. Go through from the Lth layer up to the beginning calculating the deltas and storing them in a matrix
4. As you go through each layer also calculate the gradient and store that in two matrices
5. After all of the gradients are calculated assert ready meaning that the outputs are valid

[sdwpearson]

As for timeline I'd like to get to get it coded but not verified by July 8-9 and then verify it the next week.

[49V]

Hey Stewart,

Everything looks great and I just had some comments below:

Ok here is the general plan: Inputs

a matrix y (expected value vector) z matrix weights clk rst_n go (start back propagation)

COMMENTS:

What exactly is your plan for the data structure of the a-matrix, the z-matrix, and the weights?

1. What data structure do you plan on using?

• Will it be a single structure or multiple structure for all layers? I need to know what you will be expecting from "fprop"

2. How will you implement this in VHDL?

• Multidimensional arrays, record, etc. Should have a concept before starting

Outputs

bias gradient matrix weight gradient matrix ready (ready for back propagation) busy (back propagating)

COMMENTS:

Please refer to "Inputs" section for comments with regards to data structure (bias gradient matrix and weight gradient Matrix)

General algorithm controlled by an FSM

Once go is asserted (provided that ready is as well) latch inputs and assert busy Once busy is asserted inputs are ignored (so don't try to change them) Go through from the Lth layer up to the beginning calculating the deltas and storing them in a matrix As you go through each layer also calculate the gradient and store that in two matrices After all of the gradients are calculated assert ready meaning that the outputs are valid

COMMENTS

After completing the basic algorithm please consider additional functionality.

1. Multiple sequential backpropagation instantiations (i.e. parallelizing)
2. Outer multiplication as per "FPGA Implementation of Neural Networks" - Omondi

[sdwpearson]

As for the data structures I'm going to put everything in one big matrix from fprop (so one single structure).