search

amassivek / signalpropagation

Forward Pass Learning and Inference Library, for neural networks and general intelligence, Signal Propagation (sigprop)
https://amassivek.github.io/sigprop
BSD 3-Clause "New" or "Revised" License
45 stars 6 forks source link

What's the difference between example2 and example4? #4

Closed erjiaxiao closed 1 year ago

erjiaxiao commented 1 year ago

Hi, Amassivek, Thanks for your amazing work!

After looking through the repository, I wonder what is the difference between example 2 and example 4.

Example 2: Input Target Max Rand This example feeds the inputs x (e.g. images) and their respective targets t (e.g. labels) as pairs (or one after the other). Given pair x_i,t_i, this example selects the closest matching pair x_j,t_j to compare with. If there are multiple equivalent matching pairs, it randomly selects one.

Example 4: Input Target Top-K This example feeds the inputs x (e.g. images) and their respective targets t (e.g. labels) as pairs (or one after the other). Given pair x_i,t_i, this example selects the top k closest matching pair x_j,t_j to compare with.

Do they have different loss functions for every layer? And what does it mean that " selects the top k closest matching pair x_j,t_j to compare with"? From my understanding every layer updates its weights will just compare the L2 distance between h_i and t_i. When does it use multiple pairs?

amassivek commented 1 year ago

The loss functions are different. The loss functions in this code are generalized versions of the loss functions found in the paper. They are made to support single and multi-target tasks.

Top-k refers to selecting the top k matches. Take a look at the topk method in pytorch.

erjiaxiao commented 1 year ago

@amassivek Thanks! I'll check it out.