jdcpni
commented
4 years ago Good catch. Sounds like a good thing to put on the todo list for you and/or Dillon.
jdc
Sent from my iPhone
On Jan 29, 2020, at 3:57 PM, SamKG notifications@github.com wrote:
When layer biases are initialized within PsyNeuLink during the creation of a Pytorch model in they are set to be zero (see: pytorchmodelcreator.py:84)
However, the standard Pytorch behaviour is to randomly initialize the biases. (Ex: import torch.nn as nn nn.Linear(1,1) Is equivalent to a single neuron with linear activation, but has random weights.
Current test cases don't catch this, as they avoid using nn.Linear and related Pytorch layer creation methods altogether (instead opting to directly call lowlevel torch tensor operations).
We should look into ways to correct this inconsistency by allowing manual biases, and add more robust test cases that account for this.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/PrincetonUniversity/PsyNeuLink/issues/1461?email_source=notifications&email_token=ACSHKGF4M4FBPVVJS3L7AWDRAHUSLA5CNFSM4KNLLJUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IJU2ZWQ, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACSHKGDTPEIL4LWKGM6NJ3TRAHUSLANCNFSM4KNLLJUA.
When layer biases are initialized within PsyNeuLink during the creation of a Pytorch model they are set to be zero (see:
pytorchmodelcreator.py:84)However, the standard Pytorch behaviour is to randomly initialize the biases. Example:
nn.Linear(1,1)Is equivalent to a single neuron with linear activation, but has random biases.Current test cases don't catch this, as they avoid using
nn.Linearand related Pytorch layer creation methods altogether (instead opting to directly call lowlevel torch tensor operations).We should look into ways to correct this inconsistency by allowing biases to be copied from PNL, and add more robust test cases that account for this.