Closed johann-petrak closed 6 years ago
This seems to be just the way it is, cannot find a flaw in the code. Using pretrained embeddings or mapping them (without changing the initial embeddings layer parameters) does not create results anywhere close to training new embeddings. Oddly, allowing the embeddings to get trained, starting with the initial pretrained ones produces better results but still not as good as the trained embeddings (however the trained in these experiments had different dimensionality which could have been the cause)
Closing for now, need more experiments on this later!
This could be due to the scaling of the embeddings. For the mapping, maybe we do not properly set the gradient needed for the parameters. With both it looks as if there is no proper backpropagation going on.