search

GateNLP / gate-lf-pytorch-json

PyTorch wrapper for the LearningFramework GATE plugin
Apache License 2.0
1 stars 2 forks source link

Check why training using pre-trained embeddings or mapping does not work #20

Closed johann-petrak closed 6 years ago

johann-petrak commented 6 years ago

This could be due to the scaling of the embeddings. For the mapping, maybe we do not properly set the gradient needed for the parameters. With both it looks as if there is no proper backpropagation going on.

johann-petrak commented 6 years ago

This seems to be just the way it is, cannot find a flaw in the code. Using pretrained embeddings or mapping them (without changing the initial embeddings layer parameters) does not create results anywhere close to training new embeddings. Oddly, allowing the embeddings to get trained, starting with the initial pretrained ones produces better results but still not as good as the trained embeddings (however the trained in these experiments had different dimensionality which could have been the cause)

johann-petrak commented 6 years ago

Closing for now, need more experiments on this later!