Open Zessay opened 2 years ago
@AkshitaB this is just a friendly ping to make sure you haven't forgotten about this issue 😜
PretrainedTransformerIndexer
doesn't get to see the vocab in its __init__
method. So we can't add it there.
In principle, the way it should work is that AllenNLP discovers the vocab automatically. That's how it worked before standardized vocabularies became standard practice. But it takes a long time to discover a vocab on a big dataset, and most of the time it's not necessary anymore, so we put in this hack to short-cut vocabulary discovery.
You are right that because of this hack, you don't see the whole vocab in Model.__init__()
. Usually, we pass the name of the transformer model to Model.__init__()
, so it can figure out what it needs from there. But it is not that elegant.
If you have a better idea for how to do it, feel free to make a PR. I'd be happy to review it!
I met a problem yesterday when I want to get the
vocab_size
of the vocabulary namespacetransformer_tags
which I specify inPretrainedTransformerIndexer
. I found this namespace doesn't defined when I want to use it in theModel
init phrase. To figure out this problem, I read this part of source codes. I find the operation_add_encoding_to_vocabulary_if_needed
is called intokens_to_indices
which is called util the_train_epoch
starts.It means the
transformer_tags
namespace I defined in the vocabulary only can be used in theforward
process and after it. This causes a probelm that I can't define a linear layer that transforms the model output tovocab_size
logits. Of course, I can usefrom_pretrained_transformer
constructor to get a same namespace that I can use in the init phrase. If this is planned, I wonder what's the usage of_add_encoding_to_vocabulary_if_needed
in thePretrainedTransformerIndexer
. Why not call_add_encoding_to_vocabulary_if_needed
in the init method ofPretrainedTransformerIndexer
so that we can use the specified namespace from theModel
init phrase?Looking forward to your kind reply ~~ Thanks~~