Tokenization performed with validation data (HATT)

[mukherjee-d]

Thank you for your implementation! I have a question -in your code for the hierarchical attention model, you use both the training and validation data to form the tokenizer. Won't this bias your model?

[richliao]

I'm not sure if I understand your question but the sequence of embedding and embeddings are what the model is trying to learn. If a word is out of vocabulary, the embedding will be never trained anyway. You can even fix the embedding during the training given some of SOTA embedding training approach during the supervised learning.