issue about question encoding

[sunqiang85]

    def _encode_question(self, question):
        """ Turn a question into a vector of indices and a question length """
        vec = torch.zeros(self.max_question_length).long().fill_(self.num_tokens)
        for i, token in enumerate(question):
            if i >= self.max_question_length:
                break
            index = self.token_to_index.get(token, self.num_tokens - 1)
            vec[i] = index
        return vec, min(len(question), self.max_question_length)

why fill the question vector with 'num_tokens'; since the index 0 is preserved in question vocabulary. and 'self.num_tokens-1' is used as index of '0' in question token vocabulary "vocab.json"

[KaihuaTang]

sorry i forgot the detail. the difference of index 0 and index num_tokens might be "unknown token" and "empty token".