From transformers>=4.34, PreTrainedTokenizer.__init__ requires self.vocab to be set.
Move the super(...).__init__ call to the end of JanomeSubwordsTokenizer.__init__ (and use unk_token instead of self.unk_token before init), following changes of BertTokenizer at that time.
Also move the call of self.add_tokens since it requires super(...).__init__ is done.
fix #335.
From transformers>=4.34,
PreTrainedTokenizer.__init__requiresself.vocabto be set. Move thesuper(...).__init__call to the end ofJanomeSubwordsTokenizer.__init__(and useunk_tokeninstead ofself.unk_tokenbefore init), following changes of BertTokenizer at that time.Also move the call of
self.add_tokenssince it requiressuper(...).__init__is done.