From transformers>=4.34, PreTrainedTokenizer.__init__ requires self.vocab to be set.
Move the super(...).__init__ call to the end of JanomeSubwordsTokenizer.__init__ (and use unk_token instead of self.unk_token before init), following changes of BertTokenizer at that time.
Also move the call of self.add_tokens since it requires super(...).__init__ is done.
fix #335.
From transformers>=4.34,
PreTrainedTokenizer.__init__
requiresself.vocab
to be set. Move thesuper(...).__init__
call to the end ofJanomeSubwordsTokenizer.__init__
(and useunk_token
instead ofself.unk_token
before init), following changes of BertTokenizer at that time.Also move the call of
self.add_tokens
since it requiressuper(...).__init__
is done.