search

cl-tohoku / bert-japanese

BERT models for Japanese text.
Apache License 2.0
509 stars 55 forks source link

Help on using the model for finetuning #13

Closed wailoktam closed 3 years ago

wailoktam commented 4 years ago

The part about tokenizing with Mecab is clear but what about the sub-word tokenization? And what if there are words found in the data used for finetuning but not found in the data used for pretraining? Some guide on using your pretrained model would be great.

singletongue commented 4 years ago

The subword tokenization is done with WordPiece algorithm, samely as in the original BERT models by Google. Unseen tokens will be represented as [UNK] tokens.

We have an example usage of a pretrained model with a tokenizer here (it doesn't contain much information, though): https://github.com/cl-tohoku/bert-japanese/blob/master/masked_lm_example.ipynb

wailoktam commented 4 years ago

Hi, I am looking at your example

Can I assume the tokenizer created with:

BertJapaneseTokenizer.from_pretrained('bert-base-japanese-whole-word-masking')

already include tokenization (word segementation) by Mecab so that I dont need to process the input by Mecab first?

How about sentence splitting? If my input is just a passage not broken down into sentences split by newline, I need to apply a sentence splitter before sending the input to tokenizer created with BertJapaneseTokenizer.from_pretrained('bert-base-japanese-whole-word-masking'), right?

singletongue commented 4 years ago

Yes, BertJapaneseTokenizer includes word segmentation so you don't need to split a text into words by yourself.

On the other hand, You will have to split a text into sentences by yourself. (This design decision also follows the original BERT models.)