Closed Rababalkhalifa closed 4 years ago
Hello, I want to add special tokens to my bert model ? for example [NUMBER] or [URL] ? how can I do this?
FullTokenizer = bert.bert_tokenization.FullTokenizer bert_layer = hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1",trainable=False) vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy() do_lower_case = bert_layer.resolved_object.do_lower_case.numpy() tokenizer = FullTokenizer(vocab_file,do_lower_case)
I can see many unused in side vocab_file but how to utilize them?
There is a extra_tokens_vocab_size which allows for passing extra/custom tokens as negative token ids.
extra_tokens_vocab_size
how to add [CLS] and [SEP] at begin and end ?
Hello, I want to add special tokens to my bert model ? for example [NUMBER] or [URL] ? how can I do this?
https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1
FullTokenizer = bert.bert_tokenization.FullTokenizer bert_layer = hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1",trainable=False) vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy() do_lower_case = bert_layer.resolved_object.do_lower_case.numpy() tokenizer = FullTokenizer(vocab_file,do_lower_case)
I can see many unused in side vocab_file but how to utilize them?