Closed ptamas88 closed 3 years ago
@ptamas88 Did you succeed to make it works ? I have the same question
yes, usually the tokenizer is not part of the graph. For this you'll need a tokenizer that has a TF implementation, like sentencepiece when using albert. For BERT you might try the tf.text BertTokenizer (https://github.com/tensorflow/text/blob/master/docs/api_docs/python/text/BertTokenizer.md) - I haven't used it myself, but it should work.
hope that helps:
pip install tensorflow-text
and then try something along those lines:
import tensorflow_text as text
tokenizer = text.BertTokenizer(os.path.join(ckpt_dir, 'vocab.txt'))
tok_ids = tokenizer.tokenize(["hello, cruel world!", "abcccccccd"]).merge_dims(-2,-1).to_tensor(shape=(2, max_seq_len))
@ptamas88 Did you succeed to make it works ? I have the same question
haven't tried since, but i will check out the solution @kpe mentioned
it didn't work, still throw OperatorNotAllowedInGraphError
Hi, I would like to incorporate the tokenization process into a model which is using bert layer. Here is my custom layer:
And here is my code to test the custom layer within a dummy model:
I get the following traceback:
Can you lease help how to solve this issue? I think the problem is that the tokenizer gets tensors not string and that is why it can't tokenize it. But if that is the case how should I mkae this work? Thanks