Closed Jeffresh closed 3 years ago
This happens in all the following code that use feature.text
module and is solved using the deprecated
statement. I try to find where this modules are moved, but I can't find it. I find that now tensorflow has tensorflow_text
module, a dedicated module for preprocessing text, that you have to install separately, like dataset module, but the logic , method's names and use of them are changed a lil bit, so don't now the cost to migrate the code to use that.
Thanks for the note! Just updated it so that it should work with newer tensorflow_dataset versions via
try:
tokenizer = tfds.features.text.Tokenizer()
except AttributeError:
tokenizer = tfds.deprecated.text.Tokenizer()
In page 587: you use :
tfds.feature.text.Tokenizer()
but is deprecated.tokenizer = tfds.deprecated.text.Tokenizer()
instead.Dont know which is the new implementation.