keras-team / keras-preprocessing

Utilities for working with image data, text data, and sequence data.
Other
1.02k stars 444 forks source link

Tokenizer ignores `filters` if `char_level` is `True` #301

Open paw-lu opened 4 years ago

paw-lu commented 4 years ago

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or on the Keras Slack channel instead of opening a GitHub issue.

Thank you!

What I expect

Tokenizer should not tokenize characters listed in filters.

What happens

If char_level=True, Tokenizer will tokenize all characters—even those listed in filters.

Code

❯ text = "ae"
❯ tokenizer = keras.preprocessing.text.Tokenizer(filters="e")  # Ignore "e"
❯ tokenizer.fit_on_texts(text)
❯ tokenizer.word_index
{'a': 1}  # Ignores "e" as expected
❯ tokenizer = keras.preprocessing.text.Tokenizer(char_level=True, filters="e")
❯ tokenizer.fit_on_texts(text)
❯ tokenizer.word_index
{'a': 1, 'e': 2}  # "e" is tokenized
paw-lu commented 4 years ago

A fix has been attempted in #302