Closed adamsolomou closed 3 years ago
The brackets are required. This issue has been fixed internally and we are going to push an update to this codebase soon (In Jan). Since this public version acts as a mirror, we have not sync-ed for awhile now.
Thanks for clarifying!
So the correct way to go would be to tokenize according to this vocabulary
( ']', '1', '4', '7', '9', '8', '2', '5', '3', '0', '6', '[MIN', '[SM', '[MAX', '[MED'])
?
I suppose the fix would be to pass something like reserved_tokens=['[', ']']
to tfds.deprecated.text.Tokenizer
to prevent it from ignoring these?
If you address the issue this way it gives rise to the following vocabulary
('MED', '5', '8', '4', '0', ']', '1', '3', '6', '7', '2', 'MIN', '9', '[', 'MAX', 'SM')
which differs from this
( ']', '1', '4', '7', '9', '8', '2', '5', '3', '0', '6', '[MIN', '[SM', '[MAX', '[MED')
in the sense that [
is encoded as a token itself rather than being part of '[MIN', '[SM', '[MAX', '[MED'
.
@vanzytay could you please clarify which is the right way to go?
Hi,
According to a previous issue #6, the bracketed parentheses '[', ']' should be preserved during tokenization, resulting to the following set of unique tokens
( ']', '1', '4', '7', '9', '8', '2', '5', '3', '0', '6', '[MIN', '[SM', '[MAX', '[MED'])
.However, based on your code (
input_pipeline.py
), the following set of tokens is used to encode the input('SM', '2', 'MAX', '8', '5', '0', '4', '1', '7', '6', '3', 'MIN', '9', 'MED')
.Evidently, if you run
where x is an encoded input sample, the result does not contain the bracketed parentheses '[', ']. For instance, the following is the (truncated) outcome of decoding the first encoded sample in the training set
MIN 1 MED 8 0 1 3 0 6 1 7 MED SM 4 MED SM 2 9 MED 9 9 2 MIN 8 3 5 4 8 5 6 2 6 0 5 8 2 6 8 9 8 3 4 SM 3 SM MIN MIN MED 5 8 7 9 1 7 1 8 8 MAX 2 5 7 1 1 0 MAX 6 2 5 0 MIN 2 2 4 8 7 1 MIN 1 SM 3 6 4 0 5 5 MED 6 3 8 0 5 6 4 3 7 8 0 6 8 8 3 MAX 5 1 3 5 MED 3 SM 3 2 6 MAX 2 MIN 8 MAX 8 2 5 8 7 1 5 9 0 0 9 MAX 8 1 MAX 5 9 4 1 1 3 2 3 2 2 MIN 1 9 0 2 9 3 8 MAX ...
.So I was wondering why the bracketed parentheses '[', ']' are ignored? This way the nested structure is no longer preserved.