Closed fatguy96 closed 3 years ago
((self.train_x, self.train_y), (self.test_x, self.test_y)) = map(self._prepare, [train, test]) print("max seq_len", self.max_seq_len) self.max_seq_len = min(self.max_seq_len, max_seq_len) ((self.train_x, self.train_x_token_types), (self.test_x, self.test_x_token_types)) = map(self._pad, [self.train_x, self.test_x])
this demo first call _prepare, then call _pad, but when cut off, the "[SEP]" token will be removed?
this demo first call _prepare, then call _pad, but when cut off, the "[SEP]" token will be removed?