Is there a way to look at the output tokens before they are detokenised? Right now I am using model.predict() (as shown in the t5-trivia example) to generate the output for a sequence to sequence model, but this saves the detokenized output to a file.
I have tried looking at the result in the decode method, but it also returns output which is already detokenized. I want to see the output token ids before sentencepiece detokenization. How can I do this?
Hi there,
Is there a way to look at the output tokens before they are detokenised? Right now I am using
model.predict()(as shown in the t5-trivia example) to generate the output for a sequence to sequence model, but this saves the detokenized output to a file.I have tried looking at the
resultin the decode method, but it also returns output which is already detokenized. I want to see the output token ids before sentencepiece detokenization. How can I do this?Thanks