python encode_input.py \
"Alan Turing theorized that computers would one day become very powerful, but even he could not imagine" \
-n 100
I get the following output with tanh() (equal to PyTorch):
Output tokens:
703 484 561 307 973 13 198 198 1 40 836 470 892 314 1053 1683 1775 257 3644 326 714 466 1997 326 257 1692 852 714 466 553 339 531 13 198 198 1537 783 11 5176 284 262 670 286 257 1074 286 4837 379 262 2059 286 3442 11 14727 11 9061 389 852 973 284 466 1243 326 547 1752 1807 5340 13 198 198 464 1074 468 4166 257 3644 326 460 711 262 983 286 1514 11 257 3716 4811 983 326 9018 3867 5207 1088 257 3096 13 198 198 464 3644
Decoded output as text:
how they would be used.
"I don't think I've ever seen a computer that could do anything that a human being could do," he said.
But now, thanks to the work of a team of researchers at the University of California, Berkeley, computers are being used to do things that were once thought impossible.
The team has developed a computer that can play the game of Go, a complex strategy game that involves moving pieces around a board.
The computer
But with fast_tanh() I get the following output:
Output tokens:
703 484 561 307 973 13 198 198 1 40 836 470 892 314 1053 1683 1775 257 3644 326 714 466 1997 326 257 1692 852 714 466 553 339 531 13 198 198 1537 783 11 5176 284 262 670 286 257 1074 286 4837 379 262 2059 286 3442 11 14727 11 9061 389 852 973 284 466 1243 326 547 1752 1807 5340 13 198 198 464 1074 468 4166 257 3644 326 460 711 262 983 286 1514 11 257 3716 4811 983 326 9018 3867 257 3704 1088 257 3096 284 8006 517 7674
Decoded output as text:
how they would be used.
"I don't think I've ever seen a computer that could do anything that a human being could do," he said.
But now, thanks to the work of a team of researchers at the University of California, Berkeley, computers are being used to do things that were once thought impossible.
The team has developed a computer that can play the game of Go, a complex strategy game that involves moving a piece around a board to capture more territory
The last 9 tokens are different.
So the exact numerical shape of the tanh function makes a difference. At the very least from reproducibility perspective we have to maintain both versions. I don't know how to judge the quality, if the quality is the same, just slightly different probabilities that in the "greedy" mode give different results, but statistically equivalent.
Using the 1558M model and the following input:
I get the following output with
tanh()
(equal toPyTorch
):But with
fast_tanh()
I get the following output:The last 9 tokens are different.
So the exact numerical shape of the
tanh
function makes a difference. At the very least from reproducibility perspective we have to maintain both versions. I don't know how to judge the quality, if the quality is the same, just slightly different probabilities that in the "greedy" mode give different results, but statistically equivalent.