Closed jerryli27 closed 2 weeks ago
Interesting, my guess is that it's most likely numerical issues around convolutions that take place in cuda/cudnn. I don't get the exact same error values as in your logs (probably because of cudnn benchmarking or some other non-determinism) but when disabling cudnn with the following line your seanet decode test seems to pass.
torch.backends.cudnn.enabled = False
Alternatively you can try running the test on cpu and it seems to work properly there too.
That worked. Thank you for the quick response! I didn't realize cudnn can make a big difference.
Btw if you'd like to use the unit test, just let me know and I can submit a pull request.
Btw if you'd like to use the unit test, just let me know and I can submit a pull request.
Sure, your test commit looks pretty good so happy to include them.
Due diligence
Topic
The PyTorch implementation
Question
Hi all,
First of all, thank you for your great work and for open sourcing the Mimi Tokenizer. It's very rare to have a causal and streamable tokenizer, with the exception of Encodec.
When I tried to use the Tokenizer, it seems that some of its components, namely the SEANetDecoder is not causal.
I went ahead and coded up a few unit tests to convince myself and to further nail down the culprit. See this commit in my fork. The following classes are causal and streamable:
StreamingConv1d
,StreamingConvTranspose1d
, andSEANetResnetBlock
. But the unit test breaks forSEANetDecoder
. To put it simply, if I feed it input [1,2,3], the output part corresponding to [1,2] is different from when I give it just [1,2] as its input.I find it mildly concerning. Is there a bug, or is it just numerical instability?
Below are my unit test outputs: