Open cold-blue opened 5 months ago
Looks like Softplus in our opset and in torch behave differently, but the difference is below threshold 1e-4 which we usually use. @cold-blue do you have a model where this is important?
Looks like Softplus in our opset and in torch behave differently, but the difference is below threshold 1e-4 which we usually use. @cold-blue do you have a model where this is important?
I am trying to convert Mamba LLM into Openvino model and find this problem.
As you see, after softplus, dt is multiplied by A (large digits around 10^4). And the following updated ssm_state will be passed to the iteration, so that the error is cumulated. This model is important for LLM deployment on client, so please help me solve that. Thank you very much!
After further debugging, this error can be worked around by replacing softplus by "torch.log(torch.exp(x).unsqueeze(0) + 1).squeeze(0)". My data range is between (-29, 23) and the error may not caused by the numbers that I provided but other unknown ones.
@cold-blue So it would be better if the operation return 0. GPU seem to behave as expected, it returns 0 for softplus. I will assign CPU to look on this issue.
@mlukasze Could your team member help take a look whether this is something we can address?
yep, we will take a look
OpenVINO Version
2024.0.0-14509-34caeefd078-releases/2024/0
Operating System
Ubuntu 20.04 (LTS)
Device used for inference
CPU
Framework
PyTorch
Model used
N/A
Issue description
Got wrong outputs of a recurrent model after 10+ iterations from OpenVINO compared to torch due to cumulative errors. Reproduced the relative differences using the following short reproducer model. The problem might be related to the different implementation of OV Softplus from Torch Softplus. In my model, a large number is multiplied by the softplus and the result will be passed to the next iteration step, so the error is amplified and cumulated.
Step-by-step reproduction
Using the following script to reproduce:
Relevant log output
Issue submission checklist