Closed sqPoseidon closed 1 year ago
This issue was caused by left shift overflow, fixed by cornell-zhang/hcl-dialect-prototype@e026ca3fba46810108c61490c0945e5e8890d7a9
The test case is added to regression tests here:https://github.com/cornell-zhang/heterocl/blob/afe4471ed80455804faf96d793fb367f798344a6/tests/test_dtype.py#L633-L670
I'm using the
hcl-mlir
branch and want to convert the arbitrary integer data to fixed point data. In some test cases, I got the wrong results which are not due to the precision loss.For example,
Please help check the problem.