mindspore-ai / mindspore

MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.
https://gitee.com/mindspore/mindspore
Apache License 2.0
4.31k stars 709 forks source link

The same input runs mindspore.ops.reciprocal on both CPU and GPU outputting one as INF and one as 0 #293

Open PhyllisJi opened 5 months ago

PhyllisJi commented 5 months ago
import mindspore
import numpy as np

mindspore.context.set_context(device_target='GPU')

inp = np.load('/mnt/test_inp.npz', allow_pickle=True)['inp']
print(inp)

ms_inp = mindspore.Tensor(inp.astype(np.float32))

mindspore_layer = mindspore.ops.reciprocal(ms_inp)

print(mindspore_layer)
import mindspore
import numpy as np

mindspore.context.set_context(device_target='CPU')

inp = np.load('/mnt/test_inp.npz', allow_pickle=True)['inp']

ms_inp = mindspore.Tensor(inp.astype(np.float32))

mindspore_layer = mindspore.ops.reciprocal(ms_inp)

print(mindspore_layer)

The specific input data is in our GIthub repository https://github.com/PhyllisJi/MoCoDiff_Bug/blob/ms-issue-%23293/README.md

git clone -b ms-issue#293 https://github.com/PhyllisJi/MoCoDiff_Bug.git python test_cpu.py python test_gpu.py

singularity6033 commented 2 months ago

Could you please tell us the version and commit id of the MindSpore used in this test case? I have tried master branch with commit id 2694987 (2.3.0), the result seems to be identical in both CPU and GPU. image

PhyllisJi commented 2 months ago

Could you please tell us the version and commit id of the MindSpore used in this test case? I have tried master branch with commit id 2694987 (2.3.0), the result seems to be identical in both CPU and GPU. image

Mindspore 2.2.14, I installed it from your official website.

singularity6033 commented 2 months ago

I just tried MindSpore 2.2.14 in our official website, the result of GPU is a maximum of FP32 but not 0, can't reproduce your result, I checked the code, logic is right when input of the reciprocal is 0. image image

PhyllisJi commented 2 weeks ago

I just tried MindSpore 2.2.14 in our official website, the result of GPU is a maximum of FP32 but not 0, can't reproduce your result, I checked the code, logic is right when input of the reciprocal is 0. image image

Presumably this is hardware related? I was using an Nvidia 3090.