search

intel / torch-xpu-ops

Apache License 2.0
28 stars 20 forks source link

scatter_add needs xpu device check in aten operators #906

Open ZhiweiYan-96 opened 1 month ago

ZhiweiYan-96 commented 1 month ago

🚀 The feature, motivation and pitch

The issue is introduced in codegen pr https://github.com/intel/torch-xpu-ops/pull/310.

The FAILED UT throw errors like 

RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'

The affected UTs include

2024-09-11T10:19:05.8721701Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_add__xpu_complex64 FAILED [ 23%]
2024-09-11T10:19:05.8897299Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_add__xpu_float16 FAILED [ 25%]
2024-09-11T10:19:05.9071260Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_add__xpu_float32 FAILED [ 26%]
2024-09-11T10:19:05.9090020Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_add_mult_index_base_xpu_float32 FAILED [ 28%]
2024-09-11T10:19:06.7069522Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_bfloat16 FAILED [ 60%]
2024-09-11T10:19:06.7347844Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_float16 FAILED [ 61%]
2024-09-11T10:19:06.7621250Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_float32 FAILED [ 63%]
2024-09-11T10:19:06.7936430Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_float64 FAILED [ 64%]
2024-09-11T10:19:06.8625248Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_int16 FAILED [ 66%]
2024-09-11T10:19:06.8903041Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_int32 FAILED [ 67%]
2024-09-11T10:19:06.9178521Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_int64 FAILED [ 69%]
2024-09-11T10:19:06.9457722Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_int8 FAILED [ 70%]
2024-09-11T10:19:06.9735732Z test_scatter_gather_ops_xpu.py::TestScatterGatherXPU::test_scatter_reduce_mean_xpu_uint8 FAILED [ 71%]

test_torch_xpu.py::TestTorchDeviceTypeXPU::test_gather_backward_deterministic_path_xpu
test_torch_xpu.py::TestTorchDeviceTypeXPU::test_scatter_add_one_dim_deterministic_xpu

Alternatives

No response

Additional context

Detail failed logs

2024-09-11T10:19:09.5771275Z =================================== FAILURES ===================================
2024-09-11T10:19:09.5771711Z _____________ TestScatterGatherXPU.test_scatter_add__xpu_complex64 _____________
2024-09-11T10:19:09.5772085Z Traceback (most recent call last):
2024-09-11T10:19:09.5772925Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 197, in test_scatter_add_
2024-09-11T10:19:09.5773720Z     self._test_scatter_base(torch.Tensor.scatter_add_, device=device, dtype=dtype,
2024-09-11T10:19:09.5774679Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 110, in _test_scatter_base
2024-09-11T10:19:09.5775350Z     actual = fn(base.clone(), dim, idx, src)
2024-09-11T10:19:09.5776938Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5778235Z 
2024-09-11T10:19:09.5778398Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5778984Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_add__xpu_complex64
2024-09-11T10:19:09.5779399Z 
2024-09-11T10:19:09.5779604Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5783397Z ______________ TestScatterGatherXPU.test_scatter_add__xpu_float16 ______________
2024-09-11T10:19:09.5783840Z Traceback (most recent call last):
2024-09-11T10:19:09.5784698Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 197, in test_scatter_add_
2024-09-11T10:19:09.5785707Z     self._test_scatter_base(torch.Tensor.scatter_add_, device=device, dtype=dtype,
2024-09-11T10:19:09.5786704Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 110, in _test_scatter_base
2024-09-11T10:19:09.5787421Z     actual = fn(base.clone(), dim, idx, src)
2024-09-11T10:19:09.5789091Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5790424Z 
2024-09-11T10:19:09.5790595Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5791164Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_add__xpu_float16
2024-09-11T10:19:09.5791588Z 
2024-09-11T10:19:09.5792149Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5792618Z ______________ TestScatterGatherXPU.test_scatter_add__xpu_float32 ______________
2024-09-11T10:19:09.5793035Z Traceback (most recent call last):
2024-09-11T10:19:09.5793783Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 197, in test_scatter_add_
2024-09-11T10:19:09.5794543Z     self._test_scatter_base(torch.Tensor.scatter_add_, device=device, dtype=dtype,
2024-09-11T10:19:09.5795403Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 110, in _test_scatter_base
2024-09-11T10:19:09.5796064Z     actual = fn(base.clone(), dim, idx, src)
2024-09-11T10:19:09.5797630Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5798875Z 
2024-09-11T10:19:09.5799048Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5799603Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_add__xpu_float32
2024-09-11T10:19:09.5800016Z 
2024-09-11T10:19:09.5800211Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5800702Z ______ TestScatterGatherXPU.test_scatter_add_mult_index_base_xpu_float32 _______
2024-09-11T10:19:09.5801079Z Traceback (most recent call last):
2024-09-11T10:19:09.5801872Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 207, in test_scatter_add_mult_index_base
2024-09-11T10:19:09.5802684Z     res0 = torch.zeros(m, n, device=device, dtype=dtype).scatter_add_(0, idx, src)
2024-09-11T10:19:09.5804350Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5805603Z 
2024-09-11T10:19:09.5805772Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5806377Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_add_mult_index_base_xpu_float32
2024-09-11T10:19:09.5806864Z 
2024-09-11T10:19:09.5807057Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5807571Z __________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_bfloat16 __________
2024-09-11T10:19:09.5807938Z Traceback (most recent call last):
2024-09-11T10:19:09.5808700Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5809492Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5810382Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5811162Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5812858Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5814135Z 
2024-09-11T10:19:09.5814293Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5815039Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_bfloat16
2024-09-11T10:19:09.5815462Z 
2024-09-11T10:19:09.5815667Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5816143Z __________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_float16 ___________
2024-09-11T10:19:09.5816519Z Traceback (most recent call last):
2024-09-11T10:19:09.5817282Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5818077Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5818950Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5819731Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5821420Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5822686Z 
2024-09-11T10:19:09.5822855Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5823441Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_float16
2024-09-11T10:19:09.5823869Z 
2024-09-11T10:19:09.5824067Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5824550Z __________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_float32 ___________
2024-09-11T10:19:09.5824916Z Traceback (most recent call last):
2024-09-11T10:19:09.5825681Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5826475Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5827348Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5828196Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5829871Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5831138Z 
2024-09-11T10:19:09.5831324Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5831915Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_float32
2024-09-11T10:19:09.5832334Z 
2024-09-11T10:19:09.5832527Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5833008Z __________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_float64 ___________
2024-09-11T10:19:09.5833386Z Traceback (most recent call last):
2024-09-11T10:19:09.5834189Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5834981Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5835846Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5836623Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5838279Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5839530Z 
2024-09-11T10:19:09.5839690Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5840275Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_float64
2024-09-11T10:19:09.5840709Z 
2024-09-11T10:19:09.5840903Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5841380Z ___________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int16 ____________
2024-09-11T10:19:09.5841748Z Traceback (most recent call last):
2024-09-11T10:19:09.5842509Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5843301Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5844172Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5844937Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5846614Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5847870Z 
2024-09-11T10:19:09.5848059Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5848676Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int16
2024-09-11T10:19:09.5849094Z 
2024-09-11T10:19:09.5849287Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5849767Z ___________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int32 ____________
2024-09-11T10:19:09.5850144Z Traceback (most recent call last):
2024-09-11T10:19:09.5850926Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5851715Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5852583Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5853362Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5855111Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5856378Z 
2024-09-11T10:19:09.5856542Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5857130Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int32
2024-09-11T10:19:09.5857554Z 
2024-09-11T10:19:09.5857745Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5858222Z ___________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int64 ____________
2024-09-11T10:19:09.5858586Z Traceback (most recent call last):
2024-09-11T10:19:09.5859353Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5860148Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5861003Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5861780Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5863457Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5864721Z 
2024-09-11T10:19:09.5864883Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5865467Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int64
2024-09-11T10:19:09.5865882Z 
2024-09-11T10:19:09.5866075Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5866553Z ____________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int8 ____________
2024-09-11T10:19:09.5866929Z Traceback (most recent call last):
2024-09-11T10:19:09.5867674Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5868501Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5869400Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5870184Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5871880Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5873143Z 
2024-09-11T10:19:09.5873302Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5873881Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_int8
2024-09-11T10:19:09.5874304Z 
2024-09-11T10:19:09.5874497Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5875001Z ___________ TestScatterGatherXPU.test_scatter_reduce_mean_xpu_uint8 ____________
2024-09-11T10:19:09.5875373Z Traceback (most recent call last):
2024-09-11T10:19:09.5876131Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 237, in test_scatter_reduce_mean
2024-09-11T10:19:09.5876930Z     self._test_scatter_base(torch.Tensor.scatter_reduce_, device=device, dtype=dtype,
2024-09-11T10:19:09.5877785Z   File "/home/sdp/actions-runner-1/_work/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scatter_gather_ops.py", line 106, in _test_scatter_base
2024-09-11T10:19:09.5878563Z     actual = fn(base.clone(), dim, idx, src, reduce=reduction, include_self=include_self)
2024-09-11T10:19:09.5880241Z RuntimeError: scatter_add_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
2024-09-11T10:19:09.5881495Z 
2024-09-11T10:19:09.5881653Z To execute this test, run the following from the base repo dir:
2024-09-11T10:19:09.5882236Z     PYTORCH_TEST_WITH_SLOW=1 python test/test_scatter_gather_ops.py TestScatterGatherXPU.test_scatter_reduce_mean_xpu_uint8
2024-09-11T10:19:09.5882649Z 
2024-09-11T10:19:09.5882841Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2024-09-11T10:19:09.5883269Z =========================== short test summary info ============================
ZhiweiYan-96 commented 1 month ago

The failed UT is skipped in codegen PR currently