"could not execute a primitive" when using tanh inside of model class on multidimensional data

[gekeleda]

Describe the bug

When using the tanh activation function inside of a model class on 2d tensors, loss.backward results in a runtime error. For 1d tensors, or torch.tanh outside the model class, there are no problems. Also, other activation functions work, as well as sigmoid or tanh reproduced using torch.sigmoid.

I have provided a minimum example below illustrating the behaviour, hopefully you can reproduce it. You can also find the full traceback there.

The normally expected output of below code should be:

relu result: 0.016979079693555832
custom tanh result: 0.22109611332416534
tanh 1d result: 1.564261555671692
tanh result: (some number)

The output that I get is:

relu result: 0.016979079693555832
custom tanh result: 0.22109611332416534
tanh 1d result: 1.564261555671692
Traceback (most recent call last):
  File "/home/user/minimal_test.py", line 70, in <module>
    loss_val.backward() # crashes here <------------------------
  File "/home/user/mambaforge/envs/rl/lib/python3.8/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/home/user/mambaforge/envs/rl/lib/python3.8/site-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: could not execute a primitive

import torch
import intel_extension_for_pytorch as ipex

device = torch.device("xpu:0")

from torch import nn

class F(nn.Module):
    def __init__(self, use_ipex=False, comp_device=torch.device("cpu")):
        super(F, self).__init__()
        self.fc1 = nn.Linear(1, 200)
        self.fc2 = nn.Linear(200, 1)

    def forward(self, x, act):
        with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
            x = self.fc1(x)
            x = act(x)
            x = self.fc2(x)
        return x

net = F()
net = net.to(device)

x = torch.randn((1,1), device=device)
target = torch.randn((1,1), device=device)

optimizer = torch.optim.Adam(net.parameters(), lr=0.01)

# optimizing makes no difference
# net, optimizer = ipex.optimize(net, optimizer=optimizer, dtype=torch.bfloat16)

loss = nn.MSELoss()
# loss = loss.to(device) # makes no difference

############################################################

with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
    y = torch.tanh(net.forward(x, torch.relu))
    loss_val = loss(y, target)
loss_val.backward()

print("relu result:", loss_val.item())

############################################################

mytanh = lambda x: 2 * torch.sigmoid(2 * x) - 1
with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
    y = torch.tanh(net.forward(x, mytanh))
    loss_val = loss(y, target)
loss_val.backward()

print("custom tanh result:", loss_val.item())

############################################################

x_1d = torch.randn(1, device=device)
target_1d = torch.randn(1, device=device)
with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
    y = torch.tanh(net.forward(x_1d, torch.tanh))
    loss_val = loss(y, target_1d)
loss_val.backward()

print("tanh 1d result:", loss_val.item())

############################################################

with torch.xpu.amp.autocast(enabled=True, dtype=torch.bfloat16):
    y = torch.tanh(net.forward(x, torch.tanh))
    loss_val = loss(y, target)
loss_val.backward() # crashes here <------------------------

print("tanh result:", loss_val.item())

Versions

Collecting environment information... PyTorch version: 2.0.1a0+cxx11.abi PyTorch CXX11 ABI: Yes IPEX version: 2.0.110+xpu IPEX commit: ba7f6c127 Build type: Release

OS: Ubuntu 22.04.3 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: N/A IGC version: 2023.2.0 (2023.2.0.20230721) CMake version: N/A Libc version: glibc-2.35

Python version: 3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:06:00) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.3.13-060313-generic-x86_64-with-glibc2.10 Is XPU available: True DPCPP runtime version: 2023.2.1 MKL version: 2023.2.0 GPU models and configuration: [0] _DeviceProperties(name='Intel(R) Arc(TM) A770 Graphics', platform_name='Intel(R) Level-Zero', dev_type='gpu, support_fp64=0, total_memory=15473MB, max_compute_units=512, gpu_eu_count=512) Intel OpenCL ICD version: 23.17.26241.33-647~22.04 Level Zero version: 1.3.26241.33-647~22.04

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: GenuineIntel Model name: 12th Gen Intel(R) Core(TM) i5-12600K CPU family: 6 Model: 151 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 1 Stepping: 2 CPU max MHz: 4900,0000 CPU min MHz: 800,0000 BogoMIPS: 7372.80 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 416 KiB (10 instances) L1i cache: 448 KiB (10 instances) L2 cache: 9,5 MiB (7 instances) L3 cache: 20 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-15 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] intel-extension-for-pytorch==2.0.110+xpu [pip3] numpy==1.24.4 [pip3] torch==2.0.1a0+cxx11.abi [pip3] torchaudio==2.0.2 [pip3] torchvision==0.15.2a0+cxx11.abi [conda] blas 2.116 mkl conda-forge [conda] blas-devel 3.9.0 16_linux64_mkl conda-forge [conda] cpuonly 2.0 0 pytorch [conda] ffmpeg 4.3 hf484d3e_0 pytorch [conda] intel-extension-for-pytorch 2.0.110+xpu pypi_0 pypi [conda] libblas 3.9.0 16_linux64_mkl conda-forge [conda] libcblas 3.9.0 16_linux64_mkl conda-forge [conda] liblapack 3.9.0 16_linux64_mkl conda-forge [conda] liblapacke 3.9.0 16_linux64_mkl conda-forge [conda] mkl 2022.1.0 h84fe81f_915 conda-forge [conda] mkl-devel 2022.1.0 ha770c72_916 conda-forge [conda] mkl-include 2022.1.0 h84fe81f_915 conda-forge [conda] numpy 1.24.4 py38h59b608b_0 conda-forge [conda] pytorch-mutex 1.0 cpu pytorch [conda] torch 2.0.1a0+cxx11.abi pypi_0 pypi [conda] torchaudio 2.0.2 py38_cpu pytorch [conda] torchvision 0.15.2a0+cxx11.abi pypi_0 pypi

[jingxu10]

@gujinghui @tye1 seems to be an issue related to oneDNN.

[ZhaoqiongZ]

Hi @gekeleda may I know your version of torch and ipex? Hi @gekeleda @jingxu10 I tried this with recently ipex master branch and oneAPI 2023.2.0 and it works, the output is relu result: 0.0016836061840876937 custom tanh result: 0.0422641821205616 tanh 1d result: 0.013959823176264763 tanh result: 0.04388555511832237

[gekeleda]

Hello @ZhaoqiongZ , thanks for the feedback.

My torch version is 2.0.1a0+cxx11.abi, and my ipex version is 2.0.110+xpu, i.e the latest release. All the other versions should also be visible in my first comment.

I just reinstalled everything, but nothing changed. Maybe it is a problem of my configuration, however everything except the tanh activation works perfectly fine, so that would seem unlikely I suppose?

If there is anything else I can do to debug it further, please let me know. Otherwise, I can wait to see if the bug persists in the next release, as it is not a big issue for me.

[ZhaoqiongZ]

Hi @gekeleda , thanks a lot for your willing to help! Could you please run the collect env script so that I can know the detailed version for difference library.

wget https://raw.githubusercontent.com/intel/intel-extension-for-pytorch/master/scripts/collect_env.py

python collect_env.py

[gekeleda]

@ZhaoqiongZ

Here is the output of collect_env.py:

PyTorch version: 2.0.1a0+cxx11.abi
PyTorch CXX11 ABI: Yes
IPEX version: 2.0.110+xpu
IPEX commit: ba7f6c127
Build type: Release

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: N/A
IGC version: 2023.2.0 (2023.2.0.20230721)
CMake version: N/A
Libc version: glibc-2.35

Python version: 3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:06:00)  [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.3.13-060313-generic-x86_64-with-glibc2.10
Is XPU available: True
DPCPP runtime version: 2023.2.1
MKL version: 2023.2.0
GPU models and configuration: 
[0] _DeviceProperties(name='Intel(R) Arc(TM) A770 Graphics', platform_name='Intel(R) Level-Zero', dev_type='gpu, support_fp64=0, total_memory=15473MB, max_compute_units=512, gpu_eu_count=512)
Intel OpenCL ICD version: 23.17.26241.33-647~22.04
Level Zero version: 1.3.26241.33-647~22.04

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          16
On-line CPU(s) list:             0-15
Vendor ID:                       GenuineIntel
Model name:                      12th Gen Intel(R) Core(TM) i5-12600K
CPU family:                      6
Model:                           151
Thread(s) per core:              2
Core(s) per socket:              10
Socket(s):                       1
Stepping:                        2
CPU max MHz:                     4900,0000
CPU min MHz:                     800,0000
BogoMIPS:                        7372.80
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       416 KiB (10 instances)
L1i cache:                       448 KiB (10 instances)
L2 cache:                        9,5 MiB (7 instances)
L3 cache:                        20 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.0.110+xpu
[pip3] numpy==1.24.4
[pip3] torch==2.0.1a0+cxx11.abi
[pip3] torchaudio==2.0.2
[pip3] torchvision==0.15.2a0+cxx11.abi
[conda] blas                      2.116                       mkl    conda-forge
[conda] blas-devel                3.9.0            16_linux64_mkl    conda-forge
[conda] cpuonly                   2.0                           0    pytorch
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] intel-extension-for-pytorch 2.0.110+xpu              pypi_0    pypi
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] mkl                       2022.1.0           h84fe81f_915    conda-forge
[conda] mkl-devel                 2022.1.0           ha770c72_916    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.24.4           py38h59b608b_0    conda-forge
[conda] pytorch-mutex             1.0                         cpu    pytorch
[conda] torch                     2.0.1a0+cxx11.abi          pypi_0    pypi
[conda] torchaudio                2.0.2                  py38_cpu    pytorch
[conda] torchvision               0.15.2a0+cxx11.abi          pypi_0    pypi```

[ZhaoqiongZ]

Hi @gekeleda , I found that our DPC++ library version is different, where do you install the oneAPI? do you install with the following line ?

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397_offline.sh

sudo sh ./l_BaseKit_p_2023.2.0.49397_offline.sh

Also we have a different driver version, I will test with your driver verison.

PyTorch version: 2.0.1a0+cxx11.abi
PyTorch CXX11 ABI: Yes
IPEX version: 2.0.110+xpu
IPEX commit: ba7f6c127
Build type: Release

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: N/A
IGC version: 2023.2.0 (2023.2.0.20230622)
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 20 2023, 00:27:19) [GCC 13.1.0] (64-bit runtime)
Python platform: Linux-6.2.0-26-generic-x86_64-with-glibc2.35
Is XPU available: True
DPCPP runtime version: 2023.2.0
MKL version: 2023.2.0
GPU models and configuration:
[0] _DeviceProperties(name='Intel(R) Graphics [0x56a0]', platform_name='Intel(R) Level-Zero', dev_type='gpu, support_fp64=0, total_memory=15473MB, max_compute_units=512, gpu_eu_count=512)
Intel OpenCL ICD version: 1:23.31.026957.14424-main
Level Zero version: 1.3.026957.14424-main

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          20
On-line CPU(s) list:             0-19
Vendor ID:                       GenuineIntel
Model name:                      12th Gen Intel(R) Core(TM) i7-12700
CPU family:                      6
Model:                           151
Thread(s) per core:              2
Core(s) per socket:              12
Socket(s):                       1
Stepping:                        2
CPU max MHz:                     4900.0000
CPU min MHz:                     800.0000
BogoMIPS:                        4224.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       512 KiB (12 instances)
L1i cache:                       512 KiB (12 instances)
L2 cache:                        12 MiB (9 instances)
L3 cache:                        25 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-19
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.0.110+xpu
[pip3] numpy==1.25.2
[pip3] torch==2.0.1a0+cxx11.abi
[pip3] torchvision==0.15.2a0+cxx11.abi
[conda] intel-extension-for-pytorch 2.0.110+xpu              pypi_0    pypi
[conda] numpy                     1.25.2                   pypi_0    pypi
[conda] torch                     2.0.1a0+cxx11.abi          pypi_0    pypi
[conda] torchvision               0.15.2a0+cxx11.abi          pypi_0    pypi

[gekeleda]

Hi @ZhaoqiongZ ,

yes I installed it using these lines. I am not sure why it say 2023.2.1, I definitely have 2023.2.0 installed (as no other directory exists in /opt/intel/oneapi/compiler)

About the driver version, as far as I am aware it matches the one specified in the installation instructions.

[ZhaoqiongZ]

Hi @gekeleda ， I also verify the code on a machine with 647 driver installed but still not able to reproduce the issue, will you try to install oneAPI with the following commands in a docker and to verify the compiler and mkl versions?

wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397_offline.sh

sudo sh ./l_BaseKit_p_2023.2.0.49397_offline.sh