daisyden
commented
1 week ago This is a random issue caused by the timing of autograd reentrant feature, @guangyey is investigating it.
Open PenghuiCheng opened 1 week ago
daisyden
commented
1 week ago This is a random issue caused by the timing of autograd reentrant feature, @guangyey is investigating it.
🐛 Describe the bug
Sometimes，There is an error 'AssertionError: "Simulate error" does not match "grad can be implicitly created only for scalar outputs"' in case: test_autograd_xpu.py::TestAutogradDeviceTypeXPU::test_reentrant_parent_error_on_cpu_xpu. reproduce cmd:
Versions
Collecting environment information... 2024-11-11T15:11:49.2844689Z PyTorch version: 2.6.0a0+gitceb44b2 2024-11-11T15:11:49.2845004Z Is debug build: False 2024-11-11T15:11:49.2845270Z CUDA used to build PyTorch: None 2024-11-11T15:11:49.2845562Z ROCM used to build PyTorch: N/A 2024-11-11T15:11:49.2845735Z 2024-11-11T15:11:49.2845848Z OS: Ubuntu 22.04.2 LTS (x86_64) 2024-11-11T15:11:49.2846380Z GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 2024-11-11T15:11:49.2846718Z Clang version: Could not collect 2024-11-11T15:11:49.2847007Z CMake version: version 3.30.5 2024-11-11T15:11:49.2847278Z Libc version: glibc-2.35 2024-11-11T15:11:49.2847440Z 2024-11-11T15:11:49.2847859Z Python version: 3.10.15 | packaged by conda-forge | (main, Oct 16 2024, 01:24:24) [GCC 13.3.0] (64-bit runtime) 2024-11-11T15:11:49.2848449Z Python platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.35 2024-11-11T15:11:49.2848793Z Is CUDA available: False 2024-11-11T15:11:49.2849014Z CUDA runtime version: No CUDA 2024-11-11T15:11:49.2849264Z CUDA_MODULE_LOADING set to: N/A 2024-11-11T15:11:49.2849533Z GPU models and configuration: No CUDA 2024-11-11T15:11:49.2850018Z Nvidia driver version: No CUDA 2024-11-11T15:11:49.2850256Z cuDNN version: No CUDA 2024-11-11T15:11:49.2850466Z HIP runtime version: N/A 2024-11-11T15:11:49.2850680Z MIOpen runtime version: N/A 2024-11-11T15:11:49.2850916Z Is XNNPACK available: True 2024-11-11T15:11:49.2851154Z 2024-11-11T15:11:49.2851224Z CPU: 2024-11-11T15:11:49.2851392Z Architecture: x86_64 2024-11-11T15:11:49.2851705Z CPU op-mode(s): 32-bit, 64-bit 2024-11-11T15:11:49.2852046Z Address sizes: 46 bits physical, 57 bits virtual 2024-11-11T15:11:49.2852392Z Byte Order: Little Endian 2024-11-11T15:11:49.2852671Z CPU(s): 192 2024-11-11T15:11:49.2852965Z On-line CPU(s) list: 0-191 2024-11-11T15:11:49.2853240Z Vendor ID: GenuineIntel 2024-11-11T15:11:49.2853584Z Model name: Intel(R) Xeon(R) Platinum 8468V 2024-11-11T15:11:49.2853923Z CPU family: 6 2024-11-11T15:11:49.2854157Z Model: 143 2024-11-11T15:11:49.2854491Z Thread(s) per core: 2 2024-11-11T15:11:49.2854741Z Core(s) per socket: 48 2024-11-11T15:11:49.2855026Z Socket(s): 2 2024-11-11T15:11:49.2855250Z Stepping: 8 2024-11-11T15:11:49.2855497Z Frequency boost: enabled 2024-11-11T15:11:49.2855771Z CPU max MHz: 2401.0000 2024-11-11T15:11:49.2856038Z CPU min MHz: 800.0000 2024-11-11T15:11:49.2856315Z BogoMIPS: 4800.00 2024-11-11T15:11:49.2860012Z Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities 2024-11-11T15:11:49.2863840Z Virtualization: VT-x 2024-11-11T15:11:49.2864152Z L1d cache: 4.5 MiB (96 instances) 2024-11-11T15:11:49.2864484Z L1i cache: 3 MiB (96 instances) 2024-11-11T15:11:49.2864833Z L2 cache: 192 MiB (96 instances) 2024-11-11T15:11:49.2865166Z L3 cache: 195 MiB (2 instances) 2024-11-11T15:11:49.2865448Z NUMA node(s): 2 2024-11-11T15:11:49.2865754Z NUMA node0 CPU(s): 0-47,96-143 2024-11-11T15:11:49.2866080Z NUMA node1 CPU(s): 48-95,144-191 2024-11-11T15:11:49.2866376Z Vulnerability Itlb multihit: Not affected 2024-11-11T15:11:49.2866679Z Vulnerability L1tf: Not affected 2024-11-11T15:11:49.2866974Z Vulnerability Mds: Not affected 2024-11-11T15:11:49.2867251Z Vulnerability Meltdown: Not affected 2024-11-11T15:11:49.2867549Z Vulnerability Mmio stale data: Not affected 2024-11-11T15:11:49.2867835Z Vulnerability Retbleed: Not affected 2024-11-11T15:11:49.2868301Z Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp 2024-11-11T15:11:49.2868953Z Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization 2024-11-11T15:11:49.2869649Z Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence 2024-11-11T15:11:49.2870168Z Vulnerability Srbds: Not affected 2024-11-11T15:11:49.2870455Z Vulnerability Tsx async abort: Not affected 2024-11-11T15:11:49.2870651Z 2024-11-11T15:11:49.2870737Z Versions of relevant libraries: 2024-11-11T15:11:49.2870974Z [pip3] mypy==1.11.2 2024-11-11T15:11:49.2871205Z [pip3] mypy-extensions==1.0.0 2024-11-11T15:11:49.2871427Z [pip3] numpy==1.22.4 2024-11-11T15:11:49.2871616Z [pip3] onnx==1.17.0 2024-11-11T15:11:49.2871809Z [pip3] onnxscript==0.1.0.dev20240817 2024-11-11T15:11:49.2872054Z [pip3] optree==0.13.0 2024-11-11T15:11:49.2872271Z [pip3] torch==2.6.0a0+gitceb44b2 2024-11-11T15:11:49.2872501Z [pip3] torchvision==0.20.0a0+66c5629 2024-11-11T15:11:49.2872742Z [pip3] triton==3.1.0 2024-11-11T15:11:49.2873079Z [conda] mkl-include 2025.0.0 pypi_0 pypi 2024-11-11T15:11:49.2873523Z [conda] mkl-static 2025.0.0 pypi_0 pypi 2024-11-11T15:11:49.2873922Z [conda] numpy 1.22.4 pypi_0 pypi 2024-11-11T15:11:49.2874308Z [conda] optree 0.13.0 pypi_0 pypi 2024-11-11T15:11:49.2874691Z [conda] torch 2.6.0a0+gitceb44b2 pypi_0 pypi 2024-11-11T15:11:49.2875109Z [conda] torchvision 0.20.0a0+66c5629 pypi_0 pypi 2024-11-11T15:11:49.2875508Z [conda] triton 3.1.0 pypi_0 pypi