soulitzer
commented
3 months ago Thanks for the report. Do you have a short self-contained script that can be used to reproduce the issue?
Open danyow-cheung opened 3 months ago
soulitzer
commented
3 months ago Thanks for the report. Do you have a short self-contained script that can be used to reproduce the issue?
malfet
commented
3 months ago Crash in UVM driver strongly suggests that at least part of the problem is there, as kernel driver should never crash, even if user input is completely wrong
danyow-cheung
commented
3 months ago Thanks for the report. Do you have a short self-contained script that can be used to reproduce the issue?
Thank you for replying .I used the bert-vits2 for inferencing
The main problem comes from not knowing the location and time of the program crash. In addition, when I use BERT VITS2 to start the program, an error immediately appears in the log. But the program can work
Also I would like to provide more informations about this error . If needed , I can share my inference info about bert-vits2
2024-05-27T09:52:30.369058+08:00 trana6a-Default-string kernel: [ 238.766333] Modules linked in: nvidia_uvm(POE) veth xt_nat xt_tcpudp xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_netfilter bridge stp llc rfcomm snd_seq_dummy snd_hrtimer nls_utf8 cifs cifs_arc4 cifs_md4 fscache netfs ccm overlay cmac algif_hash algif_skcipher af_alg bnep binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_hda_codec_realtek snd_soc_acpi soundwire_generic_allocation snd_hda_codec_generic soundwire_bus ledtrig_audio snd_hda_codec_hdmi snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine x86_pkg_temp_thermal intel_powerclamp snd_hda_intel coretemp snd_intel_dspcfg
2024-05-27T09:52:30.369060+08:00 trana6a-Default-string kernel: [ 238.766369] snd_intel_sdw_acpi iwlmvm nvidia_drm(POE) snd_hda_codec kvm_intel nvidia_modeset(POE) snd_hda_core snd_hwdep mac80211 kvm snd_pcm btusb btrtl btbcm snd_seq_midi btintel snd_seq_midi_event btmtk libarc4 snd_rawmidi nvidia(POE) irqbypass crct10dif_pclmul bluetooth cmdlinepart polyval_clmulni snd_seq polyval_generic iwlwifi ghash_clmulni_intel spi_nor ecdh_generic aesni_intel ecc snd_seq_device mtd snd_timer crypto_simd cryptd rapl snd cfg80211 mei_me spi_intel_pci intel_cstate i2c_i801 gigabyte_wmi wmi_bmof intel_lpss_pci drm_kms_helper intel_lpss spi_intel soundcore i2c_smbus mei idma64 intel_hid acpi_tad acpi_pad sparse_keymap mac_hid msr parport_pc ppdev lp parport drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 hid_generic usbhid hid nvme crc32_pclmul r8169 nvme_core ahci video xhci_pci realtek libahci xhci_pci_renesas nvme_common wmi pinctrl_alderlake
2024-05-27T09:52:30.369060+08:00 trana6a-Default-string kernel: [ 238.766408] CR2: ffffb699c264ff58
2024-05-27T09:52:30.369061+08:00 trana6a-Default-string kernel: [ 238.766409] ---[ end trace 0000000000000000 ]---
2024-05-27T09:52:30.369061+08:00 trana6a-Default-string kernel: [ 238.833290] RIP: 0010:0xffffb699c264ff58
2024-05-27T09:52:30.369062+08:00 trana6a-Default-string kernel: [ 238.833303] Code: ff ff b8 99 33 92 ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e6 00 40 92 ff ff ff ff <38> 9d 3d 9d 0b 7f 00 00 80 99 3d 9d 0b 7f 00 00 80 03 00 00 00 00
2024-05-27T09:52:30.369062+08:00 trana6a-Default-string kernel: [ 238.833304] RSP: 0018:ffffb699c264fed8 EFLAGS: 00010046
2024-05-27T09:52:30.369063+08:00 trana6a-Default-string kernel: [ 238.833306] RAX: 0000000000000000 RBX: ffffb699c264ff58 RCX: 0000000000000000
2024-05-27T09:52:30.369063+08:00 trana6a-Default-string kernel: [ 238.833307] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
2024-05-27T09:52:30.369064+08:00 trana6a-Default-string kernel: [ 238.833307] RBP: ffffb699c264fed8 R08: 0000000000000000 R09: 0000000000000000
2024-05-27T09:52:30.369064+08:00 trana6a-Default-string kernel: [ 238.833308] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
2024-05-27T09:52:30.369064+08:00 trana6a-Default-string kernel: [ 238.833309] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
2024-05-27T09:52:30.369065+08:00 trana6a-Default-string kernel: [ 238.833310] FS: 00007f0b973f86c0(0000) GS:ffff8e96ff840000(0000) knlGS:0000000000000000
2024-05-27T09:52:30.369065+08:00 trana6a-Default-string kernel: [ 238.833311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-05-27T09:52:30.369066+08:00 trana6a-Default-string kernel: [ 238.833312] CR2: ffffb699c264ff58 CR3: 000000010ac06000 CR4: 0000000000750ee0
2024-05-27T09:52:30.369066+08:00 trana6a-Default-string kernel: [ 238.833313] PKRU: 55555554
2024-05-27T09:52:30.369067+08:00 trana6a-Default-string kernel: [ 238.833314] note: python[2945] exited with irqs disabled
my env info when using bert-vits2
torch 2.0.0
torchaudio 2.0.0
torchvision 0.15.0
vector_quantize_pytorch 1.12.17when i tap
import torch;print(torch.version.cuda)the log shows
2024-05-29T17:01:41.081082+08:00 trana6a-Default-string kernel: [198781.508082] pip[1071441]: segfault at 0 ip 0000000000000000 sp 00007ffc96360338 error 14 in python3.10[400000+1f000] likely on CPU 8 (core 16, socket 0)
2024-05-29T17:01:41.081093+08:00 trana6a-Default-string kernel: [198781.508089] Code: Unable to access opcode bytes at 0xffffffffffffffd6.
🐛 Describe the bug
I opened two python programs on a linux machine . using the same gpu . However sometimes python programes crash , even when i just open a program , the
/var/log/kern.logimmediately display an error message . Here are some information belowI have sought help from the NVIDIA forum, but they have asked me to seek Torch's help
here's the link https://forums.developer.nvidia.com/t/linux-always-crash-when-run-the-python-program-using-gpu/292905
Looking forward to a reply
Versions
torch version is different
2.1.12.0.0nvidia-smi driver version :
nvcc --version
cc @ptrblck @msaroufim