Open GammaSQ opened 4 years ago
Any more details about libxl_device_pci_add failed: -3
? Maybe something in /var/log/xen/console/hypervisor.log
about that time? Or dom0 kernel messages?
I had the same problem with the vm sys-net. To be able to launch the sys-net vm, I had to disable all vms started at startup (including sys-net so sys-whonix, sys-firewall, and even sys-usb). After unlocking my user session, I am able to run sys-net manually without getting this error message: "Domain sys-net has failed to start: internal error: libxenlight failed to create new domain 'sys-net". Then simply create a script that launches the desired vms at startup. I hope that this will temporarily solve your problem until the developers find a solution to this problem. I guarantee to provide the logs that may be necessary to understand and correct the problem.
I think the most relevant piece of information is a message about RMRR, suggesting usage of iommu_inclusive_mapping=1
parameter. I don't know why I don't see this comment on github web interface, have you removed it?
This suggests a BIOS issue providing incorrect memory map. Try adding iommu_inclusive_mapping=1
parameter to Xen (options=
line in/boot/efi/EFI/qubes/xen.cfg
for UEFI systems).
Check also if VM start order matters here - what happens if you start sys-usb
first and then sys-net
- does it still work?
This just happened again, but this time with sys-net
(which only controls WIFI, not ethernet).
There was nothing in /var/log/xen/console/hypervisor.log
either and /var/log/libvirt/libxl/libxl-driver.log
showed the exact same three lines I already posted. (The file ends with them, so there is nothing more I could show.)
In fact, boot order did fix the problem: I shut down sys-usb, restarted sys-usb, started sys-net, now everything works.
For completeness: sys-usb initially started with an error:
qrexec-daemon startup failed: Connection to the VM failed
But I think this was just a fluke, second attempt immediately went through.
I had the same issue.
During the installation (first boot) I received this error message: "Domain sys-net has failed to start: internal error: libxenlight failed to create new domain 'sys-net".
@marmarek I tried the parameter iommu_inclusive_mapping=1
with no effect.
These are the steps I took to get successfully through the installation:
This is just a hot fix to come through the installation process.
The hot fix ensures that the installation process isn't interrupted. Maybe the user should be informed about the exception (at the moment the hot fix doesn't inform the user).
Same issue observed on Elitebook 2170p and 2570p right after installation (R4.0.3) WIth the exception that sys-usb won't start even with sys-net terminated. Same error:
Domain sys-usb has failed to start: internal error: Unable to reset PCI device 0000:00.14.0: internal error: libxenlight failed to create new domain for 'sys-usb'.
If I make sys-usb part of sys-net during the final configuration, then sys-net won't start. 00:14.0 is a USB controller.
Couldn't try the fix with iommu_inclusive_mapping=1
as xen.cfg is empty in my installation (MBR boot)
@asymcon pretty sure yours is a different and rather standard issue. Please try this in dom0:
qvm-pci detach sys-usb dom0:00_14.0 qvm-pci attach sys-usb --persistent --option no-strict-reset=true dom0:00_14.0
See also: https://www.qubes-os.org/doc/pci-devices/#additional-attach-options
@SvenSemmler I tried your suggestion, the commands returned no error, but still getting the same error on trying to start sys-usb. Terminating all other domains and then starting with sys-usb made no difference.
On Thu, Apr 16, 2020 at 10:03:36AM -0700, asymcon wrote:
@SvenSemmler I tried your suggestion, the commands returned no error, but still getting the same error on trying to start sys-usb. Terminating all other domains and then starting with sys-usb made no difference.
Can you please post the output of "qvm-pci" (is the USB controller assigned to more than one qube)?
Have you restarted qubes after applying my suggestion (this might be necessary)?
/Sven
-- public key: https://www.svensemmler.org/0x8F541FB6.asc fingerprint: D7CA F2DB 658D 89BC 08D6 A7AA DA6E 167B 8F54 1FB6
Yes, qubes was restarted. qvm-pci output below, USB controller is only assigned to sys-usb:
dom0:00_00.0 Host bridge: Intel Corporation 3rd Gen Core processor DRAM Controller
dom0:00_02.0 VGA compatible controller: Intel Corporation 3rd Gen Core processor Graphics Controller
dom0:00_14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller sys-usb (no-strict-reset=true)
dom0:00_16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1
dom0:00_16.3 Serial controller: Intel Corporation 7 Series/C210 Series Chipset Family KT Controller
dom0:00_19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) sys-net
dom0:00_1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 sys-usb (no-strict-reset=True)
dom0:00_1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller
dom0:00_1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1
dom0:00_1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2
dom0:00_1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3
dom0:00_1c.3 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 4
dom0:00_1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 sys-usb (no-strict-reset=True)
dom0:00_1f.0 ISA bridge: Intel Corporation QM77 Express Chipset LPC Controller
dom0:00_1f.2 SATA controller: Intel Corporation 7 Series Chipset Family 6-port SATA Controller [AHCI mode]
dom0:23_00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller
dom0:23_00.2 SD Host controller: JMicron Technology Corp. Standard SD Host Controller
dom0:24_00.0 Network controller: Intel Corporation Centrino Advanced-N 6205 [Taylor Peak] sys-net
Also an output of libxldriver:
2020-04-16 05:26:35.852+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/1/console/tty': Resource temporarily unavailable
2020-04-16 05:28:37.075+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/2/console/tty': Resource temporarily unavailable
2020-04-16 05:30:18.093+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/3/console/tty': Resource temporarily unavailable
2020-04-16 05:32:20.911+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/4/console/tty': Resource temporarily unavailable
2020-04-16 05:36:59.681+0000: libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2020-04-16 05:37:59.856+0000: libxl: libxl_exec.c:226:libxl__xenstore_child_wait_deprecated: Device Model not ready
2020-04-16 05:37:59.858+0000: libxl: libxl_pci.c:1016:qemu_pci_add_xenstore: qemu refused to add device: 0000:00:14.0,msitranslate=0,power_mgmt=0
2020-04-16 05:37:59.859+0000: libxl: libxl_pci.c:1361:libxl__add_pcidevs: libxl_device_pci_add failed: -3
2020-04-16 05:37:59.859+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add pci devices
2020-04-16 05:38:00.218+0000: libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2020-04-16 05:39:15.528+0000: libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2020-04-16 05:40:15.688+0000: libxl: libxl_exec.c:226:libxl__xenstore_child_wait_deprecated: Device Model not ready
2020-04-16 05:40:15.690+0000: libxl: libxl_pci.c:1016:qemu_pci_add_xenstore: qemu refused to add device: 0000:00:14.0,msitranslate=0,power_mgmt=0
2020-04-16 05:40:15.691+0000: libxl: libxl_pci.c:1361:libxl__add_pcidevs: libxl_device_pci_add failed: -3
2020-04-16 05:40:15.691+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add pci devices
2020-04-16 05:40:16.051+0000: libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2020-04-16 05:40:15.690+0000: libxl: libxl_pci.c:1016:qemu_pci_add_xenstore: qemu refused to add device: 0000:00:14.0,msitranslate=0,power_mgmt=0
This looks suspicious. Can you check /var/log/xen/console/guest-sys-usb-dm.log
and see what qemu complains about?
Also, I've noticed that for 00:14.0 device there is lowercase true
in no-strict-reset=true
, while in other cases it is True
. But I don't think that makes any difference.
Also, I've noticed that for 00:14.0 device there is lowercase
true
inno-strict-reset=true
, while in other cases it isTrue
. But I don't think that makes any difference.
Yes I noted that too, rather strange. Here's the guest-sys-usb-dm.log, not sure what I'm supposed to look for. guest-sys-usb-dm.log
@asymcon while waiting for @marmarek feedback it might be an idea to assign one USB controller at a time to test out if this is a general issue or one specific to one of your three controllers.
{"execute": "device_add", "arguments": {"driver": "xen-pci-passthrough", "id": "xen-pci-pt_0000-00-14.0", "hostaddr": "0000:00:01.00", "machine_addr": "0000:00:14.0", "permissive": false}}
[00:07.0] xen_pt_realize: Assigning real physical device 00:01.0 to devfn 0x38
[00:07.0] xen_pt_register_regions: IO region 0 registered (size=0x00010000 base_addr=0xd4720000 type: 0x4)
[00:07.0] xen_pt_config_reg_init: Offset 0x0006 mismatch! Emulated=0x0010, host=0x0290, syncing to 0x0290.
[00:07.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x0000, syncing to 0x0000.
[00:07.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x0000, host=0xd4720004, syncing to 0xd4720004.
[00:07.0] xen_pt_config_reg_init: Offset 0x0072 mismatch! Emulated=0x0000, host=0x01c2, syncing to 0x0002.
[00:07.0] xen_pt_config_reg_init: Offset 0x0082 mismatch! Emulated=0x0000, host=0x0086, syncing to 0x0080.
[00:07.0] xen_pt_pci_intx: intx=1
[00:07.0] xen_pt_realize: Real physical device 00:01.0 registered successfully
{"error": {"class": "GenericError", "desc": "Binding of interrupt 0 failed: Device or resource busy"}}
"interrupt 0" sounds wrong. Can you check lspci -v
in dom0 what value you see there? And also check dom0's kernel messages (pciback driver) and Xen (xl dmesg
) if nothing related to it is logged (but I doubt).
lspci.log xl_dmesg.log dmesg.log
@SvenSemmler Yes, I tested this yesteday - only 00:14.0 seem to fail this way.
Both 00:1a.0 and 00:1d.0 start correctly through sys-usb, and internal camera, which is connected to 00.1a.0 is correctly recognized. I tried playing with various USB related settings in bios, and also tried bios factory reset, with no effect unfortunately.
Starting sys-usb produces the last line in xl dmesg
(XEN) pt_irq_create_bind failed (-16) for dom7
and pciback errors in dom0 kernel
This sounds related:
[ 1203.879285] pciback 0000:00:14.0: can't derive routing for PCI INT A
[ 1203.879289] pciback 0000:00:14.0: PCI INT A: no GSI
But TBH I don't know why it happens.
Could it be due to older BIOS version? 2570p is listed under tested machines for R3.2, but mine has lower BIOS version.
Is there a workaround for the affected machines for the latest qubes release, or is the only option to use r3.2 and older?
I ran in to this issue when attempting to attach USB controllers in the Device Tab to a new Standalone HVM.
Start failed: internal error: libxenlight failed to create new domain '<vmName>'. see /var/log/libvirt/libxl/libxl-driver.log for details
This is on a fresh install of Qubes R4.0.3 and a new empty standalone qube, but was also happening on an older install with an existing VM.
It doesn't seem to happen for any other devices, only the two USB devices (same ones that sys-usb uses). Strict PCI Reset doesn't seem to change it.
It was working in the past, but had stopped working which is why I reinstalled Qubes.
It's possible this is due to a BIOS issue as I had flashed my BIOS at some point but I don't remember for sure if the error only started afterward. It might also be a specific setting in BIOS that needs to be changed.
Will try flashing it with another BIOS version and see if it makes a difference.
I reverted my BIOS to an older version, and using the same install of Qubes as before, encountered the same issue. However, after a clean install on the reverted BIOS, i was able to successfully attach my USB devices again without this error. I don't know for sure if the BIOS was the culprit as it's a pain to switch back and forth (complete disassembly and external flashing).
It could also be related to the status of the TPM/Security Chip during install. I believe my "Security Chip" was marked as inactive for the previous install, and the problem would persist even if enabled. The current working install had it enabled during install, and it continues to work even when disabled. I don't know if this is the actual cause or not or if there's some other issue.
Still experiencing same issue on R4.0.3 Tested machines: HP 8470p, HP 8570p, HP 2570p
Want to write issue, but in last moment found this. R4.1 Qubes-20210610-kernel-latest-x86_64.iso
Affected component(s) or functionality
qubesd sys-usb
Brief summary
I start vm "work" but vm freeze in 1-2min and i go to system tray for shutdown then kill vm In couple seconds i got error, something about "qubesd got empty response" After this error xfce panel crash too (propose - detach "qubes domains" commands from xfce panel) My next actions:
so, power off PC affected the sys-usb
Additional context
Here journalctl, but i think nothing about qubesd. Some about sys-usb
Jun 13 09:04:25 dom0 libvirtd[2937]: internal error: libxenlight failed to create new domain 'sys-usb'
Jun 13 09:04:25 dom0 libvirtd[2937]: internal error: Active 0000:12:00.2 devices on bus with 0000:12:00.0, not doing bus reset
Jun 13 09:04:25 dom0 libvirtd[2937]: internal error: Unable to reset PCI device 0000:12:00.0: internal error: Active 0000:12:00.2 devices on bus with 0000:12:00.0, not doing bus reset
Jun 13 09:04:25 dom0 qubesd[3072]: vm.sys-usb: Start failed: internal error: Unable to reset PCI device 0000:12:00.0: internal error: Active 0000:12:00.2 devices on bus with 0000:12:00.0, not doing bus reset
Jun 13 09:04:25 dom0 systemd-homed[2831]: block device /sys/devices/virtual/block/dm-43 has been removed.
Jun 13 09:04:26 dom0 systemd-homed[2831]: block device /sys/devices/virtual/block/dm-43 has been removed.
Jun 13 09:04:26 dom0 systemd-homed[2831]: block device /sys/devices/virtual/block/dm-44 has been removed.
Jun 13 09:04:26 dom0 systemd-homed[2831]: block device /sys/devices/virtual/block/dm-44 has been removed.
Jun 13 09:04:26 dom0 systemd-homed[2831]: block device /sys/devices/virtual/block/dm-45 has been removed.
Jun 13 09:04:26 dom0 systemd-homed[2831]: block device /sys/devices/virtual/block/dm-45 has been removed.
Jun 13 09:04:27 dom0 qvm-start[3142]: Start failed: internal error: Unable to reset PCI device 0000:12:00.0: internal error: Active 0000:12:00.2 devices on bus with 0000:12:00.0, not doing bus reset, see /var/log/libvirt/libxl/libxl-driver.log for details
Jun 13 09:04:27 dom0 systemd[1]: qubes-vm@sys-usb.service: Main process exited, code=exited, status=1/FAILURE
Jun 13 09:04:27 dom0 systemd[1]: qubes-vm@sys-usb.service: Failed with result 'exit-code'.
Jun 13 09:04:27 dom0 systemd[1]: Failed to start Start Qubes VM sys-usb.
Jun 13 09:04:27 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-vm@sys-usb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jun 13 09:04:27 dom0 kernel: kauditd_printk_skb: 67 callbacks suppressed
Jun 13 09:04:27 dom0 kernel: audit: type=1130 audit(1623589467.208:156): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-vm@sys-usb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
libxl-driver.log but time is different
2021-06-13 13:04:14.577+0000: libxl: libxl_pci.c:1488:libxl__device_pci_reset: write to /sys/bus/pci/devices/0000:12:00.0/reset returned -1: Inappropriate ioctl for device
2021-06-13 13:04:24.762+0000: libxl: libxl_pci.c:1772:device_pci_add_done: Domain 1:libxl__device_pci_add failed for PCI device 0:12:0.0 (rc -9)
2021-06-13 13:04:24.762+0000: libxl: libxl_pci.c:1772:device_pci_add_done: Domain 1:libxl__device_pci_add failed for PCI device 0:29:0.3 (rc -9)
2021-06-13 13:04:24.763+0000: libxl: libxl_create.c:1904:domcreate_attach_devices: Domain 1:unable to add pci devices
2021-06-13 13:04:24.929+0000: libxl: libxl_pci.c:1488:libxl__device_pci_reset: write to /sys/bus/pci/devices/0000:12:00.0/reset returned -1: Inappropriate ioctl for device
2021-06-13 13:04:25.274+0000: libxl: libxl_exec.c:120:libxl_report_child_exitstatus: killed fork (dying as expected) [5055] unexpectedly exited status zero
2021-06-13 13:07:01.882+0000: libxl: libxl_pci.c:1488:libxl__device_pci_reset: write to /sys/bus/pci/devices/0000:12:00.0/reset returned -1: Inappropriate ioctl for device
2021-06-13 13:07:12.098+0000: libxl: libxl_pci.c:1772:device_pci_add_done: Domain 4:libxl__device_pci_add failed for PCI device 0:29:0.4 (rc -9)
2021-06-13 13:07:12.102+0000: libxl: libxl_pci.c:1772:device_pci_add_done: Domain 4:libxl__device_pci_add failed for PCI device 0:12:0.0 (rc -9)
2021-06-13 13:07:12.102+0000: libxl: libxl_create.c:1904:domcreate_attach_devices: Domain 4:unable to add pci devices
2021-06-13 13:07:12.446+0000: libxl: libxl_pci.c:1488:libxl__device_pci_reset: write to /sys/bus/pci/devices/0000:12:00.0/reset returned -1: Inappropriate ioctl for device
2021-06-13 13:10:57.333+0000: libxl: libxl_pci.c:1488:libxl__device_pci_reset: write to /sys/bus/pci/devices/0000:12:00.0/reset returned -1: Inappropriate ioctl for device
guest-sys-usb-dm.log
hypervisor.log editor say about contains invalid \00\00\00\00\00\00
this relates #6677
lspci.log
xl_dmesg.log
Active 0000:12:00.2 devices on bus with 0000:12:00.0
According to lspci:
12:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43bc (rev 02) (prog-if 30 [XHCI])
12:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b8 (rev 02) (prog-if 01 [AHCI 1.0])
12:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b3 (rev 02) (prog-if 00 [Normal decode])
Those three devices (or rather: functions of a single device) needs to be assigned to the same VM. This is potentially bad, if you use this SATA controller for your primary disk, because that would mean you need to keep this USB controller in dom0 too. You do have another SATA controller in the system, I really hope the other one is for your disk...
Assuming you don't need 12:00.1 SATA controller in dom0, the solution is to assign all three to sys-usb. But to be on the safe side, I'd check first detaching them from dom0 only to see if you really don't need them there (if that breaks something, simply restart and the change will be reverted) - execute in dom0:
sudo xl pci-assignable-add 12:00.0
sudo xl pci-assignable-add 12:00.1
sudo xl pci-assignable-add 12:00.2
PS this is related, but a different issue than the original here.
Well, with a MacBookPro Retina 15" mid-2012 (Macbookpro 10,1 - with EFI ROM : MBP101.00EE.B0B) I have the exact same problem as you do, the same logs.
I can't detach the USB controller in PCI 0000:00:14.0 that handles the two external USB Ports of the MacBookPro (There are 3 USB controllers listed in the macbookpro motherboard, the two other USB controller are listed as PCI 0000:00:1d.0 and 0000:00:1a.0) from dom0 to attach it to my sys-usb qube and be able to handle networking thanks to this.
This is really terrible and raging, because I can't use any USB network adapter, or USB Wifi network adapter, and as the original PCI Wifi card of the Mac is removed (for security reasons, I like to be able to cut wifi physicaly by unpluging something WHEN I want, you know, usual crypto-anarchist legit paranoia), I have strictly no way to access the internet or use any USB device under Qubes-OS for the moment.
If you have a way to correct this small error, because it must be a very simple misconfiguration somewhere, please let me know because it is raging not to be able to make Qubes-OS run on this damned macbookpro just for this.
I am okay to spend time on this issue, if somebody gives me clues and where to look at in the developper documentation, it must be something very trivial.
@asymcon pretty sure yours is a different and rather standard issue. Please try this in dom0:
qvm-pci detach sys-usb dom0:00_14.0 qvm-pci attach sys-usb --persistent --option no-strict-reset=true dom0:00_14.0
See also: https://www.qubes-os.org/doc/pci-devices/#additional-attach-options
Simply from one moment to the next when opening the pc I had the same problem, on Qubes 4.1, both with debian and fedora template. Whenever the problem occurs to me, sometimes when I turn on the pc, sometimes when I close the laptop screen the temporary solution is to try to open sys-usb and after the error message use this command, and then the VM sys-usb starts normally.
I tried to add the command iommu_inclusive_mapping=1
as suggested even though my xen.cfg file is also empty, and it had no effect. Should I add something else to the file?
What could be a definitive solution to the problem? Thanks in advance
I also have this sometimes in 4.1, didn't have it in 4.0.
Qubes OS version 4.0
Affected component(s) or functionality PCI / VM-start
Brief summary I have a VM handling the ethernet-device. It suddenly doesn't start due to
libxl
-errorTo Reproduce Unsure. I power cycle this VM quite often, I guess during the last power down, the PCI wasn't reset correctly? It happened for the first time though, so I will update when it happens again.
Expected behavior VMs that had no problem starting once should start again.
Actual behavior with pci attached, I get the error
Additional context qvm-start told me to look at
/var/log/libvirt/libxl/libxl-driver.log
:Solutions you've tried detaching pci allows the VM to boot smoothly. Attaching with any additional options (
permissive
andno-strict-reset
) doesn't fix the problem.Relevant documentation you've consulted https://www.qubes-os.org/doc/pci-devices/
Related, non-duplicate issues https://github.com/QubesOS/qubes-issues/issues/5336 (Definitely support VT-d, runs QubesOS smoothly since years) https://github.com/QubesOS/qubes-issues/issues/4841 (no minimal template) https://github.com/QubesOS/qubes-issues/issues/4183 (no kernel- or other update between working and failing boot up)