Run xhyve with network without sudo

[rickard-von-essen]

This is the main pain point with xhyve. I'm very interested in getting this fixed or getting a workaround.

Could we detail the problem and possible solutions/work arounds, here.

[rimusz]

+1

[xez]

Perhaps as a workaround one could use setuid at install time:

sudo chown root build/xhyve
sudo chmod +s build/xhyve

So xhyve will not require an explicit sudo to run with networking. Even better would be to add a patch to xhyve to drop the root privileges as soon as it has instantiated the network connections.

[AntonioMeireles]

@xez the sudo is a OSX's Hypervisor framework 'defense'. afaict one would just need to run a signed build of xhyve for the sudo stop being required.

[xez]

Hypervisor.framework does not require sudo. If it is sandboxed it requires an entitlement. vmnet.framework however requires sudo or an entitlement. From my experiments these don't seem to be public entitlements (App Store only). Bummer :(

[AntonioMeireles]

@xez fwiw if i'm reading things right one doesn't really need to go thru App store just have a valid apple developer cert to use with codesign (which implies an Apple's Developer Program membership and USD $99/yr). something along what's bellow may work (would be nice, to get sure, if there's someone here which is a Apple registered dev could try snippet bellow (just replacing "VALID APPLE ID" in the Makefile by the real thing...)

❯❯❯ git diff edgy                                                                                                                                                                                                                   
diff --git a/Makefile b/Makefile
index 3294494..306708c 100644
--- a/Makefile
+++ b/Makefile
@@ -77,6 +77,7 @@ DEP := $(OBJ:%.o=%.d)
 INC := -Iinclude

 CFLAGS += -DVERSION=\"$(GIT_VERSION)\"
+LDFLAGS += -mmacosx-version-min=10.10 -sectcreate __TEXT __info_plist xhyve.info.plist

 TARGET = build/xhyve

@@ -95,6 +96,7 @@ build/%.o: src/%.c
        @mkdir -p $(dir $@)
        $(VERBOSE) $(ENV) $(CC) $(CFLAGS) $(INC) $(DEF) -MMD -MT $@ -MF build/$*.d -o $@ -c $<

+
 $(TARGET).sym: $(OBJ)
        @echo ld $(notdir $@)
        $(VERBOSE) $(ENV) $(LD) $(LDFLAGS) -Xlinker $(TARGET).lto.o -o $@ $(OBJ)
@@ -104,6 +106,8 @@ $(TARGET).sym: $(OBJ)
 $(TARGET): $(TARGET).sym
        @echo strip $(notdir $@)
        $(VERBOSE) $(ENV) $(STRIP) $(TARGET).sym -o $@
+       codesign -fs "VALID APPLE ID" --prefix hypervisor. -v --entitlements xhyve.entitlements.plist -r xhyve.rqset  $(TARGET) --deep 
+       codesign -d -r- $(TARGET)

 clean:
        @rm -rf build
diff --git a/xhyve.entitlements.plist b/xhyve.entitlements.plist
new file mode 100644
index 0000000..83e90e0
--- /dev/null
+++ b/xhyve.entitlements.plist
@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+  <dict>
+    <key>com.apple.security.app-sandbox</key>
+    <true/>
+    <key>com.apple.security.inherit</key>
+    <true/>
+    <key>com.apple.security.network.client</key>
+    <true/>
+    <key>com.apple.security.network.server</key>
+    <true/>
+  </dict>
+</plist>
diff --git a/xhyve.info.plist b/xhyve.info.plist
new file mode 100644
index 0000000..d5fcee9
--- /dev/null
+++ b/xhyve.info.plist
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>CFBundleDevelopmentRegion</key>
+    <string>English</string>
+    <key>CFBundleIdentifier</key>
+    <string>hypervisor.xhyve</string>
+    <key>CFBundleInfoDictionaryVersion</key>
+    <string>6.0</string>
+    <key>CFBundleName</key>
+    <string>xhyve</string>
+    <key>CFBundleVersion</key>
+    <string>0.3</string>
+</dict>
+</plist>
diff --git a/xhyverun.sh b/xhyverun.sh
index fcd7185..4f6dedf 100755
--- a/xhyverun.sh
+++ b/xhyverun.sh
@@ -12,7 +12,7 @@ CMDLINE="earlyprintk=serial console=ttyS0"

 MEM="-m 1G"
 #SMP="-c 2"
-#NET="-s 2:0,virtio-net"
+NET="-s 2:0,virtio-net"
 #IMG_CD="-s 3,ahci-cd,/somepath/somefile.iso"
 #IMG_HDD="-s 4,virtio-blk,/somepath/somefile.img"
 PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
@@ -21,6 +21,7 @@ ACPI="-A"
 #UUID="-U deadbeef-dead-dead-dead-deaddeafbeef"

[rimusz]

@AntonioMeireles is right, the same way I sign my https://github.com/rimusz/coreos-xhyve-ui App

[AntonioMeireles]

@rimusz so you 're volunteering to get us all signed builds :smile_cat: (hint, hint) ? @mist64, while on this, fwiw, it may start to make sense to think in simething along a xhyve org in github to distribute load, etc (and keep entropy manageable)...

[rimusz]

@AntonioMeireles Yes, I'm, as that should take that pain of sudo away I hope :)

[alimoeeny]

It's not all about "the pain of sudo" right? if you require sudo you leave open the possibility of a whole set of problems and exploits and stuff, if I understand things correctly. That's why we love homebrew :)

[avsm]

@rimusz Do you have a signed binary of xhyve available that I could try out? When I try signing using a developer key and App Sandboxing active, I end up with an illegal instruction exception in the Hypervisor.framework. The binary releases of xhyve in coreos-xhyve-ui don't seem to have any entitlements in them.

[rimusz]

@avsm no, I do not have the signed binary yet, as I was too busy on different things for a while. I will start working on updated coreos-xhyve-ui in the next week and see what I can come with the xhyve without sudo.

[rimusz]

right, finally got a time to try to build the signed xhyve binary, still no joy without the sudo :(

[AntonioMeireles]

would need more details... but at sight something would be missing along the line in the entitlements part as it should just work afaict.

[gpolitis]

@AntonioMeireles I'm using a self-signed certificate and your patch to codesign xhyve, but xhyve.rqset is missing from your patch. If I codesign without it, xhyve fails to create the VM (with or without virtio-net) with:

vmx_init: processor not supported by Hypervisor.framework
Unable to create VM (-85377018)

I'm not very familiar with the code signing process on OSX but I'm curious to see what's inside the missing xhyve.rqset.

[rimusz]

@AntonioMeireles @gpolitis I have got the same problem with that missing xhyve.rqset and without it xhyve fails to create the VM too

[AntonioMeireles]

@rimusz @gpolitis et al.

sorry, xhyve.rqsetshouldn't had been there - was a leftover/ oversight of my part (when i was trying to 'fool' OS X to let me sign stuff without (a valid but) non apple issues certificate)... (from here) what does codesign -d -r $BLOB / codesign -dvvvv -r $BLOB / spctl --assess --type execute $BLOB give on your side after you build them signed ?

[gpolitis]

$ codesign -dvvv -r- build/xhyve
Executable=/Users/gp/Projects/xhyve/build/xhyve
Identifier=hypervisor.xhyve
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=1225 flags=0x0(none) hashes=53+5 location=embedded
Hash type=sha1 size=20
CDHash=b96eea61753977602e62ca41560ec225fb439b4a
Signature size=1278
Authority=gp
Signed Time=Oct 25, 2015, 3:09:33 PM
Info.plist entries=5
TeamIdentifier=not set
Sealed Resources=none
designated => identifier "hypervisor.xhyve" and certificate leaf = H"92ce583d98ed5a0762f95365157ee78614afb4ed"
$ codesign -d -r- build/xhyve
Executable=/Users/gp/Projects/xhyve/build/xhyve
designated => identifier "hypervisor.xhyve" and certificate leaf = H"92ce583d98ed5a0762f95365157ee78614afb4ed"
$ spctl --assess --type execute build/xhyve
build/xhyve: rejected

[gpolitis]

When I run the sandboxed xhyve executable, sandboxd refuses to open some device needed by the hv_vm_create function:

xhyve(16791) deny iokit-open AppleHVClient

Process:         xhyve [16791]
Path:            /Users/gp/Projects/xhyve/build/xhyve
Load Address:    0x1033f6000
Identifier:      xhyve
Version:         ??? (???)
Code Type:       x86_64 (Native)
Parent Process:  sh [16790]

Date/Time:       2015-10-29 21:29:03.419 -0500
OS Version:      Mac OS X 10.11.1 (15B42)
Report Version:  8

Thread 0:
0   libsystem_kernel.dylib          0x00007fff89e15c96 mach_msg_trap + 10
1   IOKit                           0x00007fff928548c9 io_service_open_extended + 128
2   IOKit                           0x00007fff927f7332 IOServiceOpen + 45
3   Hypervisor                      0x0000000103635e65 hv_vm_create + 149
4   xhyve                           0x000000010341324d main + 1957 (vmx.c:468)
5   libdyld.dylib                   0x00007fff881225ad start + 1

Binary Images:
       0x1033f6000 -        0x10341fff7  xhyve (0) <b2d864dc-b8c8-3d94-b0af-3cc5f8abe391> /Users/gp/Projects/xhyve/build/xhyve
       0x103635000 -        0x103639fff  com.apple.Hypervisor (1.0 - 1) <4deeba77-3b83-3b64-bbb8-25ca44d31e38> /System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor
    0x7fff8811f000 -     0x7fff88122ffb  libdyld.dylib (360.17) <99396189-e53a-3a78-bc49-449236da7cff> /usr/lib/system/libdyld.dylib
    0x7fff89e04000 -     0x7fff89e22fff  libsystem_kernel.dylib (3247.10.11) <fa4dcf42-27b0-3878-9158-e148d208b21e> /usr/lib/system/libsystem_kernel.dylib
    0x7fff927f1000 -     0x7fff92864fff  com.apple.framework.IOKit (2.0.2) <6a9d7e11-156a-3fdb-850e-114ecb1ae7fd> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit

The protected device could be the raw CPU but I'm not sure because I have no way of verifying which device it is.

[gpolitis]

@xez you mention that

If it is sandboxed it requires an entitlement. vmnet.framework however requires sudo or an entitlement.

Do you know which entitlements are required?

[xez]

Mh, just going by /System/Library/Sandbox/Profiles/application.sb:

(if (entitlement "com.apple.vm.hypervisor")
  (allow iokit-open (iokit-user-client-class "AppleHVClient")))
(if (entitlement "com.apple.vm.networking")
  (allow mach-lookup (global-name "com.apple.NetworkSharing")))

so com.apple.vm.hypervisor for Hypervisor.framework and com.apple.vm.networkingfor vmnet.framework, I guess.

[gpolitis]

11/3/15 7:14:09.954 PM taskgated[219]: no application identifier provided, can't use provisioning profiles [pid=7146] 11/3/15 7:14:09.954 PM taskgated[219]: killed hypervisor.xhyve[pid 7146] because its use of the com.apple.vm.hypervisor entitlement is not allowed (error code -67050)

As @xez would have expected, I guess :-)

[jeremyhu]

The changes that I made in pull request #75 should allow for easier development in this area. I just grabbed the entitlements from Antonio's comment above.

[geoff-nixon]

@AntonioMeireles :+1: Thanks for this.

[AntonioMeireles]

@geoff-codes welcome!

[dalehamel]

Has anyone managed to come up with a working, code-signing+entitlements based solution?

I have an apple developer account and I provisioned an app with a developer ID deployment and certificate for this. I added the certificate to my keyring, and applied the patch provided by @AntonioMeireles above. I ran into the same issue as @gpolitis (xhyve killed because of missing entitlement).

2015-12-05 11:51:15.821 PM taskgated[1143]: killed hypervisor.xhyve[pid 30819] because its use of the com.apple.vm.hypervisor entitlement is not allowed (error code -67050)

I tried patching the entitlements to include networking and hypervisor, as below. I'm still seeing the app get killed, only now it happens immediately when even trying to invoke xhyve (won't even print help), where previously without these entitlements xhyve would at least run.

If anyone has gotten this working, I'd like to know how. It seems like these entitlements may not be supported yet, as the app gets killed when trying to use them. Further, they don't show up in any documentation I could find, and don't have corresponding xcode options.

I am trying to build a rubygem that uses xhyve (https://github.com/dalehamel/xhyve-ruby), and would prefer users not have to run the bundled xhyve binary code as root for security reasons.

Below is my adjusted patch:

iff --git a/Makefile b/Makefile
index 3294494..eedf336 100644
--- a/Makefile
+++ b/Makefile
@@ -77,6 +77,8 @@ DEP := $(OBJ:%.o=%.d)
 INC := -Iinclude
 CFLAGS += -DVERSION=\"$(GIT_VERSION)\"
+LDFLAGS += -mmacosx-version-min=10.10 -sectcreate __TEXT __info_plist xhyve.info.plist
@@ -104,6 +106,8 @@ $(TARGET).sym: $(OBJ)
 $(TARGET): $(TARGET).sym
        @echo strip $(notdir $@)
        $(VERBOSE) $(ENV) $(STRIP) $(TARGET).sym -o $@
+       codesign -fs "VALID APPLE ID" --prefix hypervisor. -v --entitlements xhyve.entitlements.plist -r xhyve.rqset  $(TARGET) --deep 
+       codesign -d -r- $(TARGET)
diff --git a/xhyve.entitlements.plist b/xhyve.entitlements.plist
new file mode 100644
index 0000000..e0e11cf
--- /dev/null
+++ b/xhyve.entitlements.plist
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+  <dict>
+    <key>com.apple.security.app-sandbox</key>
+    <true/>
+    <key>com.apple.security.inherit</key>
+    <true/>
+    <key>com.apple.security.network.client</key>
+    <true/>
+    <key>com.apple.security.network.server</key>
+    <true/>
+    <key>com.apple.vm.hypervisor</key>
+    <true/>
+    <key>com.apple.vm.networking</key>
+    <true/>
+  </dict>
+</plist>
diff --git a/xhyve.info.plist b/xhyve.info.plist
new file mode 100644
index 0000000..d5fcee9
--- /dev/null
+++ b/xhyve.info.plist
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
+"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>CFBundleDevelopmentRegion</key>
+    <string>English</string>
+    <key>CFBundleIdentifier</key>
+    <string>hypervisor.xhyve</string>
+    <key>CFBundleInfoDictionaryVersion</key>
+    <string>6.0</string>
+    <key>CFBundleName</key>
+    <string>xhyve</string>
+    <key>CFBundleVersion</key>
+    <string>0.3</string>
+</dict>
+</plist>

From my reading of this issue and #75 it seems as though no one actually has this working yet?

@rimusz it looks like you never got this working for your coreos-ui app?

But, if anyone has an entitlement /code signing/ sandbox solution working, please help me know what I'm doing wrong :)

[geoff-nixon]

@dalehamel No, I never did get it working; I ran into the same issue you did.

Maybe one of us should open a radar bug (or a TSI) and see if we can get a response from Apple?

[jeremyhu]

@geoff-codes taskgate isn't complaining here when building it from my Xcode project:

$ git clone git@github.com:jeremyhu/xhyve.git
$ cd xhyve
$ xcodebuild
$ build/Release/xhyve --help

Do you have issues when building like that? If not, I suggest you figure out what is different about the Makefile build failing.

[dalehamel]

Are you able to actually start a Guest with networking without sudo using the Xcode build? @jeremyhu

On Monday, 7 December 2015, Jeremy Huddleston Sequoia < notifications@github.com> wrote:

@geoff-codes https://github.com/geoff-codes taskgate isn't complaining here when building it from my Xcode project:

$ git clone git@github.com:jeremyhu/xhyve.git $ cd xhyve $ xcodebuild $ build/Release/xhyve --help

Do you have issues when building like that? If not, I suggest you figure out what is different about the Makefile build failing.

— Reply to this email directly or view it on GitHub https://github.com/mist64/xhyve/issues/60#issuecomment-162418333.

[dalehamel]

@geoff-codes i could file one, i found out my company has access to two.

Perhaps we can all put our heads together to come up with a succinct description of the problem?

It would be great to submit the xcode stuff, as it is likely going to be better supported by apple. Once that's working we can always rev eng it.

[geoff-nixon]

@jeremyhu

Do you have issues when building like that?

Yep.

~> git clone git://github.com/jeremyhu/xhyve
~> cd xhyve
~> xcodebuild CODE_SIGN_IDENTITY="Developer ID Application" # Actually, ad-hoc signing seems to do fine with these four entitlements.
~> build/Release/xhyve --help # Yes, this is fine...
Usage: xhyve [-behuwxACHPWY] [-c vcpus] [-g <gdb port>] [-l <lpc>]
...
~> ln build/Release/xhyve build/xhyve
~> ./xhyverun.sh # ... but this isn't. :(
vmx_init: processor not supported by Hypervisor.framework
Unable to create VM (-85377018)

In system.log we get hyve(12345) deny iokit-open AppleHVClient (as @gpolitis did, above), so I edit src/xhyve-entitlements.plist to add com.apple.vm.hypervisor and com.apple.vm.networking, and ...

~> xcodebuild clean
~> xcodebuild CODE_SIGN_IDENTITY="Developer ID Application"
~> ./build/Release/xhyve --help
'./build/Release/xhyve --help' terminated by signal SIGKILL (Forced quit)

... and now its taskgate complaining: taskgated[236]: no application identifier provided, can't use provisioning profiles [pid=44315] taskgated[236]: killed xyz.xhyve.xhyve[pid 44315] because its use of the com.apple.vm.hypervisor entitlement is not allowed (error code -67050)

and we get this crash report:

Process:               xhyve [44315]
Path:                  /Users/USER/*/xhyve
Identifier:            xhyve
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        fish [7159]
Responsible:           xhyve [44315]
User ID:               501

Date/Time:             2015-12-07 07:46:27.519 -0800
OS Version:            Mac OS X 10.11.1 (15B42)
Report Version:        11
Anonymous UUID:        7738B1E3-9213-F179-1B36-9C45C052E7DD

Time Awake Since Boot: 10000 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_CRASH (Code Signature Invalid)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

---

@dalehamel I would say maybe lets hold off on that for now, as long as we have our man @jeremyhu on the case. He might actually be able to see something on the rdar://, unlike the rest of us. :wink:

[jeremyhu]

Ah, /facepalm. I copied an earlier entitlements.plist into my xcode patch and didn't look back. I just assumed that it was the same and based my earlier reply on that false assumption (and the comment that the error occurred by taskgated denying the entitlement). Sorry for the confusion. I'm looking into it.

[jeremyhu]

I asked around, and it turns out that those entitlements are "App Store”-only entitlements and can not be self-signed or signed with a Developer ID. As such, I think the current options are unfortunately:

1. do not use the com.apple.vm.hypervisor entitlement, and run under sudo (or setuid root). =(
2. use the entitlements, but disable code signature verification on your system. =(

Both have obvious downsides and are sub-optimal. I'll continue looking into this problem space, but I likely won't have updates on this until (and if) changes in the base OS allow it.

[AntonioMeireles]

@jeremyhu so, if i got you right, you're saying that ALL OSX's sandboxing features regarding whitelisting of priv escalation ONLY work when apps are signed with an (Apple issued thru a valid ADC membership) certificate (which may look brutal but actually makes tons of sense, as it gives AAPL a quick way to revoke/blacklist it if need arises) but ALSO and ONLY got distributed via the App Store ? is this specific to Hypervisor.framework or, in general, codesign only gets really usefull is app goes via the App Store ? (also, sorry in advance if this is in some obvious place i've missed - where is (url) this stated on some docs ?) /cc @rimusz

[dalehamel]

Looking. At Xcode, it only lets you do sandboxing on App Store apps. So it looks like it's a matter of any sandboxing/entitlements.

Of course, a workaround would be for someone to make a free xhyve app. I'll give this a try.

On Monday, 7 December 2015, António Meireles notifications@github.com wrote:

@jeremyhu https://github.com/jeremyhu so, if i got you right, you're saying that ALL OSX's sandboxing features regarding whitelisting of priv escalation ONLY work when apps are signed with an (Apple issued thru a valid ADC membership) certificate (which may look brutal but actually makes tons of sense, as it gives AAPL a quick way to revoke/blacklist it if need arises) but ALSO and ONLY got distributed via the App Store ? is this specific to Hypervisor.framework or, in general, codesign only gets really usefull is app goes via the App Store ? (also, sorry in advance if this is in some obvious place i've missed - where is (url) this stated on some docs ?) /cc @rimusz https://github.com/rimusz

— Reply to this email directly or view it on GitHub https://github.com/mist64/xhyve/issues/60#issuecomment-162663500.

[geoff-nixon]

I wonder if an App Store App would get approved. Are pure-CLI tools allowed? Looking at the Mac App Store Review Guidelines, it looks like this might be trouble:

Apps that do not use system provided items, such as buttons and icons, correctly and as described in the OS X Human Interface Guidelines will be rejected.

So maybe we need to put the xhyve binary in Resources and make the application's executable be something like:

#!/usr/bin/osascript
display dialog "The xhyve binary is in the application bundle." buttons {"Awesome"} with icon path to resource "Terminal.icns" in bundle {POSIX path of "/Applications/Utilities/Terminal.app"}

:laughing: :speak_no_evil:

[geoff-nixon]

@AntonioMeireles Yes, after doing some move digging/experimenting it appears that is correct.

I've tried:

• Rebooting into Recovery and disabling System Integrity Protection (csrutil disable). No dice.
• Creating a custom provisioning profile (I have a paid developer program account), using my machine's UDID, installing the profile on my machine, and code-signing with this profile and the corresponding "Mac Developer" certificate – which is how you're supposed to be able to beta-test App Store apps. It still rejects the com.apple.vm.* entitlements.
  • So either I'm doing something wrong (quite possible - I've never actually done that before, nor submitted anything to the App Store),
  • or you can't even test a program using these entitlements before submitting (?).

Apparently, this business of App Store-only entitlements has actually been around for a while. The most illuminating piece of "documentation" regarding this business is @landonf's task-unchain. Alas, I even tried applying this patch to my 10.11 install; no dice.

I still think this is pretty bizarre. @jeremyhu, you mentioned,

1. use the entitlements, but disable code signature verification on your system. =(

Have you actually been able to do this? What's the secret?

[jeremyhu]

use the entitlements, but disable code signature verification on your system. =(

Have you actually been able to do this? What's the secret?

My boot-args are "-v debug=0x146 kext-dev-mode=1 amfi=0xff cs_enforcement_disable=1"

-v : verbose boot debug=0x146 : DB_LOG_PI_SCRN | DB_ARP | DB_NMI | DB_PRT, see Kernel documentation (currently Table 20-1) kext-dev-mode=1 : Disable codesigning verification of kexts amfi=0xff : Disable AMFI cs_enforcement_disable=1 : Disable codesignature enforcement

Plus, I set the following preferences:

sudo defaults write /Library/Preferences/com.apple.security.coderequirements Entitlements -string always
sudo defaults write /Library/Preferences/com.apple.security.coderequirements AllowUnsafeDynamicLinking -bool YES

The first overrides requirements for restricted entitlements. The second causes the system to not mark the process as having codesignature restrictions (specifically, csops(pid, CS_OPS_MARKRESTRICT, NULL, 0) is not called on the process).

What you require is a subset of all of that.

[jeremyhu]

Out of curiosity, why do you have com.apple.security.inherit in your entitlements.plist? I don't see any calls to fork(2), posix_spawn(2), system(3), or popen(3) in xhyve, so there should be no child processes. Am I missing something?

[AntonioMeireles]

@jeremyhu no, you're not missing anything. that is/was specific to my use case. there (https://github.com/TheNewNormal/corectl) i hacked things around a cli so that one could call xhyve in detached or non-detached mode, or even lunch multiple VMs in a single pass, and for that com.apple.security.inherit would end handy.

[darkn3rd]

👍

[dalehamel]

I think this issue should be closed as it doesn't not appear to be possible to do this without setuid

On Tuesday, 3 May 2016, Joaquin Menchaca notifications@github.com wrote:

👍

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/mist64/xhyve/issues/60#issuecomment-216674713

[rickard-von-essen]

Closing since it seems impossible.

[xpepermint]

Is this still the case?

[jeremyhu]

Is this still the case?

No. If you build using the Xcode project, it should setup the entitlements correctly for you.

[dexwolfram]

Hi, I've been spending quite some time trying to build xhyve using xcodebuild according to the instructions. Trying to run the resulting xcodebuild app, results in:

$ xhyve -h Killed: 9

I've tried various methods from within Xcode as well.

Instructions for xhyve say that:

macOS limits access to the networking API (vmnet) to builds that are code signed and have the appropriate entitlement. The code signing/entitlement requirement can be bypassed by running xhyve as root (via sudo)... Building xhyve via xcodebuild signs the build. Building xhyve via make does not.

Is my reading of this thread correct, that build with xcodebuild as per the instructions is not longer possible? It's possible that make is working, but I can't get xcodebuild to produce a functional exectuable, and it appears from this thread that there may be no point in this, since root may well be required regardless of whether or not the executable is signed. Is that correct? (I'm not a programmer.)

FYI, it appears others have run into this problem as well (google translate helps):

https://cloud-atlas.readthedocs.io/zh_CN/latest/kvm/macos/xhyve.html

编译后执行程序位于 build/Release/xhyve 。 在最新的 macOS Mojave 10.14.1 编译成功，运行 xhyve -h 失败，显示 Killed: 9 不过，实际上发现，使用完整的路径运行 buid/xhyve 则可以正常工作: /Users/huatai/github/xhyve/build/xhyve -h

https://houraku365.hatenablog.com/entry/2019/01/06/ubuntu18-on-xhyve-on-macosx-failed-again

しかし下記のように起動できなかったので、build 結果を削除しました。 $ ./build/Release/xhyve -h Killed: 9 $ sudo ./build/Release/xhyve -h Killed: 9 $ xcodebuild clean

[dexwolfram]

Hm. Looks like from the last couple of posts that build & run without su using xcodebuild is now supported. I am using a self-signed certificate. Do I need to pay $99 for a certificate-authority issued cert to get this to work?

[dexwolfram]

FYI, originally on xcodebuild I would get an app xhyve that terminates with "killed -9" (with or without sudo).

I just tried following these instructions:

https://kubadownload.com/news/codesign-sign-app

codesign --force --deep --sign - xhyve

... which re-codesigns xhyve. Now the app runs again with sudo, but still no networking.

I also tried turning on sandboxing with networking from within Xcode, then re- code signing with "codesign --force --deep --sign - xhyve", with the same outcome. I must be doing something wrong if they are saying this should now work. I don't have a developer's account, but Apple support tells me one should not be necessary to run on my own machine. They say to follow these instructions:

https://help.apple.com/xcode/mac/current/#/dev5a825a1ca

which... has a link to https://help.apple.com/xcode/mac/current/#/dev23aab79b4 where it says "If you are not a member of the Apple Developer Program, Xcode will create a personal team for you." But still this produces xhyve that terminates with killed -9, and that on re-sign "codesign --force --deep --sign - xhyve" only runs with sudo.

[jeremyhu]

FYI, originally on xcodebuild I would get an app xhyve that terminates with "killed -9" (with or without sudo).

I just tried following these instructions:

https://kubadownload.com/news/codesign-sign-app

codesign --force --deep --sign - xhyve

... which re-codesigns xhyve. Now the app runs again with sudo, but still no networking.

because your re-signing nuked the entitlements.

The build coming out of Xcode should be signed. What's the output of codesign -vvv /path/to/xhyve?

I also tried turning on sandboxing with networking from within Xcode

I don't know what you mean by this.

, then re- code signing with "codesign --force --deep --sign - xhyve", with the same outcome. I must be doing something wrong if they are saying this should now work. I don't have a developer's account, but Apple support tells me one should not be necessary to run on my own machine. They say to follow these instructions:

https://help.apple.com/xcode/mac/current/#/dev5a825a1ca

which... has a link to https://help.apple.com/xcode/mac/current/#/dev23aab79b4 where it says "If you are not a member of the Apple Developer Program, Xcode will create a personal team for you." But still this produces xhyve that terminates with killed -9, and that on re-sign "codesign --force --deep --sign - xhyve" only runs with sudo.

You should not need to re-sign. The binary is signed by Xcode. What does the crash log say? I suspect the SIGKILL is due to you signing adhoc. Please select a valid developer profile (pretty sure your free one is fine) to sign with, and you should be ok.

[dexwolfram]

Was finally able to get a developer account with Apple, but still no luck, now having made two code sign certificates, "Apple Development" and "Developer ID Application." I start with:

$ xcode-select --print-path /Applications/Xcode.app/Contents/Developer $ xcode-select --version xcode-select version 2373.

Catalina, ran 'sudo spctl --master-disable' to allow possibility of "Allow apps downloaded from anywhere" in MacOS security control panel.

Running xcodebuild from within xhyve directory appears to go well until the code signing part at the end, where there is an error:

$ xcodebuild

CodeSign /Users/dad/xhyve/build/Release/xhyve (in target 'xhyve' from project 'xhyve') cd /Users/dad/xhyve export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate

Signing Identity: "-"

/usr/bin/codesign --force --sign - -o library --entitlements /Users/dad/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/dad/xhyve/build/Release/xhyve

RegisterExecutionPolicyException /Users/dad/xhyve/build/Release/xhyve (in target 'xhyve' from project 'xhyve') cd /Users/dad/xhyve builtin-RegisterExecutionPolicyException /Users/dad/xhyve/build/Release/xhyve note: Execution policy exception registration failed and was skipped: Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted" (in target 'xhyve' from project 'xhyve')

BUILD SUCCEEDED

And again:

$ ~/xhyve/build/Release/xhyve -h

Killed: 9'

Based on a nebulous tip here: https://forums.developer.apple.com/thread/124529 , I tried alternatively:

$ xcodebuild -UseModernBuildSystem=NO

Interestingly, the RegisterExecutionPolicyException error at the end disappears, but the resulting binary is still killed.

ProcessProductPackaging "" build/xhyve.build/Release/xhyve.build/xhyve.xcent cd /Users/dad/xhyve

Entitlements:

{ "com.apple.security.get-task-allow" = 1; "com.apple.security.network.client" = 1; "com.apple.security.network.server" = 1; "com.apple.vm.hypervisor" = 1; "com.apple.vm.networking" = 1; }

builtin-productPackagingUtility -entitlements -format xml -o /Users/dad/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent

CodeSign build/Release/xhyve cd /Users/dad/xhyve export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate

Signing Identity: "-"

/usr/bin/codesign --force --sign - -o library --entitlements /Users/dad/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/dad/xhyve/build/Release/xhyve

BUILD SUCCEEDED

$ ~/xhyve/build/Release/xhyve -h

Killed: 9

With:

$ codesign -vvv ~/xhyve/build/Release/xhyve

/Users//xhyve/build/Release/xhyve: valid on disk /Users//xhyve/build/Release/xhyve: satisfies its Designated Requirement

I noticed that xcodebuild is not automatically selecting one of my signing certificates. Tried:

$ xcodebuild -SigningIdentity="Developer ID Application" -UseModernBuildSystem=NO

Still, the signing certificate is not chosen at the end

Signing Identity: "-"

So I went ahead and built again and followed up with codesign command to re-codesign using the same parameters as were invoked in the build, but still no luck:

$ xcodebuild -UseModernBuildSystem=NO $ codesign --verbose --force --sign "Developer ID Application" -o library --entitlements /Users/*/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/*/xhyve/build/Release/xhyve $ ~/xhyve/build/Release/xhyve -h Killed: 9

Same with this:

$ xcodebuild -UseModernBuildSystem=NO $ codesign --verbose --force --sign "Apple Development" -o library --entitlements /Users/*/xhyve/build/xhyve.build/Release/xhyve.build/xhyve.xcent --timestamp=none /Users/*/xhyve/build/Release/xhyve $ ~/xhyve/build/Release/xhyve -h Killed: 9

Crash log for xhyve always shows the same thing:

System Integrity Protection: enabled

Crashed Thread: Unknown

Exception Type: EXC_CRASH (Code Signature Invalid) Exception Codes: 0x0000000000000000, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY

Termination Reason: Namespace CODESIGNING, Code 0x1

A bit of a loss here. I feel like invoking 'xcodebuild -UseModernBuildSystem=NO' should have picked out a better signing certificate from the get-go, and not gone with 'Signing Identity: "-"'. But I'm not able to force it.

Must be doing something wrong, just not sure what. Change the bundle identifier? Add the mac to a provisioning profile and again try to codesign with the "Apple Development" certificate as a development version? Seems like the executable might expire before long in that case?

[dexwolfram]

On the bright side, I was able to get Ubuntu Server 20 running on xhyve very nicely (with sudo, nothing to do with this thread). I used the legacy BIOS server install image (as opposed to regular UEFI), found here:

http://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/

... not absolutely sure if that was critical, but at the very least it made things a lot easier. Also, I removed "acpi=off" from CMDLINE in the install script.

Absolutely critical was to pay close attention to the disk partitioning during install, and to apply a manual disk partition. Perhaps because of disk lettering (vba1, vba2...), the installer does not apply appropriate formatting to the partitions. I manually made a single ext4 partition for ubuntu, and manually marked it as bootable.

People are having trouble with Ubuntu 18 (https://github.com/machyve/xhyve/issues/161) and I suspect are running into the same troubles with 20.