iOS 14 seems to SIGKILL the ptrace trick

[saagarjha]

Describe the bug Just tried it out in a sample app and got a SIGKILL due to codesigning. I can confirm that CS_DEBUGGED is no longer being set for the process if you try to ptrace yourself.

To Reproduce Steps to reproduce the behavior:

1. ptrace yourself.
2. Allocate a writable page and put shellcode in it.
3. Mark the page R-X.
4. Run it.

let ptrace = unsafeBitCast(dlsym(dlopen(nil, RTLD_LAZY), "ptrace"), to: (@convention(c) (CInt, pid_t, caddr_t?, CInt) -> CInt).self)
print(ptrace(0, 0, nil, 0))

let page = mmap(nil, 0x1000, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0)
memcpy(page, [
    // mov x0, 1
    0x20,
    0x00,
    0x80,
    0xd2,

    // ret
    0xc0,
    0x03,
    0x5f,
    0xd6,
] as [UInt8], 8)
mprotect(page, 0x1000, PROT_READ | PROT_EXEC)
guard unsafeBitCast(page, to: (@convention(c) () -> Int).self)() == 1 else {
    preconditionFailure("wrong return")
}

Crash log

GetTaskAllowTest 6-24-20, 11-18.log

[osy]

Thanks for the confirmation. I think there’s a couple ways forward:

1. Make UTM a jailbreak only app
2. Something like AltStore for “tethered” apps to launch with debugger attached. I think this would be a lot of work and not really useful (if you have access to a PC, just use VNC)
3. Some sort of Linux powered dongle that can fake a debugger. Probably would be a massive undertaking and not that useful.
4. Find another way/pray Apple will allow dynamic codesigning in the future.

[saagarjha]

FWIW, I should probably grab a kernel at some point and see what the new check are. I wouldn't be all that surprised if Apple got rid of it because they couldn't be bothered to fix SIGKILL not working on traced processes…

[osy]

I have faint hope that Apple will bring the new JIT apis for APRR to iOS. https://developer.apple.com/documentation/apple_silicon/porting_just-in-time_compilers_to_apple_silicon

[saagarjha]

I guess we'll see soon enough. I wouldn't hold my breath, though.

(FWIW, I chatted with some Core OS engineers today and did get confirmation that an unkillable process is indeed considered a bug, though I'll be filing a macOS report for that ;) )

[professorUnknown]

Here is my analysis for the state of debugging on device in iOS 14.

1. PT_TRACE_ME is sigkilled, and doesn’t set the CS_DEBUGGED flag properly.
2. Another request that can be called on device would be PT_ATTACH, but it produces Mach exceptions.
3. The only request that works is the PT_ATTACHEXC, but this can’t currently be called on device.
4. Surprisingly the PT_ATTACHEXC request isn’t handled on Xcode, but instead on device by debug server.
5. https://iphonedevwiki.net/index.php/Debugserver Debug server in stock ios can’t attach to AppStore, TestFlight and non Xcode apps, but AltStore the preferred sideloading utility uses Xcode to install apps, therefore it is supported by AltStore with little to no changes by taking advantage of debug server you can run bulletproof jit.
6. There are 2 ways to do this, first is by loading debug server, and second is reverse engineering debug server.
7. In order to load debug server you need reverse engineer the Xcode interface for debug server.
8. Loading debug server would maybe require the vpn app extension in order to fake Xcode.
9. Loading debug server can be patched easily that’s why I suggest reverse engineering debug server.
10. By reverse engineering debug server you can easily send PT_ATTACHEXC requests on device.

[professorUnknown]

What Xcode uses to interact with debug server is Xcode connector that’s where we need to find how it can interface without privileges. https://appkitbox.com/en/testkit/support/tools/tools11/

[robmorgan]

Just so I understand correctly and for other users that may be reading this, the ptrace trick you are referring to is essentially: https://github.com/utmapp/UTM/blob/master/UTM/main.m#L61 and without this UTM can’t open a VM on iOS 14?

[professorUnknown]

Yes

[rileytestut]

5. https://iphonedevwiki.net/index.php/Debugserver Debug server in stock ios can’t attach to AppStore, TestFlight and non Xcode apps, but AltStore the preferred sideloading utility uses Xcode to install apps, therefore it is supported by AltStore with little to no changes by taking advantage of debug server you can run bulletproof jit.
6. There are 2 ways to do this, first is by loading debug server, and second is reverse engineering debug server.
7. In order to load debug server you need reverse engineer the Xcode interface for debug server.
8. Loading debug server would maybe require the vpn app extension in order to fake Xcode.
9. Loading debug server can be patched easily that’s why I suggest reverse engineering debug server.
10. By reverse engineering debug server you can easily send PT_ATTACHEXC requests on device.

The libimobiledevice project includes an open source implementation of communicating with debugserver, which might serve as a good reference for figuring out how to accomplish this on-device. I've considered using debugserver myself to launch AltStore in the background on demand whenever AltServer detects it on the same network, but in that case I'd be connecting to debugserver from AltServer on the computer, not from the iOS device itself.

As for reverse engineering debugserver, while that may be possible I'm slightly more skeptical since IMO it's likely the functionality we need is gated behind a private entitlement or something. I haven't actually looked much into that specifically tho so would love to be wrong 🤷‍♂️

[osy]

Thanks for looking into it. How easy would it be to have an AltStore feature that essentially lets you trigger AltServer launching an app with debugger attached? That might be a workaround for non-jb users on iOS 14 if we can't find anything better.

[saagarjha]

No need to reverse engineer debugserver, it's open source: https://github.com/llvm/llvm-project/tree/master/lldb/tools/debugserver. It sends out ptrace(PT_ATTACHEXC, …), which it can do because it has task_for_pid-allow. Connecting to it directly via a loopback or something would be interesting and something probably worth looking into.

[tbodt]

If it's possible to connect to debugserver over loopback it's probably also possible to do everything else AltStore does over loopback as well. Which would be a win for sure.

[OatmealDome]

No need to reverse engineer debugserver, it's open source: https://github.com/llvm/llvm-project/tree/master/lldb/tools/debugserver. It sends out ptrace(PT_ATTACHEXC, …), which it can do because it has task_for_pid-allow. Connecting to it directly via a loopback or something would be interesting and something probably worth looking into.

I had a look into this, since this hack being patched directly affects me too (I run a project which maintains a GameCube/Wii emulator port for iOS).

Here's how Xcode does debugging in case anyone is not aware:

1. Xcode connects to lockdownd (over TCP or usbmuxd) and uploads the iOS developer tools disk image.
2. lockdownd mounts the DMG at /Developer.
3. Xcode asks lockdownd to launch debugserver from the disk image.
4. lockdownd does so and returns a port to Xcode (seems to be in the 5xxxx range and increments with every subsequent launch unless a lower port is freed up?).
5. Xcode sends commands over TCP to debugserver to attach to or launch an app.
6. lldb is launched with the remote-ios platform and a connection is established.

Since an app can't start debugserver by itself and the port changes with each new debugserver instance, it's probably necessary to connect to lockdownd to start one. Unfortunately, sandboxed applications on the device are forbidden from communicating with lockdownd since iOS 7. Unless there's some sort of workaround for that (for example, maybe it's possible to port scan for the currently running debugserver? though one needs to be running in the first place), I'm not really sure if this can be done all on the device.

I think @osy86's idea to create an AltServer feature that spawns a debugserver instance for us (all the APIs necessary are already in libimobiledevice) might be the best way to proceed given the restrictions. Not sure how much effort this would be on @rileytestut's part though.

[osy]

Thanks for looking into it. And it’s unfortunate that you can’t do it all on-device.

[ntindle]

Looking towards previously mentioned Option 3 by osy, how difficult would the software side of this be? I pretty well understand the hardware side but am woefully under skilled to understand the difficulty for the software

[osy]

Since libimobiledevice supports it, I think you just need to boot Linux and run it. I personally haven’t looked into libimobiledevice myself so maybe there’s some additional difficulty there.

But such a “dongle” idea might also be used as an altstore resigning “on the go”. And maybe even trigger checkra1n as well. Business opportunity ;)

[hriaksnv5678]

But such a “dongle” idea might also be used as an altstore resigning “on the go”. And maybe even trigger checkra1n as well. Business opportunity ;)

wouldn't we need AltSever for Linux?

Probably would be a massive undertaking and not that useful.

eh, depends on how much power it needs. I mean if it's not that bad we could use something like a raspberry pi zero.

but can ios connect to USB devices in this way? If it could this would be an amazing thing, AltStore resigner, checkra1n, UTM helper and maybe even mini PC accessed through UTM (not as a VM) all in one.

[ntindle]

Before I design something, is it known if the NanoPi Duo2 would work?

[professorUnknown]

Indirect loop-backing is a long shot, but should be tested out. The type of vpn I am looking at is a app proxy provider https://developer.apple.com/documentation/networkextension/app_proxy_provider. One of the drawbacks is that the vpn only works with managed devices. The others vpn types aren’t useful since either doesn’t use tcp, or need a external server. A app that creates a local vpn and a app proxy provider is lockdown and luckily it is open-source https://github.com/confirmedcode/lockdown-ios. This if it works out can be also used to solve https://github.com/utmapp/UTM/issues/358 , and in turn make AltServer work on UTM. Edit 1: Get-task-allow lets you bypass app proxy provider being locked to managed devices. https://developer.apple.com/forums/thread/74464?answerId=217808022#217808022

[hriaksnv5678]

Before I design something, is it known if the NanoPi Duo2 would work?

Maybe, but to fit it into a usb, we could try the nano pi 3 but it also gets expensive. Maybe we make some hub like small thing, with a raspi or sth, as all NanoPis run only Ubuntu core which is a problem of its own, in some ways.

We should probably test this first before even trying to get it on hardware.

[rileytestut]

I think @osy86's idea to create an AltServer feature that spawns a debugserver instance for us (all the APIs necessary are already in libimobiledevice) might be the best way to proceed given the restrictions. Not sure how much effort this would be on @rileytestut's part though.

I agree that does seem to be the best way forward at the moment, which is unfortunate because it does come with several caveats — notably that you'll need to be on same WiFi as AltServer whenever you just want to use UTM/DolphiniOS.

As for the difficulty implementing it, the good news is I did manage to make a proof-of-concept implementation of AltServer launching apps on demand on an iOS device over WiFi using debugserver, so I can confirm it's possible and relatively straightforward. However, this assumed the iOS device had already been connected to Xcode and had DeveloperDiskImage.dmg installed + mounted, which is where the bigger problems lie (IMO). The exact DeveloperDiskImage.dmg mounted depends on the device's iOS version, and is not compatible between different iOS versions (excluding 0.0.x patch updates). Luckily libimobiledevice does support installing + mounting these developer images, so assuming AltServer was able to download the correct DeveloperDiskImage.dmg, it could take it from there.

Where is DeveloperDiskImage.dmg though? As far as I can tell, it is only available by downloading the entire Xcode .xip archive containing the relevant SDK, then extracting it from the archive. On the Mac we can somewhat sidestep this by just asking people to install Xcode from the Mac App Store before sideloading apps, but Windows is (of course) more tricky. Sure, AltServer could download 10GB+ for over an hour and somehow convince the user it's worth it, but then there's the next problem: Xcode archives are .xip, not .zip. I haven't looked into the format much myself, but it appears the exact underlying format has changed over time, so I'm not sure how hard it would be to reverse-engineer to work on Windows.

tl;dr: launching apps via debugserver over WiFi with AltServer is possible, and will allow JIT to work as long as you remain connected to the same WiFi as AltServer. To do so, however, you need to mount DeveloperDiskImage.dmg on the iOS device, which can either be obtained by installing Xcode from Mac App Store (Mac users), or by downloading the entire 10GB+ Xcode download distributed as a proprietary .xip archive — which may or may not be straightforward to unzip on Windows — and then repeating this process every single time you update your iOS device.

[rileytestut]

Edit 1: Get-task-allow lets you bypass app proxy provider being locked to managed devices. https://developer.apple.com/forums/thread/74464?answerId=217808022#217808022

Oh wow, I had no idea get-task-allow allowed for use of those NetworkExtension APIs without enrolling device in MDM, that's very useful to know!

I did just try it out, but it appears these APIs require the com.apple.developer.networking.networkextension entitlement, which is only available to paid developer accounts 🙁 So while this definitely could work as a solution for developers, it wouldn't help the vast majority of AltStore users wanting to sideload these apps without paying for a dev account.

[professorUnknown]

Where is DeveloperDiskImage.dmg though?

I was using mitmproxy on Xcode trying to find anisette data, and I noticed specific links for developer disk images, so try going for each device disk image go to Xcode preferences and then click the downloads tab then components then make sure you have a proxy running and press check and install now then you get the link.

[osy]

@rileytestut would it also allow for resigning with AltDaemon on non-jailbroken devices?

[professorUnknown]

I did just try it out, but it appears these APIs require the com.apple.developer.networking.networkextension entitlement, which is only available to paid developer accounts 🙁 So while this definitely could work as a solution for developers, it wouldn't help the vast majority of AltStore users wanting to sideload these apps without paying for a dev account.

No worries, I was researching just on that, and got a better idea. What about instead of hosting it on device making it vulnerable to patching we host it in a cloud application? The only issue was that cloud application has to be in the same WiFi network. Well not no more since while researching I came across https://github.com/corellium/usbfluxd it makes remote devices appear as local devices. Therefore connecting to it would be simple just as using https://developer.apple.com/documentation/network . The free cloud service I suggest is Heroku more information https://elements.heroku.com/buttons it has some limits https://devcenter.heroku.com/articles/limits, but if the cloud app is well written those limits could be satisfied. The cloud service supports docker and other important things https://devcenter.heroku.com/articles/exec. Though the UTM consumer has to setup the cloud app themselves.

[tbodt]

You need a computer to run usbfluxd on though.

[professorUnknown]

You need a computer to run usbfluxd on though.

We run it on server containers on custom Linux computers, and more details here https://www.heroku.com/dynos https://devcenter.heroku.com/articles/exec .

[saagarjha]

Xcode archives are .xip, not .zip. I haven't looked into the format much myself, but it appears the exact underlying format has changed over time, so I'm not sure how hard it would be to reverse-engineer to work on Windows.

Nobody really uses it, but Apple has an implementation you can presumably steal from. If you're willing to play it fast and loose you could have people enter links to DeveloperDiskImage "repositories" where people could share these instead of having to download the full thing–but I'm sure you have much more experience about the legality of this than I do :P

[tbodt]

No, you have to run it on a computer in the same wifi network as your device. At that point you might as well run the whole thing locally.

[professorUnknown]

Oh I got confused, but I wonder if there is a router setting like port forwarding that makes remote traffic look like local traffic?

[tbodt]

I doubt any consumer routers can do that

[saagarjha]

You might be able to come up with some convoluted bouncing/rerouting scheme, perhaps involving another computer to receive the traffic and send it around the local network, but at some point you really have to consider whether that's something that is reasonable to work on…

[Spidy123222]

You might be able to come up with some convoluted bouncing/rerouting scheme, perhaps involving another computer to receive the traffic and send it around the local network, but at some point you really have to consider whether that's something that is reasonable to work on…

I'd say that rerouting would be nice. Involving another computer to receive traffic and send it around a local network is possible.

For mimicking a Lan like professor said I think would require to have a server that Creates a VPN. Then the device with the app would connect to that vpn and the app would do a request to the server to get Debugserver from lockdownd. Sense Xcode can do remote debugging by Ip or in network.

This is a example of using it on heroku but I found out that a vpn Ian possible on their network but this is example of what I'm talking about a little.

How I got this idea is from services like Portmap.io or hamachi from them using a vpn to send lan data to their servers and host it.

A good test imo to see if such a thing would work if you don't have a Mac is to make a Minecraft Bedrock server (without port forwarding Minecraft itself) on a pc and create a vpn. Then all you would need to do is go on LtE if on a phone to test if you can connect to the lan or through its local ipv4 address.

[OatmealDome]

@rileytestut

so assuming AltServer was able to download the correct DeveloperDiskImage.dmg, it could take it from there.

What if it was possible for us to create our own DeveloperDiskImage.dmg with the bare minimum copied over from Apple's DMG to run a debugserver instance? The DMG itself is definitely not codesigned:

$ codesign -d -vvvv DeveloperDiskImage.dmg
DeveloperDiskImage.dmg: code object is not signed at all

I'm not 100% sure how the verification works since I haven't looked at it much, though it does appear to rely on AMFI's trustcache stuff from what I've read so far. There is a file in the root named .TrustCache, which appears to be an IMG4 file.

(And about the obvious question regarding legality: I think it might be OK to include a copy of debugserver from Apple's official DMGs, since the tool itself is open source and the only addition is Apple's own code signature, but IANAL...)

[saagarjha]

This isn't really my issue to administer, but I would personally prefer if we used this as a place to discuss technical information about changes in iOS 14 and how to work around them rather than pricing information about your VPN AWS service for a debugserver proxy that to my knowledge does not exist yet. Perhaps another place would be more relevant for that topic?

[Spidy123222]

Seems like that in Xcode12 beta 4 they changed debug server and now have to use (com.apple.debugserver.DVTSecureSocketProxy) Which is used in ios 13 but don't know how far down it goes on iOS versions. I wonder if this process is the same as regular Debugserver with ssl.

This is where I found out.

https://github.com/ios-control/ios-deploy/issues/469

https://github.com/libimobiledevice/libimobiledevice/issues/1004#issuecomment-674071552

[OatmealDome]

Seems like that in Xcode12 beta 4 they changed debug server and now have to use (com.apple.debugserver.DVTSecureSocketProxy) Which is used in ios 13 but don't know how far down it goes on iOS versions. I wonder if this process is the same as regular Debugserver with ssl.

This is where I found out.

ios-control/ios-deploy#469

libimobiledevice/libimobiledevice#1004 (comment)

This shouldn't really affect anything, since it's seemingly just a rename of the service with SSL layered on top. If an AltStore solution is introduced, then AltStore just needs to be updated to use the latest libimobiledevice.

[asdfugil]

How about not to JIT at all and use TCG Interpreter on iOS 14+ when not attached to debugger? (yes I know it is very slow)

[osy]

May be possible but currently QEMU doesn’t provide an easy way to disable JIT. So we would have to either add such a feature (non trivial work) or include two copies to QEMU (double the size) or two different releases (annoying to maintain). If there is a lot of interest it may be worth trying but I suspect not many people would care to use it without JIT.

[brunocastello]

Do we have any data from tests to see the real impact on performance without JIT?

[Spidy123222]

Do we have any data from tests to see the real impact on performance without JIT?

You can go to any emulator that has jit. For example DolphinIos when you run it to Cached interpreter the fps decreases a lot. Most emulators benefit from jit by a big amount.

A usual amount I see jit have is %40 increase in performance depending on how they use jit

[osy]

That doesn’t answer the question because different emulators benefit differently from JIT.

[brunocastello]

Hmmm, let me expand my question a little bit, because I want to understand the real impact: I have two VMs on my 2017 iPad Pro 12.9 inch, Windows 98 and Windows 3.11. Both do not have a custom JIT cache size, they are using the default. So are you saying that I could benefit from more performance by increasing the default if I wanted to, say, use Windows 98 or XP or newer? UTM says the default is 1/4 of the allocated memory size. From what I am reading here, without JIT any newer Windows like XP or 7 would really be a nightmare to use.

Thanks in advance if someone can answer me so I can have a more informed opinion about the issue that is being discussed here...

[Spidy123222]

Hmmm, let me expand my question a little bit, because I want to understand the real impact: I have two VMs on my 2017 iPad Pro 12.9 inch, Windows 98 and Windows 3.11. Both do not have a custom JIT cache size, they are using the default. So are you saying that I could benefit from more performance by increasing the default if I wanted to, say, use Windows 98 or XP or newer? UTM says the default is 1/4 of the allocated memory size. From what I am reading here, without JIT any newer Windows like XP or 7 would really be a nightmare to use.

Thanks in advance if someone can answer me so I can have a more informed opinion about the issue that is being discussed here...

That's essentially what I'm trying to say and yes using those windows would likely run that bad. Can you discuss this in the UTM discord I have the same name. https://discord.gg/fW3ctTV

This is for trying to find methods on getting jit working or alternatives. But if you want to discuss this we can go to the UTM discord.

[brunocastello]

Thank you, but I wouldn't be really helpful. I'm just an UTM user and my best collaboration to UTM was on the design side of things. I asked it because I wanted to understand the suggestion from @Assfugil above, because, as it stands we would apparently need to jailbreak iOS 14 to run UTM with JIT, since the benefits from using JIT are clear.

EDIT: I actually remembered this: https://developer.apple.com/documentation/hypervisor/apple_silicon Not sure if iOS devices will benefit from it in the future or if it will be just for Apple Silicon Macs. But apparently next gen iPads and First gen rumoured AS MacBook will share a similar AS version model. Technically speaking, if they do, we could use HVF as we currently use with QEMU on Intel macOS?

[saagarjha]

It'd have to be exposed to iOS applications to be usable as well. FWIW, I think I remember running the TCG interpreter once and it was like 2x slower or something like that on whatever I was doing. On a basic test on an A12Z with the interpreter with an Alpine kernel it seems to be slow but not unreasonable, so perhaps it could still be useful to people doing simple things.

[itselijahciali]

Do we have any data from tests to see the real impact on performance without JIT?

You can go to any emulator that has jit. For example DolphinIos when you run it to Cached interpreter the fps decreases a lot. Most emulators benefit from jit by a big amount.

A usual amount I see jit have is %40 increase in performance depending on how they use jit

Have we tested this with ARM-based guests? What would be the deficiency with a cached interpreter in that case? I know that the DolphiniOS emulates a PowerPC chip so that architectural difference would create a wider gap but I don’t think ARM-to-ARM would be as bad.

[osy]

Let's move discussion about TCG interpreter to https://github.com/utmapp/UTM/issues/465

[brunocastello]

Can the recently discovered blackbird vulnerability be used for something here in this issue? Assuming that the Checkra1n team can hopefully find it useable for A11 and newer devices since they have found it working for A9 and older and expect to have it too on A10.

[osy]

No that has no relevance here.