Signed App Installation

[PythEch]

So apparently there was a python implementation of libimobiledevice here: pymobiledevice

It is written in Python meaning that you won't even need to compile it as that is an interpreted langauge. Downside is, we need to use PyInstaller for Windows distribution because Python is not included in Windows, unlike Mac and Linux.

Anyways, if you don't want cross-language development (in other words, if you want just pure Java), we can look at that code and use it as reference to figure out how mobile installation_proxy is used. Since Python is a high-level language, it'll be easier to understand how things can be ported to Java.

I did those on a Windows machine, though it is relatively easier to get this done on Unix.

Requirements: Python 2.7 M2Crypto Construct

After the installation, head over to pymobiledevice and download the zip package. Extract pymobiledevice-master to C:\Python27 rename it to something like pymobiledevice otherwise Python will give syntax error. After that, fire up IDLE (Python GUI) from Start.

Now welcome to the Python Shell. You'd be surprised because all we need is 3 lines of code:

>>> from pymobiledevice import apps
>>> lockdown = apps.LockdownClient()
Connecting to device: 0aa4cc**********************************
Found pairing record for device 0aa4cc**********************************
>>> apps.mobile_install(lockdown, "C:\\Python27\\app.ipa")
Connecting to device: 0aa4cc**********************************
Connecting to device: 0aa4cc**********************************
Installing, C:\Python27\app.ipa: 5 % Complete
Installing, C:\Python27\app.ipa: 15 % Complete
Installing, C:\Python27\app.ipa: 20 % Complete
Installing, C:\Python27\app.ipa: 20 % Complete
Installing, C:\Python27\app.ipa: 30 % Complete
Installing, C:\Python27\app.ipa: 40 % Complete
Installing, C:\Python27\app.ipa: 50 % Complete
Installing, C:\Python27\app.ipa: 60 % Complete
Installing, C:\Python27\app.ipa: 70 % Complete
Installing, C:\Python27\app.ipa: 80 % Complete
Installing, C:\Python27\app.ipa: 90 % Complete
Installation Complete

Or you can just:

C:\Python27\pymobiledevice>apps.py --install=C:\Python27\app.ipa
Connecting to device: 0aa4cc**********************************
Found pairing record for device 0aa4cc**********************************
Connecting to device: 0aa4cc**********************************
Connecting to device: 0aa4cc**********************************
Installing, C:\Python27\app.ipa: 5 % Complete
Installing, C:\Python27\app.ipa: 15 % Complete
Installing, C:\Python27\app.ipa: 20 % Complete
Installing, C:\Python27\app.ipa: 20 % Complete
Installing, C:\Python27\app.ipa: 30 % Complete
Installing, C:\Python27\app.ipa: 40 % Complete
Installing, C:\Python27\app.ipa: 50 % Complete
Installing, C:\Python27\app.ipa: 60 % Complete
Installing, C:\Python27\app.ipa: 70 % Complete
Installing, C:\Python27\app.ipa: 80 % Complete
Installing, C:\Python27\app.ipa: 90 % Complete
Installation Complete

[PythEch]

So you asked

EDIT: Actually, I also need to be able to backup (but not restore) ipas through SSH. How can I do that without unsigning the ipa?

I had success with Archive command through afc. I highly doubt this is the command which is used by iTunes, iFunbox etc (I'll revEngineer later). For the development purposes, you can try this...

So when you archive an app you get signed zip in /var/mobile/Media/ApplicationArchives. I tested various apps and I can install those with iTunes, so you get indeed signed ipas. However, archiving an app without ClientOptions normally delete the app after the operation. So we need to set SkipUninstall to True. Not only that, if you don't use RemoveArchive after Archive, you'll get "Error: AlreadyArchived" when you try to Archive second time, even if you've deleted the zip.

The default pymobiledevice doesn't have Archive and RemoveArchive functions. Use my fork to get it work: https://github.com/PythEch/pymobiledevice

Archive:

from pymobiledevice import apps
apps.archive_app(apps.LockdownClient(), "com.ifttt.ifttt")

Remove Archive:

from pymobiledevice import apps
apps.remove_archive(apps.LockdownClient(), "com.ifttt.ifttt")

Install App:

from pymobiledevice import apps
apps.mobile_install(apps.LockdownClient(), "C:\\Python27\\com.ifttt.ifttt.zip")

[alexhulbert]

This is amazing! I was almost ready to abandon this libimobiledevice thing. It should be easy enough to get this working since its written in pure Python rather than bit and pieces of C, etc. Thanks for getting this to work, its much smoother than I expected. Now I know that this project is actually possible :).

[PythEch]

Thanks for adding my name to the credits!

I'll look for better ways of backing up ipas in weekend. P.S: I'm very happy with you being OK with Python lol

[PythEch]

I read a bit about the signing process of ipas. It turns out that we don't need anything special for backup process. IPA is a simple zip.

You can do this at home. Run iFunbox and check "Disable PNG Conversion" and "Disable bplist Conversion" from Options. For this example I'll be using "Authenticator.app". So copy ".app" folder, iTunesArtwork and finally iTunesMetadata.plist. Make directory called "Payload". Put Authenticator.app to Payload. So the filesystem should look like this:

|   iTunesArtwork
|   iTunesMetadata.plist
|   
\---Payload
    \---Authenticator.app
     ...

Then just zip everything and try to install it. It'll work.

To do that programmatically we should use "com.apple.mobile.house_arrest" service. I'll write a backup script in a few days.

[alexhulbert]

Cool! After a few minor adjustments, I've got the code so that it can run with Jython. From what I've heard, its as simple as creating a new org.python.util.PythonInterpreter, assigning it to a file, and executing python on it in the form of a string. It seems to good to be true, but I'll give it a shot and hope for the best. I've always liked python because its one of the few languages that seems to always work the way it should. I never really have those bugs where you know your code should work, but it doesn't for some reason or another. With things like C++, I always have to worry about how I'm compiling, what OS I'm on, etc. That's the beauty in cross-platform and directly interpreted languages. Anyway, I'll try to get it working and report back.

You said that we should use "com.apple.mobile.house_arrest." Why can't you just copy them from "~mobile/Applications?" Are there any advantages to using this service?

It also might be important to note that since the device will be in DFU, the actual device is going to be mounted to /mnt1 and /mnt2, while a second bare-bones ramdisk will be mounted to / using msftguy's SSH-RD. Will the services still work when executed from that alternate directory? The root partition will be in /mnt1 and the private partition (the one containing Applications) will be mounted to /mnt2. I don't believe its possible to chroot to the newly mounted directories, as I don't believe that command works properly with iOS devices.

EDIT: Would it be possible to build upon this? It should work on a regular iDevice, but I may have to change some paths around to get it to work on a DFU device.

[alexhulbert]

Just as I figured I would, I ran across a problem. Jython said it was compatible, but it turns out its not. Pymobiledevice runs on cpython, which is needed to access C. I need to use Jython, though, since I'm accessing the code through Jython's interpreter. I there any way I can get the required libraries from cpython to work under Jython's environment, or will I have to resort to something like JNI?

Here's the error I got at first:

  File "<stdin>", line 1, in <module>
  File "apps.py", line 1, in <module>
    from lockdown import LockdownClient
  File "lockdown.py", line 1, in <module>
    from plist_service import PlistService
  File "plist_service.py", line 3, in <module>
    import plistlib
ImportError: No module named plistlib

I decided to google plistlib, thinking that it was just some extra library I needed. I got it and added it to the folder. Then I got this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "apps.py", line 1, in <module>
    from lockdown import LockdownClient
  File "lockdown.py", line 1, in <module>
    from plist_service import PlistService
  File "plist_service.py", line 4, in <module>
    import ssl
ImportError: No module named ssl

Thinking that I just had to do this for a couple libraries, I went on to download ssl.py and copy that in, too. After doing that, I got this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "apps.py", line 1, in <module>
    from lockdown import LockdownClient
  File "lockdown.py", line 1, in <module>
    from plist_service import PlistService
  File "plist_service.py", line 4, in <module>
    import ssl
  File "ssl.py", line 359
    except socket_error as e:
                       ^
SyntaxError: mismatched input 'as' expecting COLON

I decided to look at the place I got ssl.py to see if I could figure out what I was even trying to do. I noticed they both came from websites about "CPython." Turns out that Jython and CPython are both implementations of Python. That being said, I have no clue if its even possible to use parts CPython and Jython all at the same time. Any ideas? If I'm correct, wouldn't that be like this: C/C++ <----> CPython <----> Jython <----> Java

I'm completely clueless here, but this may shed some light on how both of these work.

[PythEch]

Cool! After a few minor adjustments, I've got the code so that it can run with Jython. From what I've heard, its as simple as creating a new org.python.util.PythonInterpreter, assigning it to a file, and executing python on it in the form of a string. It seems to good to be true, but I'll give it a shot and hope for the best. I've always liked python because its one of the few languages that seems to always work the way it should. I never really have those bugs where you know your code should work, but it doesn't for some reason or another. With things like C++, I always have to worry about how I'm compiling, what OS I'm on, etc. That's the beauty in cross-platform and directly interpreted languages. Anyway, I'll try to get it working and report back.

That's great, I.. totally forgot about Jython. It'd be interesting to see it works

You said that we should use "com.apple.mobile.house_arrest." Why can't you just copy them from "~mobile/Applications?" Are there any advantages to using this service?

AFAIK, you can't just mount /var/mobile/Applications because Apple said no. The default afc service allows you r/w to /var/mobile/Media. Not just that, I think using house_arrest is easier because you don't need to walk in to weird paths, all you need is to put AppID as a parameter, and then you are in, allowed to read (and write to some files). house_arrest is implemented in pymobiledevice.

It also might be important to note that since the device will be in DFU, the actual device is going to be mounted to /mnt1 and /mnt2, while a second bare-bones ramdisk will be mounted to / using msftguy's SSH-RD. Will the services still work when executed from that alternate directory? The root partition will be in /mnt1 and the private partition (the one containing Applications) will be mounted to /mnt2. I don't believe its possible to chroot to the newly mounted directories, as I don't believe that command works properly with iOS devices.

To be honest, I don't understand the question. I don't think the services will work (because iOS isn't booted?), but since you have /mnt2 (/var/mobile) you don't even need to use Apple's limited services. You are root, you have all the control, so just read every folders in /var/mobile/Applications and package them nicely. Then, we also have backups of applications because we have now Documents and Library directories (though some applications store their information on Keychain).

EDIT: Would it be possible to build upon this? It should work on a regular iDevice, but I may have to change some paths around to get it to work on a DFU device.

I don't think you can use Bash scripts on non-jailbroken iDevice but, the only advantage of using that is, you don't need to write code (you have the code ready).

Just as I figured I would, I ran across a problem. Jython said it was compatible, but it turns out its not. Pymobiledevice runs on cpython, which is needed to access C. I need to use Jython, though, since I'm accessing the code through Jython's interpreter. I there any way I can get the required libraries from cpython to work under Jython's environment, or will I have to resort to something like JNI?

Like I said I totally forgot about Java implementations. That is a very good idea, I'll try to get those work. Though I can't say anything before I get my hands on.

Edit:

So the problem is, SSL module was introduced into CPython 2.6. You have installed Jython 2.5 (because it is the stable one), and SSL module uses Python 2.6 only features (except syntax was changed in 2.6).

So I tried that in Jython 2.7 beta 1, this time I have this error:

Jython 2.7b1 (default:ac42d59644e9, Feb 9 2013, 15:24:52)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_40
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymobiledevice import apps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pymobiledevice\apps.py", line 1, in <module>
    from lockdown import LockdownClient
  File "pymobiledevice\lockdown.py", line 1, in <module>
    from plist_service import PlistService
  File "pymobiledevice\plist_service.py", line 1, in <module>
    from usbmux import usbmux
  File "pymobiledevice\usbmux\usbmux.py", line 21, in <module>
    import socket, struct, select, sys
  File "C:\jython2.7b1\Lib\select.py", line 14, in <module>
    import Queue
  File "C:\jython2.7b1\Lib\Queue.py", line 8, in <module>
    from collections import deque
  File "C:\jython2.7b1\Lib\collections.py", line 12, in <module>
    import heapq as _heapq
  File "C:\jython2.7b1\Lib\encodings\__init__.py", line 85, in search_function
    norm_encoding = normalize_encoding(encoding)
  File "C:\jython2.7b1\Lib\encodings\__init__.py", line 69, in normalize_encodin
g
    return '_'.join(encoding.translate(_norm_encoding_map).split())
TypeError: translate() only works for 8-bit character strings

There are some weird things going on. Maybe it's too good to be true like you said. :( While, I agree that having a direct access to the pymobiledevice would be a great feature but, in worst case scenario we can make a python script to install, backup apps and upload/download files with parameters. e.g:

script.py --install=<path>
script.py --backup=<appid>
script.py --upload=<path>
script.py --download=<path>

[PythEch]

Signed backup script, run this on your computer:

https://ghostbin.com/paste/jao79

As for Jython problem, I solved those library problems but now I am asked to install M2Crypto and Construct libraries. I can't just install them straightforward:

building 'M2Crypto.__m2crypto' extension
error: I don't know how to find (much less run) SWIG on platform 'java'

So I searched some stackoverflow questions, it seems that http://jepp.sourceforge.net/ can be a better alternative for this problem.

[alexhulbert]

This should clear up all the confusion on DFU, etc.:

On all the older processors (Not A5), there's a bootrom exploit that's packaged with limera1n (I forgot what it's called) that can be used to upload a custom ramdisk to any A4 device while its in DFU mode. Variations of this exploit also make up most of the A4 tethered/semi-tethered jailbreaks you see (ex: opensn0w). Using a tool like this, you can upload a custom ramdisk containing some shell scripts, etc. and modify the root filesystem before the device has eve booted! The only problem is that this exploit only works for devices with A4 processors (like iPT 4G). I keep hearing rumors that evasi0n has one for A5+ in their alleged exploit arsenal, but I'm a little skeptical. Either way, its a nice tool to have, since manually backing everything up under DFU mode can take forever. As for the shell script you included, that won't go to waste :). I'm going to modify that so it can be used to backup data rather than apps.

Right now I'm looking at ca.py (the one that references M2Crypto) and it looks like its not doing anything special. From what I can see, it might be best to reimplement the cryptography part using something a little more cross-platform friendly. All the functions that are being used (and there's only 1 or 2 dozen) clearly state what their doing. For example, this code seems simple enough to port:

def generateRSAKey():
    return RSA.gen_key(2048, m2.RSA_F4)

def makePKey(key):
    pkey = EVP.PKey()
    pkey.assign_rsa(key)
    return pkey

The only two functions I'm worried about are makeRequest and makeCert, but those are heavily commented.

As for construct, I think that will work just fine with jep. It installs the same on different OSs, so that won't be a problem.

Anyway, I'm starting to port ca.py now. I'll let you know what happens.

[PythEch]

Hm, I was writing relatively a bit better version of that: https://ghostbin.com/paste/z9qjo

if backup_app_data:
        if "Library" in files:
            recursive_save(join(path, "Container", "Library"), join('.', "Library").replace("\\","/"))
        if "Documents" in files:
            recursive_save(join(path, "Container", "Documents"), join('.', "Documents").replace("\\","/"))

This should do everything you want. Keep in mind that some apps store their info on Keychain. Keychain can be extracted from iTunes backup. Mobilebackup is already included in pymobiledevice. If the device is jailbroken just download keychain-2.db from the device.

For Jython, what I did was installing both 2.5.2 and 2.7b1 versions. After that copy and paste ssl.py and plistlib.py from jython2.7b1/Lib/ to jython2.5.2/Lib.

Good luck with the port process :)

P.S: Also thanks for explaining everything detailed, it's fun to work on something together with a nice person like you

[alexhulbert]

I have one question before I start working on this thing: Will I be able to somehow include M2Crypto and Construct in Jepp natively? Or will it be different per OS?

[PythEch]

Well, I have the same question in my mind. I'm on a trip so I can't just boot Linux here, when I come back to home I can test it for you, but probably you'll have more time to test that.

[alexhulbert]

Okay! It seems pretty straightforward, so I'll try it out and post my results here later.

[PythEch]

Good luck, here is the improved version of the backup script (FYI, you've said bash script but this is a python script :D):

https://ghostbin.com/paste/j98jj

[alexhulbert]

Aha! Look at this!

[alexhulbert]

Ironically, it seems that now Jython might be better suited. I was reading about how to import packages using Jepp, and somebody said to use Jython. I read the exact opposite, though about 5 minutes earlier.

[PythEch]

If Jython worked, it'd be awesome because from what I've seen it generates .class files which can be used by any Java program easily. The problem is I can't get Jython to work... We might need someone else to fix this part of the project.

[alexhulbert]

I thought that too. Turns out that part of the project is depricated. All you have to do now is just use a function to execute a method. I tink its exec or something. Check this out!

[alexhulbert]

All I think you need to do is extract the packages to a temporary folder, and put from pkgutil import extend_path __path__ = extend_path(<tmppath>, "construct") at the top of every file that uses construct, where tmppath is the folder containing the construct package. M2Crypto should work too

[alexhulbert]

Aha! Another breakthrough! This is exactly what we're looking for! Its specific to packages located in the site-packages folder!

[PythEch]

I don't understand what you are doing (sorry :( ), since M2Crypto and Construct are platform-dependant (although cross-platform), in either way we'll need python to be executed. If the solution you have provided will need compiled libraries of M2Crypto and Construct, it is wasted effort because the same can be achieved by executing python script like I said before.

I think we need to find java equivalent of those libraries to succeed.

Edit: Lol, I've found this on wikipedia:

A port to Java is available on GitHub. Examples in Java, the ethernet header (layer 2)

That's great because you've said pymobiledevice doesn't do anything special with M2Crypto. Construct is no problem. If you can port ca.py, I'm very positive that we can burry this problem like forgotten stories of history.

[PythEch]

It might be a better idea to scrap ca.py and write java equivalent:

http://www.mayrhofer.eu.org/create-x509-certs-in-java

I believe this is the original ca.py file:

https://github.com/Hypernode/M2Crypto/blob/master/demo/x509/ca.py

I don't know why this is needed, there may be a workaround. When pymobiledevice paired my device for the first time, I got this alert:

http://www.macobserver.com/imgs/tmo_articles/ios7trustconfirm.jpg

When I delete .pymobiledevice folder (which is stored in %homepath%), thus deleting pairs, it tried to pair with the device again. Why I am saying this is, when using iFunbox, I don't get this alert on my device.

[alexhulbert]

Hi, I'm back! I just saw your message. I whipped this up in about 15 min., so its bound to have some bugs, but here's a prototype for the first function in ca.py The only thing I'm worried about is what I'm calling a string and what I'm calling a byte. I don't know how python handles the two, but it seems to be somewhat different than java.

public static String convertPKCS1toPKCS8pubKey(String raw_data) {
    byte[] subjectpublickeyrsa_start = Utils.fromHex("30819E300D06092A864886F70D010101050003818C00");
    raw_data = data.replace("-----BEGIN RSA PUBLIC KEY-----\n","");
    raw_data = data.replace("-----END RSA PUBLIC KEY-----","");
    byte[] data = base64.decode(data);
    byte[] starts = Utils.fromHex("30818902818100");
    if (data.startsWith(starts)) {
        byte[] newstarts = Utils.fromHex("308188028180");
        for (int i = 0; i < starts.length; i++) {
            data[i] = newstarts[i];
        }
    }
    byte[] newdata_raw = new byte[](subjectpublickeyrsa_start.length + data.length);
    for (i = 0; i < subjectpublickeyrsa_start.length + data.length; i++) {
        if (i < subjectpublickeyrsa_start.length) {
            newdata_raw[i] = subjectpublickeyrsa_start[i];
        } else {
            newdata_raw[i] = data[i - subjectpublickeyrsa_start.length];
        }
    }
    byte[] newdata = base64.encode(newdata_raw);
    String res = "-----BEGIN PUBLIC KEY-----\n";
    for (int i = 0; i < (newdata.length/64 + 1); i++) {
        for (int j = i*64; j <= (i+1*64); j++) {
            res += new String(newdata.toCharArray()[j]);
        }
        res += "\n";
    }
    res += "-----END PUBLIC KEY-----";
    return res;
}

EDIT: Come to think of it, do we even need to port functions that don't rely on M2Crypto? facepalm

[alexhulbert]

Also, I put the question out there just in case there's a way do use packages natively in Jython. Here's the question: http://stackoverflow.com/questions/20805980/reference-m2crypto-and-construct-from-within-jython

[PythEch]

Aha, I found this in lockdown.py:

        record = readHomeFile(HOMEFOLDER, "%s.plist" % self.identifier)
        if record:
            pair_record = plistlib.readPlistFromString(record)
            certPem = pair_record["HostCertificate"].data
            privateKeyPem = pair_record["HostPrivateKey"].data
            print "Found pairing record for device %s" % self.udid
        else:
            print "No pairing record found for device %s" % self.identifier
            return
        if False:
            if sys.platform == "win32":
                folder = os.environ["ALLUSERSPROFILE"] + "/Apple/Lockdown/"
            elif sys.platform == "darwin":
                folder = "/var/db/lockdown/"
            pair_record = plistlib.readPlist(folder + "%s.plist" % self.identifier)
            print "Using iTunes pair record"

Apparently it can use iTunes pair records but developer forgot to remove "return". I've checked that folder and yes, they do exist. We can use iTunes pair records and scrap ca.py, but this will break Linux support (unless Apple starts to think having iTunes on Linux is a good idea). But that doesn't matter, openssl is installed on most Linux distros so we can ShellExecute a command to generate CA certificates. I think I can handle this side of the project when I come back to home. Right now I'm editing the file to use iTunes pair records instead of generating its own.

So what I thought is true, iFunbox and several other tools use iTunes records.

[alexhulbert]

That's awesome! I can't believe it fit together so perfectly! I'll work on my side of the project in the mean time.

[PythEch]

My fork is online again with before said changes:

https://github.com/PythEch/pymobiledevice

Edit: Just confirmed that iFunbox is using iTunes' pair: http://i.imgur.com/fzYurrp.png

Also it's funny to see that iTunes will give error 0xE8000003 forever when you delete folder "%AllUsersProfile%\Apple\Lockdown". Apple forgot to use mkdir? haha

[alexhulbert]

I tried to sent a comment about an hour ago, but I guess it didn't work. Anyway, everything works except for sockets. I get this error:

  File "<stdin>", line 1, in <module>
  File "pymobiledevice\lockdown.py", line 21, in __init__
    self.c = PlistService(62078,udid)
  File "pymobiledevice\plist_service.py", line 12, in __init__
    self.connect(udid)
  File "pymobiledevice\plist_service.py", line 15, in connect
    mux = usbmux.USBMux()
  File "pymobiledevice\usbmux\usbmux.py", line 222, in __init__
    self.listener.listen()
  File "pymobiledevice\usbmux\usbmux.py", line 222, in __init__
    self.listener.listen()
  File "pymobiledevice\usbmux\usbmux.py", line 191, in listen
    ret = self._exchange(self.proto.TYPE_LISTEN)
  File "pymobiledevice\usbmux\usbmux.py", line 185, in _exchange
    recvtag, data = self._getreply()
  File "pymobiledevice\usbmux\usbmux.py", line 164, in _getreply
    resp, tag, data = self.proto.getpacket()
  File "pymobiledevice\usbmux\usbmux.py", line 106, in getpacket
    dlen = self.socket.recv(4)
  File "pymobiledevice\usbmux\usbmux.py", line 51, in recv
    raise MuxError("socket connection broken")
pymobiledevice.usbmux.usbmux.MuxError: socket connection broken

I'll send you the patched python files tomorrow. You have to modify some stuff to work with jython

[PythEch]

That's weird, I didn't encounter that error. I guess it was temporary and the changes I made has nothing to do with usbmux. I'm waiting for your patch.

Edit:

Are you talking about jython? If so yes, I have the same problem:

Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_40
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymobiledevice import apps
>>> apps.LockdownClient()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pymobiledevice\lockdown.py", line 21, in __init__
    self.c = PlistService(62078,udid)
  File "pymobiledevice\plist_service.py", line 12, in __init__
    self.connect(udid)
  File "pymobiledevice\plist_service.py", line 15, in connect
    mux = usbmux.USBMux()
  File "pymobiledevice\usbmux\usbmux.py", line 220, in __init__
    self.listener = MuxConnection(socketpath, BinaryProtocol)
  File "pymobiledevice\usbmux\usbmux.py", line 157, in __init__
    self.socket = SafeStreamSocket(address, family)
  File "pymobiledevice\usbmux\usbmux.py", line 37, in __init__
    self.sock = socket.socket(family, socket.SOCK_STREAM)
  File "C:\jython2.5.2\Lib\socket.py", line 1200, in __init__
    _sock = _realsocket(family, type, proto)
  File "C:\jython2.5.2\Lib\socket.py", line 586, in _realsocket
    assert family in (AF_INET, AF_INET6), "Only AF_INET and AF_INET6 sockets are
 currently supported on jython"
AssertionError: Only AF_INET and AF_INET6 sockets are currently supported on jyt
hon

and now this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pymobiledevice\lockdown.py", line 21, in __init__
    self.c = PlistService(62078,udid)
  File "pymobiledevice\plist_service.py", line 12, in __init__
    self.connect(udid)
  File "pymobiledevice\plist_service.py", line 15, in connect
    mux = usbmux.USBMux()
  File "pymobiledevice\usbmux\usbmux.py", line 218, in __init__
    self.listener.listen()
  File "pymobiledevice\usbmux\usbmux.py", line 218, in __init__
    self.listener.listen()
  File "pymobiledevice\usbmux\usbmux.py", line 187, in listen
    ret = self._exchange(self.proto.TYPE_LISTEN)
  File "pymobiledevice\usbmux\usbmux.py", line 181, in _exchange
    recvtag, data = self._getreply()
  File "pymobiledevice\usbmux\usbmux.py", line 160, in _getreply
    resp, tag, data = self.proto.getpacket()
  File "pymobiledevice\usbmux\usbmux.py", line 106, in getpacket
    dlen = self.socket.recv(4)
  File "pymobiledevice\usbmux\usbmux.py", line 51, in recv
    raise MuxError("socket connection broken")
pymobiledevice.usbmux.usbmux.MuxError: socket connection broken

[PythEch]

I think Jython gives us too much headaches. Let's keep it simple and distribute the python part with PyInstaller. Even Tinyumbrella has two seperate versions which is written in pure Java.

From PyInstaller homepage:

PyInstaller is a program that converts (packages) Python programs into stand-alone executables, under Windows, Linux, Mac OS X, Solaris and AIX. ... The main goal of PyInstaller is to be compatible with 3rd-party packages out-of-the-box. This means that, with PyInstaller, all the required tricks to make external packages work are already integrated within PyInstaller itself so that there is no user intervention required. You'll never be required to look for tricks in wikis and apply custom modification to your files or your setup scripts. As an example, libraries like PyQt, Django or matplotlib are fully supported, without having to handle plugins or external data files manually.

Using

pyinstaller --onefile script.py

just works. If it doesn't work try

C:\Python27\Scripts\pyinstaller.exe --onefile script.py

[alexhulbert]

How can I run functions, though? I think it would be better to get Jython working. I tested everything else and it seems to work. Can you add me as a contributor to your PythEch/pymobiledevice? I can add the necessary code changes to get Jython running. From there, we just have to get socket.recv() working, then we're golden. I'm going to be using most of this program, so I'm not sure its possible to execute functions, get return values, and then execute more functions based on them just with PyInstaller. Plus, we've gotten this far, and it seems easy enough to fix whatever socket problem is happening. If not, I could just find a port of usbmux for Java and reference that. I know the guy didn't create usbmux. I think all that specific script is doing is tunnelling [iDevice ip]:62078 to 127.0.0.1:62078. I'll try to see if it does anything more.

[PythEch]

Done, I added you as collaborator.

To run functions we can write some piece of code so that executable will accept parameters. e.g:

script.exe --install=<ipaPath>
./script --backup=<appid>

etc. Anyways, forget what I said. You're right, if you've got socket library working there is nothing to worry about.

[alexhulbert]

I think you implemented the iTunes thing wrong. It gets stuck at "Found pairing..." It just keeps repairing to the device. I think thats because you forgot to put in pair_record = plistlib.readPlist(folder + "%s.plist" % self.identifier) I'll try that locally and see if it works. Other than that, there are a few changes you need to make (like calling Java instead of python) to get everything to fit together nicely.

[PythEch]

Hmm, are you sure? Because it's working fine here, also what's wrong with:

        record = readHomeFile(HOMEFOLDER, "%s.plist" % self.identifier)
        if record:
            pair_record = plistlib.readPlistFromString(record)

Waiting for your patches.

Edit: I think I've found the problem:

if sys.platform == "win32":
    HOMEFOLDER = os.environ["ALLUSERSPROFILE"] + "/Apple/Lockdown/"
elif sys.platform == "darwin":
    HOMEFOLDER = "/var/db/lockdown/"

Jython sets sys.platform as something like "java" so it fails to recognize OS.

I've pushed a commit

[alexhulbert]

I figured out the issue! You need to use one function from ca.py: the one that was manually added. Good thing it doesn't require M2Crypto

EDIT: That wasn't the issue. It was close, but the issue is much more in-depth. Also, iTunes pairing isn't even working in regular Python :/

[PythEch]

Great!, wonder what was the issue

[alexhulbert]

OK. Turns out there's a lot more to pairing than I though. I gave up, so about an hour or two ago I figured I'd try again. I ended up rewriting a lot of the pairing and validate_pairing stuff. I successfully got it to work on iOS 6. iOS 7 doesn't seem to be working. So far, I have counted a total of 12 different errors I'm getting when connecting with my iOS 7.1 beta 1 iPad. It could just be the iPad itself, as it has a number of other issues I plan to fix later. Could you try it on what ever devices you have? Also, about 1 minute after initializing a lockdown client, I get this when attempting to run any function:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pymobiledevice\apps.py", line 63, in mobile_install
    afc = AFCClient(lockdown)
  File "pymobiledevice\afc.py", line 129, in __init__
    self.service = self.lockdown.startService(serviceName)
  File "pymobiledevice\lockdown.py", line 158, in startService
    self.c.sendPlist({"Request": "StartService", "Service": name})
  File "pymobiledevice\plist_service.py", line 86, in sendPlist
    self.send(l + payload)
  File "pymobiledevice\plist_service.py", line 46, in send
    return self.s.send(data)
  File "C:\Python27\Lib\ssl.py", line 198, in send
    v = self._sslobj.write(data)
ssl.SSLError: [Errno 1] _ssl.c:1309: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry

I don't know what thats about.

It still doesn't work with Jython. This is all stock Python 2.7. As for the socket thing, I'm going to go on the Jython IRC channel on freenode. I might be able to get some help there.

Sorry about all the bad news, but at least device pairing is (partially) working.

[PythEch]

Sorry, I can't help with the issue because it's still working for me :( Is it possible that we can communicate via teamviewer, IRC, skype etc. (anything that you prefer)?

So this is the latest commit that you've pushed (but I removed java-only things):

https://mega.co.nz/#!5QdXDZxD!SP3Bb00HV-CylO92TLKAL1HLvrVCXRQvn6RsGkmwr24

Before that this is also working for me: https://github.com/PythEch/pymobiledevice/tree/bc9a49b5a43227cc0f3a88f7ee490a8b93be26e1

[alexhulbert]

I'm setting up a online chat server on my website. I plan to add features like screen sharing to it. It should be up at chat.alexhulbert.com within a couple days.

At first, I thought it worked too. It looks like it works, but fails when you actually try to install an app, run "ls," or actually preform an action. Try this and let me know if it works on iOS7 (It doesn't for me):

afc.AFCShell().do_ls("")

AFCShell automatically creates a lockdown instance if no arguments are passed to it, and do_ls requires a couple different services, so this should be a good indicator of whether everything is working or not.

---

Here's my interpretation of what's happening for iOS7:

For some reason, validate_pairing is returning true even if the device hasn't been paired. Then, when I run "pair()," It asks me if I trust the computer. Even if I say to trust it, Pair() then returns False with the response message saying something about the device being password protected (my device isn't, though). Trying to run any functions after that results in a number of different errors. The only thing I can do is rerun lok = lockdown.LockdownClient() Like I said, this may very well just be the iPad itself.

[PythEch]

Yes, it works:

http://imgur.com/ixXh3JB

I've even finished the backup/restore, download/upload script. Can you please download this and try a command like:

libapp.exe -b 0 com.designshed.alienblue "apps/Alien Blue.ipa"

If that works, the problem may be faulty installation of libraries, but I am out of ideas. Also I know, its name sucks. I had no idea so I just pushed a commit (maybe I should've named it pydeviceinstaller).

Edit: Aha I think I've found the problem, do you have the latest version of iTunes??? Pairing has changed since iOS 7, but you are using an older version of iTunes I believe. Maybe for that reason we shouldn't use iTunes pairing.

[alexhulbert]

Yup, its the iPad. I tried it with "com.google.chrome.ios," an app I know I have, and it said Unknown identifier 'com.google.chrome.ios' Sorry it took me so long. Are you having the issue where it will timeout after about a minute? I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pymobiledevice\apps.py", line 63, in mobile_install
    afc = AFCClient(lockdown)
  File "pymobiledevice\afc.py", line 129, in __init__
    self.service = self.lockdown.startService(serviceName)
  File "pymobiledevice\lockdown.py", line 158, in startService
    self.c.sendPlist({"Request": "StartService", "Service": name})
  File "pymobiledevice\plist_service.py", line 86, in sendPlist
    self.send(l + payload)
  File "pymobiledevice\plist_service.py", line 46, in send
    return self.s.send(data)
  File "C:\Python27\Lib\ssl.py", line 198, in send
    v = self._sslobj.write(data)
ssl.SSLError: [Errno 1] _ssl.c:1309: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry

BTW, libapp isn't such a bad name. At least it doesn't sound like a phishing site :) "Icew1nd" is kind of sketchy-sounding.

[PythEch]

I really tried hard to get some sort of error like you have but as I said before every function is working perfect for me. Without some kind of instant messaging, it's really hard to figure our what's wrong. I'm waiting for your webchat.

Also you ddn't say anything about what version of itunes you have

[alexhulbert]

Sorry it took me a week, but I've managed to figure out of all the issues:

1. (See this) I was using evasi0n7 with an iPad 2 WiFi at 7.1 beta 1. The first version of evasi0n wasn't compatible with either the iPad 2 WiFi or the beta firmware, so there were some problems there. Also, I was using an outdated version of iTunes. I'll make a note to include an into saying that you need to have the latest version of iTunes. I'm pretty sure the Socket/SSL error was caused by a flimsy cable :).
2. (See this) I figured out the reason Jython's not receiving data. I have no clue why, but any time I try to receive INET sockets, Jython crashes with an "out of memory" error. I emailed the guy who manages sockets in Jython about it, and have yet to receive a response.
3. (See this) Jython is not compatible with UNIX sockets. This can easily be solved by referencing a Java library for communicating through UNIX sockets. You can see that here. It seems simple enough. All I have to do is reference it in the recv and send methods.

---

I do have one question. The device you were using: what processor did it have? If it was A5, then we're good. If not, it might be good to get a variety of "beta testers" to test it with iOS7/A5 processor. Everybody likes beta testing, so that should be easy too.

[PythEch]

Great! I thought you've given up with Jython weirdness haha :) I have iPhone 4 (5.1.1 JB), iPhone 4S (7.0.4 evasi0n7 1.01), and iPhone 5 (7.0.4 stock) in home. They all work with pymobiledevice (or at least, libapp.exe) but since I don't own the iPhone 4 (4S is mine), it's too much of hassle to backup/restore everything on a jailbroken device, especially with a heavily customized one like my brother has.I'm sure I can convince him when we really need it. But it seems that you already have an A4. We may need a working iPad though :)

[alexhulbert]

He still hasn't responded. I know he'll respond at some point because I asked him about the UNIX socket issue earlier. My iPod Touch 4G just stopped working a couple weeks ago. I bought a 5th generation, but that's A5 (it took me weeks to find one with iOS 6). My brother also has an A4 device. I also need an A4 for completing the iCloud portion of Icew1nd (I need to sniff HTTPS traffic). I might be able to convince him to restore it if I install bl4ck0ut7 on it afterwords. I know someone who has a 4G he's not using, but he can't seem to find it. I'm going to downgrade the iPad to 7.0.4 this week and see if that clears up the remaining issues. I'll continue to search for a A4 device. Either way, I'm sure I can find somebody.

[alexhulbert]

Sorry about that

[alexhulbert]

Check this out! It has a great explanation of what lockdown.py and usbmux.py are doing. Complete with handy exemplary plists. This should be perfect if we need to rewrite some stuff for iOS 7 (which I doubt we will need to, but just in case). This will also come in handy if we have to reimplement ca.py using openssl (for linux) or rewrite the UNIX socket part in Java.

[alexhulbert]

I saw the commit you made. I don't believe that that's the error. SOCK_DGRAM and SOCK_STREAM are still dgram and stream sockets, their constant values are just switched for some reason. Either way, I'll test it out and let you know if it works.

[PythEch]

The change makes lockdown.py give error:

C:\jython2.5.3>jython libapp.py
Error: Pairing failed!
(-1, 'Unmapped exception: java.net.PortUnreachableException')
Error: Unexpected error with mobile installation_proxy!
libapp instance has no attribute 'lockdown'

And with the original lockdown.py, it freezes (before https://github.com/PythEch/pymobiledevice/commit/2dc7c976bad7aac0ed1c775dba6f9833ea9684c3)

EDIT: Yes, you're right. Reverted the commit

[PythEch]

Something I noticed today...

In BinaryProtocol:

def sendpacket(self, req, tag, payload={}):
    payload = self._pack(req, payload)
    if self.connected:
        raise MuxError("Mux is connected, cannot issue control packets")
    length = 16 + len(payload)
    data = struct.pack("IIII", length, self.VERSION, req, tag) + payload
    print data.encode("base64") # added this
    self.socket.send(data)

In CPython, I get:

cwEAAAEAAAAIAAAAAQAAADw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+Cjwh
RE9DVFlQRSBwbGlzdCBQVUJMSUMgIi0vL0FwcGxlLy9EVEQgUExJU1QgMS4wLy9FTiIgImh0dHA6
Ly93d3cuYXBwbGUuY29tL0RURHMvUHJvcGVydHlMaXN0LTEuMC5kdGQiPgo8cGxpc3QgdmVyc2lv
bj0iMS4wIj4KPGRpY3Q+Cgk8a2V5PkNsaWVudFZlcnNpb25TdHJpbmc8L2tleT4KCTxzdHJpbmc+
dXNibXV4LnB5IGJ5IG1hcmNhbjwvc3RyaW5nPgoJPGtleT5NZXNzYWdlVHlwZTwva2V5PgoJPHN0
cmluZz5MaXN0ZW48L3N0cmluZz4KCTxrZXk+UHJvZ05hbWU8L2tleT4KCTxzdHJpbmc+dGNwcmVs
YXk8L3N0cmluZz4KPC9kaWN0Pgo8L3BsaXN0Pgo=

In Jython (also worth mentioning that I forced using PlistProtocol instead of BinaryProtocol), I get:

C:\jython2.5.3>jython Lib\pymobiledevice\usbmux\usbmux.py
AAABcwAAAAEAAAAIAAAAATw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+Cjwh
RE9DVFlQRSBwbGlzdCBQVUJMSUMgIi0vL0FwcGxlLy9EVEQgUExJU1QgMS4wLy9FTiIgImh0dHA6
Ly93d3cuYXBwbGUuY29tL0RURHMvUHJvcGVydHlMaXN0LTEuMC5kdGQiPgo8cGxpc3QgdmVyc2lv
bj0iMS4wIj4KPGRpY3Q+Cgk8a2V5PkNsaWVudFZlcnNpb25TdHJpbmc8L2tleT4KCTxzdHJpbmc+
dXNibXV4LnB5IGJ5IG1hcmNhbjwvc3RyaW5nPgoJPGtleT5NZXNzYWdlVHlwZTwva2V5PgoJPHN0
cmluZz5MaXN0ZW48L3N0cmluZz4KCTxrZXk+UHJvZ05hbWU8L2tleT4KCTxzdHJpbmc+dGNwcmVs
YXk8L3N0cmluZz4KPC9kaWN0Pgo8L3BsaXN0Pgo=
Traceback (most recent call last):
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 227, in 
    mux = USBMux()
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 216, in __init__
    self.listener.listen()
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 192, in listen
    ret = self._exchange(self.proto.TYPE_LISTEN)
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 186, in _exchange
    recvtag, data = self._getreply()
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 165, in _getreply
    resp, tag, data = self.proto.getpacket()
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 144, in getpacket
    resp, tag, payload = BinaryProtocol.getpacket(self)
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 108, in getpacket
    dlen = self.socket.recv(4)
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 50, in recv
    chunk = self.sock.recv(size-len(msg))
  File "C:\jython2.5.3\Lib\socket.py", line 1199, in recv
    raise _map_exception(jlx)
socket.error: (10054, 'Software caused connection abort')

For some reason, the first 22 base64 characters are different. This means while payloads are the same, however

struct.pack("IIII", length, self.VERSION, req, tag)

gives different results... I checked the parameters, yes they're the same. Any idea why struct.pack() gives different results? I guess this is a bug with Jython itself.

Could you please check if you can reproduce the bug?

CPython:

>>> import struct, sys
>>> struct.pack("IIII", 371, 1, 8, 1)
's\x01\x00\x00\x01\x00\x00\x00\x08\x00\x00\x00\x01\x00\x00\x00'
>>> sys.byteorder
'little'

Jython:

>>> struct.pack("IIII", 371, 1, 8, 1)
'\x00\x00\x01s\x00\x00\x00\x01\x00\x00\x00\x08\x00\x00\x00\x01'
>>> sys.byteorder
'big'

Anyways, it can be fixed by:

data = struct.pack("<IIII", length, self.VERSION, req, tag) + payload

This doesn't fix the issue, also I'm not sure if this is a bug because it may be a compatibality layer for Java. However, after these changes: https://ghostbin.com/paste/te7x6

I can finally get something to work. Jython:

C:\jython2.5.3>jython Lib\pymobiledevice\usbmux\usbmux.py
4
290
Waiting for devices...
Traceback (most recent call last):
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 229, in <module>
    mux.process(0.1)
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 220, in process
    self.listener.process(timeout)
  File "Lib\pymobiledevice\usbmux\usbmux.py", line 197, in process
    rlo, wlo, xlo = select.select([self.socket.sock], [], [self.socket.sock], ti
meout)
  File "C:\jython2.5.3\Lib\select.py", line 225, in native_select
    pobj.register(fd, POLLIN)
  File "C:\jython2.5.3\Lib\select.py", line 106, in register
    raise _map_exception(jlx)
select.error: (20000, 'socket must be in non-blocking mode')

CPython:

4
290
Waiting for devices...
4
614
Devices:
<MuxDevice: ID 68 ProdID 0x12a0 Serial '*************************************' Location 0x0>

EDIT: See the latest commit. It started to work :) Needs some changes with lockdown.py though. Lesson learned: It's hard to debug Python :(

None of the previous versions of lockdown.py work. I'll try to get it work tomorrow. Let me know if usbmux.py works for you. Can't believe it was just a little-big endian problem.

EDIT 2: At the moment, from what I've seen, there is a problem with SSL in plist_service.py:

C:\jython2.5.3>jython Lib\pymobiledevice\lockdown.py
Connecting to device: ****************************************
Found pairing record for device ****************************************
Traceback (most recent call last):
  File "Lib\pymobiledevice\lockdown.py", line 151, in <module>
    l = LockdownClient()
  File "Lib\pymobiledevice\lockdown.py", line 46, in __init__
    if not self.validate_pairing():
  File "Lib\pymobiledevice\lockdown.py", line 113, in validate_pairing
    self.c.ssl_start(sslfile, sslfile)
  File "C:\jython2.5.3\Lib\pymobiledevice\plist_service.py", line 89, in ssl_sta
rt
    self.s = ssl.wrap_socket(self.s, keyfile, certfile)
  File "C:\jython2.5.3\Lib\socket.py", line 1761, in ssl
    return _realssl(sock, keyfile, certfile)
  File "C:\jython2.5.3\Lib\socket.py", line 1706, in __init__
    self.ssl_sock = self._make_ssl_socket(plain_sock)
  File "C:\jython2.5.3\Lib\socket.py", line 1720, in _make_ssl_socket
    ssl_socket.startHandshake()
        at sun.security.ssl.HandshakeMessage$CertificateMsg.<init>(Unknown Sourc
e)
        at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
        at sun.security.ssl.Handshaker.processLoop(Unknown Source)
        at sun.security.ssl.Handshaker.process_record(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source
)
        at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)

javax.net.ssl.SSLProtocolException: javax.net.ssl.SSLProtocolException: Empty is
suer DN not allowed in X509Certificates