CloudKit Protobuf definition updates

[horrorho]

Our CloudKit protobuf definitions work well but are now quite ancient. There is a newer partial dump available obtained using Protod on Windows binaries. Only a small subset are recoverable using this tool.

I don't have access to them and I haven't tried, but using Protod on dumped iOS CloudKitDaemon framework binaries could possibly work wonders.

I now also have the reverse engineering chops to pull protobuf definitions kicking and screaming out of Windows binaries, although it takes more effort and the field names aren't often present.

[michaljirman]

I have tried to run the script on the CloudKitDeamon framework binary extracted from iOS 10 ipws but it seems like nothing was found.

[horrorho]

Thank you for testing and the feedback. Unfortunately binaries (decrypted) don't always have sufficient information to extract protobufs using this tool.

[michaljirman]

Yes, seems to be like that. The script is looking for '.proto' string which seems to be stripped out from the final binary. Btw, I've just extracted for my usage iOS (10.0.0.1) headers for various frameworks (e.g. CloudKit, CloudKitDeamon etc.). Let me know if you would also need them.

[michaljirman]

Hi horrorho, I've also tried running the script on some windows iCloud binaries (AOSKit.dll, ChunkingLibrary.dll, iCloud.dll, PCS.dll) without success, although you have managed to get at least partial dump. Just wondering if you could pls tell me on which dll did you successfully run the script?

[michaljirman]

Finally I had some luck with the iCloudDrive_main.dll.

[horrorho]

Also try iCloudPhotos_main.dll.

If I recall correctly Protod gave me a few read errors at times. Rather than delve into the code, I cheated my way around and split the dll into parts which I then processed individually. Just make sure there are no proto structures sitting on the split boundaries, they are text rich so are easily recognised.

[michaljirman]

Thanks, the splitting of dlls helped to fix the errors. I had a look at the errors and tried to fix the script. You can access it from the link below:- https://drive.google.com/file/d/0B4qEZ1jtaUaUVmZadEhoZGR0cWs/view?usp=sharing

[horrorho]

Excellent. Thank you.

[michaljirman]

Btw, I've started converting your java code in python3. Hope, it is alright if it is at some point also on Github as an open-source. Despite that, I expect I will get stuck at some point and will need guidance from someone on reversing binaries (iOS 10).

[horrorho]

@Jirmi Excellent. I'd almost given up on anyone porting to Python. Please ticket any questions/ issues you may have.

If you are serious I can take some time out to write up the process. Though I must warn you that English is not my first language, so it might get a bit convoluted. Then again it will probably be easier than deciphering my Java code.

Again if you are serious, as it will take up a fair amount of time, I can rewrite the necessary CloudKit API subset as a separate Java project using the newer protobuf definitions. I had been meaning to do it at some point. This should give you a foundation to work with. It will also give others a base for further research.

[michaljirman]

Thanks for the offer of help. I'm currently implementing the SRP part but have managed to block my account. Do you know pls how long it takes before it resets the counter of remaining attempts? I will be using for time-being another testing account but I wonder if it even gets unlocked :-)

[horrorho]

You can try to reset your iCloud password. Failing that you'll need to ring Apple. Attempts are limited to prevent brute force attacks and they don't regenerate. The rest of the API is more forgiving.

If you can, use an SRP-6a library. The verifier is of the type:

 x = SHA(s | SHA(I | ":" | P))

As stated in RFC 5054 Section 2.4.

BouncyCastle didn't offer what I needed so I ended up recoding it. I also added in a fail-safe where InflatableDonkey refuses to operate on accounts with limited attempts remaining. I didn't want to be responsible for locking accounts.

I've pushed a new build that has some additional logging statements.

If you add in the logging switch to logback.xml:

    <logger name="com.github.horrorho.inflatabledonkey.crypto.srp" level="trace" additivity="false">
        <appender-ref ref="STDOUT" />
    </logger> 

Then if you run InflatableDonkey a couple of times. It will dump out the SRP6-a vectors/ intermediate values which you can use to test with.

[horrorho]

I should probably also give you RFC 2945 Section 3 as a reference.

x = SHA(<salt> | SHA(<username> | ":" | <raw password>))

[michaljirman]

Thanks for the detailed info in the previous comments. The logging for the InflatableDonkey is very useful. It is also great it support proxy server configuration which also gives good inside of what's going on. As you suggested, I have used some SRP library although had to tweak it to get finally the right result (m1). A padding seems to cause sometimes some issues but it's working now.

[michaljirman]

Hi horrorho, just to provide some update. Currently, trying to understand the format of backupBagPassword which should be encoded as DER(asn1). Based on some comments from your project it should have following format:-

BackupBagPassword format

# SEQUENCE  (6)
#      UTF8STRING             String name
#      SET  (1)               Set<PrivateKey> privateKeys
#      SET  (1)               Set<TypeData> serviceKeyIDs
#      OCTET_STRING OPTIONAL  byte[] checksum
#     ....

Hex

# 62 82 07 15 30 82 07 11
# 0C 02 6B 73 
# 31 82 06 24 30 81 D5 04 40 86 52 D0 6F BC 68 4C 

Still trying to fully understand although I am not clear enough on tag value of the SET types. They seems to have tag value of 0x31 while I would expect 0x10 or 0x11 based on this resource found on internet.

Update:- Found different different resource where the tag values are clearer.

[horrorho]

0x31 is 0x11 with bit 6 set indicating a constructed type. It is referenced in A Layman's Guide to a Subset of ASN.1, BER, and DER. I read that document a while back and it also took me a few attempts to wrap my head around the whole thing. I used hate cryptography with a passion but I'm more comfortable with it now, though not an expert by any stretch of the imagination.

Unless it interests you, I would delegate as much as you can to a library. I left most of the dirty work to BouncyCastle.

Useful command:

openssl asn1parse -inform der -in FILE

Online decoders: http://lapo.it/asn1js/ https://holtstrom.com/michael/tools/asn1decoder.php

Apple: https://opensource.apple.com/source/Heimdal/Heimdal-453.40.10/lib/asn1/

I do have a private tool I created to rip the ASN1 templates that Apple uses from it's binaries. It's not polished and not easy to use, you need to provide it with the exact offset of the template. It's this tool that is responsible for the output you see in comments. I can upload it to GitHub if you like.

Also, it might be easier just to create a new issue for horrorho-jirmi discussions rather than polluting this issue.

[Aliasr]

Hi @michaljirman. I have implemented some new protobuf requests (such as RecordRetrieveChangesRequest (213) and QueryRetrieveRequest (220)) and now I am able to retrieve some additional records from other services.

But I haven't been able to download the assets included in those records, because InflatableDonkey's AuthorizeGetRequest seems to work only for the backup chunks. Do you know how could I download these assets?

Any guidance would be greatly appreciated. Thanks!