iOS 11 beta

[ItsASmallWorld]

I'm able to choose an iOS 11 snapshots and it looks like the files in the snapshot are detected however the files are not downloaded. I'll e-mail logs.

[ghost]

I just tried this out myself. It does detect that there's an IOS 11 backup but after trying to download it this is all I get without any errors:

Enter selection (multiple values accepted, leave blank to select all, q to quit):
: 4

Selected:
DEVICE: **************************************** iPhone6,1 N51AP
  SNAPSHOT:   13 MB iPhone (11.0)  2017-06-06T06:10:48.477Z

Retrieving snapshot:   13 MB iPhone (11.0)  2017-06-06T06:10:48.477Z

MBP-2:target User$ 

Let me know if there is any way I can help out, maybe by providing an account with a backup.

[horrorho]

Heya. Thank you for the input. @ItsASmallWorld I'll certainly take a look at the logs, probably mid week now. @Yaldo425 If I can't figure things out, I may take you up on that offer! But let me examine the log files to see if there is a simple fix.

[michaljirman]

Hi @horrorho, by looking at some logs I believe a protectionInfo and protectionInfoTag are not sent in response anymore or in different format.

message Record { ...
optional ProtectionInfo protectionInfo = 13; ....

Additionally, I've noticed that a new field 13 was added under field 7 (RecordField).

[horrorho]

@Jirmi I only have partial logging available but it does confirm the absence of protection info data. Encrypted fields are still present so keys are still required and delivered in some manner.

@Yaldo425 Would you kindly contact me at horrorhorrorho@gmail.com? It's not an obviously trivial fix based on the logging information I have. I would like to take you up on your earlier offer if it still stands. Thank you!

[ghost]

@horrorho No problem. Will have the credentials sent within an hour! I'm going to try to go for another backup that is slightly bigger than just 13MB.

[horrorho]

@Yaldo425 I won't be back home from work until late evening and it's only midday now. So no rush hey. Thank you so much!

[ghost]

@horrorho I just sent the credentials. Thanks again!

[horrorho]

@Yaldo425 Ok! I've figured out this particular issue. I've partly patched InflatableDonkey and minus a few quirks it's working again. Hopefully I can complete it all over the weekend and have it pushed by next week. I'll continue to test it with your backup during that time if that's ok.

In short there is a new ProtectionInfo field in Zone. I don't know what it's called, so it's protectionInfoX for now. I'll name it once I have additional sources to work with (or maybe I'll forget and it will forever be that way).

message Zone {
  optional RecordZoneIdentifier zoneIdentifier = 1;
  optional ProtectionInfo protectionInfo = 3;
  optional ProtectionInfo protectionInfoX = 6;
}

In addition not all Records have ProtectionInfo fields, in which case the parent Zone ProtectionInfo is used. I don't think it's any more complicated than that. The usual caveats apply and there maybe additional changes that haven't surfaced yet.

Oh. There's also a change in the way asset domain fields are handled but that should be simple fix.

:smirk_cat:

[ghost]

@horrorho Nice job! You can test it with the backup for however long you want, I don't mind!

[michaljirman]

@horrorho Thanks! Btw. the new protectionInfo name is just "recordProtectionInfo".

[horrorho]

@Jirmi Oh yes, that would make sense hey. I'll amend my code.

[horrorho]

Update. Ok! Good and potentially bad news.

Good: I've fully patched InflatableDonkey. I'll test and have it out by the weekend if all is well.

Bad: Some files are compressed with bvx2. Notable sqlite database files. I'll take time out later in the week to examine it in detail. It's entirely possible the archives are encrypted and that the keys are not available in the backup itself.

[michaljirman]

@horrorho Thanks for the info, you are right. I can also confirm the bvx2 compression on my backup files.

[michaljirman]

@horrorho I've done some additional testing and I was able to simply decompress a sqlite file without using any keys, just by running lzfse command line tool.

./lzfse -decode compressed.db decompressed.db

[horrorho]

@Jirmi Hi. Thank you. I was playing with with lzfse last night and reached the same conclusions. Although I did encounter a malloc: Cannot allocate memory error on a particular file.

The issue is that there are presently no Java lzfse implementations, or at least none that I could find. We could ask non-Apple platform users to supply their own native binaries and InflatableDonkey can use those. An alternative solution is that we code our own lzfse decompressor.

I don't like either solution. However and possibly foolishly I did commence writing my own lzfse decompressor last night. I admit I've never coded a compressor before and I know more about cat butts than I do compression theory. I'll give it a few days and see if it's viable.

Otherwise we'll go for the native binary approach, although I can see some of the more casual Windows InflatableDonkey users being inconvenienced by this.

If anyone has an other ideas or solutions, I would like to hear them!

[michaljirman]

I can't also find anything for Java. For python there is module pylzfse which provides python bindings for LZFSE from the reference implementation. Not sure if the same thing would be possible with Java (JNI) but I would guess it could also work and would be surely less amount of work then writing compressor from scratch :-), it must surely be quite complex thing to develop. On the other side, I'm quite excited to see your implementation in Java ;-)

[horrorho]

@Jirmi Yes I suspect you're right. I'm already questioning the sanity of my approach.

[horrorho]

@Jirmi On further reflection, I don't think a JNI wrapper confers a huge advantage over using a native binary in our particular set of circumstances. We will be decompressing files on disk and it's trivial invoking lzfse processes from Java.

The issue is actually having the lzfse binary on the system. It's a non-issue for modern Apple device/ OS users as lzfse is baked in. Linux it's trivial to compile. Windows is the sticking point as it will require additional tool chains and possibly more effort than the casual user would like.

I'm not sure what the legal deal is with supplying pre-compiled lzfse executables for Windows users, although I suspect it's not an issue. I'll leave it up in the air for now.

I'm about 90% of the way through coding a bvx2 decompressor and hopefully I'll have all of tomorrow free to finish it. I'll update on it's progress and make a decision at that point.

If others are interested I can upload my decompressor to Github, although I'll probably have to change it's name to something other than CatButts. Something like jlzfse would be sensible but it's probably best left for someone who'll do it justice.

:cactus:

[michaljirman]

@horrorho I am quite interested to see the complexity of the decompressor written in Java. So yes, pls upload it whenever you get some time, the name jlzfse sounds good - although just slightly better than CatButts :grin: :grin:

Also, I believe it should be possible to compile the lzfse for windows in Visual Studio but I haven't tried myself. Just guessing based on some issues on github (e.g. https://github.com/lzfse/lzfse/pull/33). For best user experience, it would mean to supply InflatableDonkey with multiple binaries for each platform. Considering that, I guess it is best for you to finish it in Java since you are nearly done anyway.

[horrorho]

Ok! I've pushed the patch. I've only tested it on a single backup, thank you @Yaldo425!

@Jirmi I'll push it once it's in a sensible state. Agree in regards to targeting user experience.

[horrorho]

67 Implemented via RagingMoose.