CipherError

[devistest]

I'm getting an error while parsing the notes from my backup. The password is correct in the file, and all unencrypted notes can be displayed (tho any with images do not). The error for the cipher is:

Apple Decrypter: Apple Note: 251 Attempting decryption. Apple Decrypter: Apple Note: 251 caught a CipherError while trying final decrypt, likely the unwrapped key is not correct.

Could be user error, but I'm not sure what I'm doing incorrectly.

[threeplanetssoftware]

To make sure I understand, all of the notes that were encrypted threw a CipherError consistently? That would most likely be the password being incorrect. You might want to check for any additional whitespace on the line, remove any quotes if you have it quoted, etc. It should just be the bare password on a line.

For example, this is my test:

[notta@cuppa apple_cloud_notes_parser]$ cat passwords.txt
password
[notta@cuppa apple_cloud_notes_parser]$ ruby notes_cloud_ripper.rb -i ~/iphone/ -w passwords.txt --show-password-successes
...
------------------------------
Successfully decrypted notes using passwords: password
These are NOT logged, note it down now if you need it.
------------------------------

The images would likely be a separate issue, were you running this just from a NoteStore.sqlite file, or did you point it to a backup of some sort (Mac or iDevice)?

[devistest]

unfortunately, yes, I've made sure this is the correct password and there is no whitespace. I can copy/paste this into the icloud site and unlock the notes, but it fails still.

Failed to decrypt Apple Note: 251, unwrapped key likely isn't right.
Skipping Note ID 270 due to a missing folder or account, check the debug log for more details.
Updated AppleNoteStore object with 16 AppleNotes in 3 folders belonging to 1 accounts.
Updated AppleNoteStore object with 0 AppleNotes in 1 folders belonging to 1 accounts.
Adding the ZICNOTEDATA.ZPLAINTEXT and ZICNOTEDATA.ZDECOMPRESSEDDATA columns, this takes a few seconds
------------------------------
Successfully decrypted notes using passwords: <<REDACTED>>
These are NOT logged, note it down now if you need it.
------------------------------

As for the images, I've both from just NoteStore.sqlite and the full backup with the same results.

[threeplanetssoftware]

What I find interesting is that the log appears to indicate that you did successfully open at least one locked object. Are you able to see anything in the debug_log.txt file that indicates which one was successful? Maybe looking at that Note will help explain why the other fails.

[notta@cuppa apple_cloud_notes_parser]$ cat output/2021_06_15-22_17_07/debug_log.txt | grep "generated a decrypt" 
D, [2021-06-15T22:17:07.728370 #407832] DEBUG -- : Apple Decrypter: Apple Note: 65 generated a decrypt
D, [2021-06-15T22:17:07.740646 #407832] DEBUG -- : Apple Decrypter: Apple Note: 77 generated a decrypt
D, [2021-06-15T22:17:07.747491 #407832] DEBUG -- : Apple Decrypter: Apple Backup encrypted file generated a decrypt
D, [2021-06-15T22:17:07.754231 #407832] DEBUG -- : Apple Decrypter: Apple Backup encrypted file generated a decrypt

I'd also be interested to make sure that note was not deleted. I see there are only 16 at you're up at number 251, which makes me think there is some turnover and I have observed odd behaviors with notes after they have passed the time period wherein garbage collection should occur. These two SQLite queries might be interesting, especially if you look at the difference between whatever did open up and note 251. The first of which I would be making sure the result isn't NULL and the second of which I would be making sure there are values in each column (which there should be since we didn't fail to generate the key):

SELECT ZDATA FROM ZICNOTEDATA WHERE ZNOTE=251

SELECT ZCRYPTOSALT, ZCRYPTOTAG, ZCRYPTOWRAPPEDKEY, ZCRYPTOINITIALIZATIONVECTOR 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE Z_PK=251

Finally, if you wanted to check on what this code is doing, I have code in this blog entry that explains each step of the process and you could plug in the results of the above two SQLite queries to make sure something isn't just broken in this program without having to give me any of that information.

[devistest]

so for the images, I think my issue has something to do with this from the debug_log:

Note 46: Created a new Embedded Object of type public.jpeg
Note 46: Created a new Embedded Object of type thumbnail
Can't call back_up_file with filepath_on_disk that is nil
Note 46: Created a new Embedded Object of type thumbnail
Can't call back_up_file with filepath_on_disk that is nil
AppleNotesEmbeddedPublicJpeg 40DF4BF7-B927-4C5D-B491-6497076749FA: Filename is IMG_1158.JPG
Can't call back_up_file with filepath_on_disk that is nil
AppleNote: Note 46 replacing attachment 5556EA59-CD84-4303-9168-255EBEC3F955
Note 46: Created a new Embedded Object of type public.heic
Note 46: Created a new Embedded Object of type thumbnail
Can't call back_up_file with filepath_on_disk that is nil

The note was definitely not deleted, actually there are 3 locked notes in the db and none of them are recovered. I'm beginning to think maybe there is an issue with the database....

sqlite> SELECT ZDATA FROM ZICNOTEDATA WHERE ZNOTE=251;
 ▒▒۷▒9▒kQ-?▒d▒▒&h▒
▒oC
   B▒▒
4𲨟Q▒a▒U/▒@q▒▒&▒Xe"▒^
▒$▒VL▒6>6=▒▒▒&▒`^3F"▒▒J▒!GXV1▒▒[Ƞ=;▒ҏ$-!▒
                                         ▒s▒~h▒▒▒q`<▒~MQW▒▒1▒㧡▒▒c▒▒P y_lgq▒hõ▒▒▒T▒▒I▒F▒▒

[threeplanetssoftware]

Thanks, those are interesting observations. So the note clearly has data, clearly is not being decrypted, but (based on the first log) something was or the password should not have been listed at the end. Because you are seeing the Can't call back_up_file with filepath_on_disk that is nil errors it clearly is a backup that isn't just a single file. May I ask what type (Mac, iTunes, or Physical) and if it is a full backup or just partial?

If you go to the folder you are pointing to, can you try to find one of the files that this is claiming can't be found? From your snippet, I would suggest:

find [folder you pointed at, ~/iphone/ in my example] -name 40DF4BF7-B927-4C5D-B491-6497076749FA

This won't work directly if you're looking at an iTunes backup, wherein the filenames are hashed. In that case you would open the Manifest.db file in the folder you pointed to (~/iphone/Manifest.db for me) and run this query, then use the fileID result in your find command:

SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%40DF4BF7-B927-4C5D-B491-6497076749FA%'

[devistest]

It is an itunes backup (should be full), here is the query result on Manifest.db

sqlite> SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%40DF4BF7-B927-4C5D-B491-6497076749FA%';
bd6085165d412ccd822be2ebd99d0105de49173f|Previews/40DF4BF7-B927-4C5D-B491-6497076749FA-1-288x384-0.png
9807ae55b59a499c85d49da30f8ce391e390b370|Previews/40DF4BF7-B927-4C5D-B491-6497076749FA-1-288x384-0-oriented.png
6f9f363abaf7171f81b7d8f658644c2aae487fb4|Previews/40DF4BF7-B927-4C5D-B491-6497076749FA-1-144x192-0.png
de82a993755cd24e65a4633f62508364e6070819|tmp/HardLinkURLTemp/40DF4BF7-B927-4C5D-B491-6497076749FA
5abcbffbc86fed35eb9dd96c3336fc168d4764c8|tmp/HardLinkURLTemp/40DF4BF7-B927-4C5D-B491-6497076749FA/1593200297
624a906d4f313ab0ce6629fbb135bdd8963bc4ca|tmp/HardLinkURLTemp/40DF4BF7-B927-4C5D-B491-6497076749FA/1593200297/Electrical Hazards.JPG

[threeplanetssoftware]

Thanks, so the files should be there. What happens when you try to look at one, such as:

find [your path, such as ~/iphone/ in my example] -name bd6085165d412ccd822be2ebd99d0105de49173f find [your path] -name 9807ae55b59a499c85d49da30f8ce391e390b370 find [your path] -name 6f9f363abaf7171f81b7d8f658644c2aae487fb4

Also, it would be worth looking for the original filename in Manifest.db, and doing the same sort of find command on the resulting fileID: SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%IMG_1158.JPG%'

Thanks for helping to troubleshoot this!

[devistest]

I can search for and find all of these ids in the files and can even view the images by pulling them into VLC player.

sqlite> SELECT fileID,relativePath FROM Files WHERE relativePath LIKE '%IMG_1158  .JPG%';
aa9682b53a7ca7b73ef8cedfcec22fb01cb81a18|Media/C07A1BC9-3E92-4C0D-942A-ED20E8F9B  ECB/IMG_1158.JPG
32cfae382e8f193c5602bdf64091251c5297ad1e|Media/PhotoData/Thumbnails/V2/DCIM/101A  PPLE/IMG_1158.JPG
998dbb485313f4543740759529c25bb676defb11|Media/PhotoData/Thumbnails/V2/DCIM/101A  PPLE/IMG_1158.JPG/5005.JPG
d39e2a9e5ffd4f4ca8ed388311d6570ffbf28e50|Media/DCIM/101APPLE/IMG_1158.JPG

I searched for these files and was able to view them as well.

[threeplanetssoftware]

Thanks for working through this, it is really helpful and I'm getting a much better idea of what the problem is. If you look at the source for AppleNotesEmbeddedPublicJpeg, this is the function that gets the filepath we search for the media:

https://github.com/threeplanetssoftware/apple_cloud_notes_parser/blob/e983c6a58e2a5a86ca658f5c0c4d7c650ab5c839/lib/AppleNotesEmbeddedPublicJpeg.rb#L110-L113

You'll notice it is expecting Accounts/[UUID]/Media/[UUID]/[filename]. For example, a random one in my test data is

sqlite> SELECT fileID,relativePath FROM Files WHERE relativePath LIKE "%D533089A-83BF-44B1-A515-5DB4E157A1F2%";
2251e845879d544d85d866357797ec28b0274360|Accounts/D34714F2-F2F7-4AD0-8EA5-A54E31A74D72/Media/D533089A-83BF-44B1-A515-5DB4E157A1F2
41da368bed81d5061bb71f91828610b2221daaa4|Accounts/D34714F2-F2F7-4AD0-8EA5-A54E31A74D72/Media/D533089A-83BF-44B1-A515-5DB4E157A1F2/IMG_0014.png

However, in your data, the Account's UUID is not present and the path is starting down at the Media folder, so when get_media_filepath computes the filepath, it will be looking for something that doesn't exist. That is likely what is causing us to hit line 78 in this function, it fails to pass the File.exist? check:

https://github.com/threeplanetssoftware/apple_cloud_notes_parser/blob/e983c6a58e2a5a86ca658f5c0c4d7c650ab5c839/lib/AppleBackup.rb#L74-L80

So I think I understand what is happening, but I don't understand why. Can you tell me which version of iOS you are running and the type of device? Also, you mentioned iCloud, are these notes created on the iDevice, on your Mac, or in a browser? I hope those details will let me zero in on being able to recreate it myself.

Edit: One other thing to make sure, include the domain in your query against Manifest.db. Some of your entries are likely the CameraRollDomain and not what Notes uses, AppDomainGroup-group.com.apple.notes.

[devistest]

No, thanks for working with me on this. I'll happily see if we can iron this out. The device is an iphone 11 running 14.6 iOS. The notes were all created on this iphone. I mentioned iCloud because it was the easiest way to copy/paste the password to ensure it was correct.

[threeplanetssoftware]

I haven't yet reproduced this, but I have seen at least one oddity when running this on my Mac. Could you try running this on the actual Mac version of Notes and see if it gives you the same errors as you get from the iTunes backup?

ruby notes_cloud_ripper.rb -m ~/Library/Application\ Support/MobileSync/Backup/[device UDID] -w passwords.txt --show-password-successes

What I'm finding is I have one note that I can unlock on my phone with the right password as all the other notes which does not unlock when I run the Mac version. When I run the iTunes backup, it opens just fine. As yet undecided about exactly why, but I'm curious to see how reliably reproducible yours is in your environment. Just rmember the Note IDs will change, my problem child (117) became 84 on the Mac.

[devistest]

I don't have a mac to run this on, I'm running on a linux box.

[threeplanetssoftware]

Oh, sorry, I made an assumption based on the iTunes backup. How are you getting the iTunes rip? My test device is an "older" iPhone on iOS 14.6 and I'm ripping it down to a Mac running Big Sur. Then I check the results on my Linux test environment and the Mac, both the iPhones version and Mac version. As I said, on Mac I see a potential discrepancy, but not sure it is your issue or not.

[devistest]

well the rip is being done on windows, but I'm doing the work with your script on my linux box

[threeplanetssoftware]

I have a theory that this might be related to how and when the iCloud changes sync. I have been able to find a case on my Mac backup that seems to be the same as yours and believe it is tied to this code:

https://github.com/threeplanetssoftware/apple_cloud_notes_parser/blob/e983c6a58e2a5a86ca658f5c0c4d7c650ab5c839/lib/AppleNoteStore.rb#L610-L637

Ignoring the obvious bug of crypto_wrapped_key v crypto_key which is now fixed in the main branch, when I make the if block always fail the note of mine that won't decrypt, decrypts. It seems there are values "stuck" in the ZUNAPPLIEDENCRYPTEDRECORD and even after trying to force the phone and my Mac to sync a few times, including unlocking that note, it persists.

Would you please confirm there is a value in ZUNAPPLIEDENCRYPTEDRECORD on your problematic note? Please run this (note that I'm using "length" to avoid printing the actual contents if you want to compare the values in the two tables you could remove the length and look at the hex to compare):

SELECT ZICCLOUDSYNCINGOBJECT.Z_PK as NoteID,
       length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOSALT)) as CloudSalt,
       length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOTAG)) as CloudTag, 
       length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOWRAPPEDKEY)) as CloudUnwrappedKey, 
       length(hex(ZICCLOUDSYNCINGOBJECT.ZCRYPTOINITIALIZATIONVECTOR)) as CloudIV,
       length(hex(ZICCLOUDSYNCINGOBJECT.ZUNAPPLIEDENCRYPTEDRECORD)) as CloudUnapplied,
       length(hex(ZICNOTEDATA.ZCRYPTOINITIALIZATIONVECTOR)) as NoteIV,
       length(hex(ZICNOTEDATA.ZCRYPTOTAG)) as NoteTag
FROM ZICCLOUDSYNCINGOBJECT, ZICNOTEDATA
WHERE ZICCLOUDSYNCINGOBJECT.ZUNAPPLIEDENCRYPTEDRECORD is not NULL AND ZICNOTEDATA.ZNOTE=ZICCLOUDSYNCINGOBJECT.Z_PK

In my case, I have the note I'm having issues with (117), as well as two others that I don't have issues with, likely because what is in the ZUNAPPLIEDENCRYPTEDRECORD is properly synced on notes 11 and 22. All three of these same notes are just fine on my iTunes backup.

NoteID	CloudSalt	CloudTag	CloudUnwrappedKey	CloudIV	CloudUnapplied	NoteIV	NoteTag
11	32	32	48	32	9956	32	32
22	32	32	48	32	9790	32	32
117	32	32	48	32	9634	32	32

I need to think about the best way to catch this case, but would appreciate your confirmation that this is likely the right direction by running the above SQLite query.

[devistest]

Yeah there is definitely data there:

66|32|32|48|32|9960|32|32
108|32|32|48|32|9614|32|32
109|32|32|48|32|9646|32|32

The contents without the length parameter included is massive.

[threeplanetssoftware]

What's interesting about that output is I don't see Note 251 (just 66, 108, and 109), the one you have with the CipherError listed in the original report, included. That would make me think it is not actually the right track. As a bit of a long shot, could you check out branch 21-wal-copy and see if that makes any difference at all?

[devistest]

ran that version, and the result looks identical

[threeplanetssoftware]

Ok, well let's see if this works. I just pushed branch 23-bug-encryption-debug (7849da1a0245bda3e8bb50b6d0930d0d7b90dd35), which simply bypasses the ZUNAPPLIEDENCRYPTEDRECORD field. Based on your output I don't think it should fix it, but please try and let me know if there are any differences.

[devistest]

some success!... no longer get the encryption issue, but still no love on the images due to the Can't call back_up_file with filepath_on_disk that is nil error.

[threeplanetssoftware]

Ok, in that case this is a bug that I can reproduce and I'll work on a solution. I have broken the images off into a separate issue because it is unrelated to the CipherError that spawned this ticket and I'd prefer to clearly track the distinct problems. Otherwise things get lost.

[devistest]

understood, appreciate the help!

[threeplanetssoftware]

This isn't as quick of a change because the case I think I have to catch means that there are two sets of "valid" settings, as far as the database is concerned. Those in ZUNAPPLIEDENCRYPTEDRECORD and those in the database row itself. I've added a bit more smarts to how I handle that in 17b340af7548c4cd868fd5b022b9dd5212aa2643 on the 23-bug-encryption-debug branch and pulled in the other changes.

This is a pure stopgap for you, not intended as production code and it may work for you, or this may continue to not work. If it does not work, I'd suggest looking at the files changed in the listed commit and try changing back to and false the lines I expanded. The reason I'm not doing that as a solution is in my test database I had objects that had the right settings in the ZUNAPPLIEDENCRYPTEDRECORD and some that did not, so I could open the note, but then not the object within the note. You may run into the same issue until I solve this a bit more permanently.

[devistest]

Excellent. You're a saint for patiently delving into this niche case for me that will probably never happen to anyone else. I was able to recover everything.

[threeplanetssoftware]

That's great to hear! I will probably take a bit to figure out the best method to actually solve this. My hope is I can compare the timestamps on the ZUNAPPLIEDENCRYPTEDRECORD and not just try two sets of variables every time.