search

threeplanetssoftware / apple_cloud_notes_parser

Parser for Apple Notes data stored on the Cloud as seen on Apple handsets
MIT License
412 stars 26 forks source link

Generates empty .html and no csv (undefined method `name' for nil:NilClass and undefined method `primary_key' for nil:NilClass) #11

Closed mshibo closed 4 years ago

mshibo commented 4 years ago 
mshibo@mshibo ~/apple_cloud_notes_parser $ ruby notes_cloud_ripper.rb --file NoteStore.sqlite 

Starting Apple Notes Parser at Thu May 28 06:37:21 2020
Storing the results in ./output/2020_05_28-06_37_21

Created a new AppleBackup from single file: NoteStore.sqlite
Guessed Notes Version: 13
com.compuserve.gif is unrecognized, please submit a bug report to this project's GitHub repo to report this: https://github.com/threeplanetssoftware/apple_cloud_notes_parser/issues
com.compuserve.gif is unrecognized, please submit a bug report to this project's GitHub repo to report this: https://github.com/threeplanetssoftware/apple_cloud_notes_parser/issues
com.apple.drawing is unrecognized, please submit a bug report to this project's GitHub repo to report this: https://github.com/threeplanetssoftware/apple_cloud_notes_parser/issues
Updated AppleNoteStore object with 940 AppleNotes in 3 folders belonging to 1 accounts.
Adding the ZICNOTEDATA.ZPLAINTEXT and ZICNOTEDATA.ZDECOMPRESSEDDATA columns, this takes a few seconds
/home/mshibo/apple_cloud_notes_parser/lib/AppleNote.rb:353:in `generate_html': undefined method `name' for nil:NilClass (NoMethodError)
    from /home/mshibo/apple_cloud_notes_parser/lib/AppleNoteStore.rb:532:in `block in generate_html'
    from /home/mshibo/apple_cloud_notes_parser/lib/AppleNoteStore.rb:530:in `each'
    from /home/mshibo/apple_cloud_notes_parser/lib/AppleNoteStore.rb:530:in `generate_html'
    from notes_cloud_ripper.rb:127:in `block (2 levels) in <main>'
    from notes_cloud_ripper.rb:126:in `open'
    from notes_cloud_ripper.rb:126:in `block in <main>'
    from notes_cloud_ripper.rb:123:in `each'
    from notes_cloud_ripper.rb:123:in `<main>'
threeplanetssoftware commented 4 years ago

Thanks for reporting this. Your error appears to say there should be only one account, but I suspect there should be at least two and it looks like the one account that is seen isn't actually being pulled out. Could you please run this query against your NoteStore.sqlite file?

SELECT ZICCLOUDSYNCINGOBJECT.Z_PK
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.ZNAME IS NOT NULL

From the output, please run this query:

SELECT ZICCLOUDSYNCINGOBJECT.ZNAME, ZICCLOUDSYNCINGOBJECT.Z_PK, ZICCLOUDSYNCINGOBJECT.ZIDENTIFIER 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.Z_PK IN ([your Z_PKs])
For example, after running the first query on my iOS13 test database, the results were: Z_PK
2
10

My second query then became:

SELECT ZICCLOUDSYNCINGOBJECT.ZNAME, ZICCLOUDSYNCINGOBJECT.Z_PK, ZICCLOUDSYNCINGOBJECT.ZIDENTIFIER 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.Z_PK IN (2,10)
And the results were: ZNAME Z_PK ZIDENTIFIER
On My iPhone 2 LocalAccount
iCloud 10 A347ECF2-F2F7-4AD0-8EA5-BE4E31A74D72

Having the results of those two queries would help in validating I'm on the right track with my fix. If you want to obscure the account name if it is personally identifying, please do so.

banj commented 4 years ago

I encountered the same issue when playing around with this (thanks for creating such a great tool!)

Here's the error:

ba@arren apple_cloud_notes_parser-master % rake
/Users/ba/.rbenv/versions/2.5.1/bin/ruby notes_cloud_ripper.rb --file NoteStore.sqlite

Starting Apple Notes Parser at Tue Jul 14 18:10:36 2020
Storing the results in ./output/2020_07_14-18_10_36

Created a new AppleBackup from single file: NoteStore.sqlite
Guessed Notes Version: 13
com.microsoft.excel.xls is unrecognized, please submit a bug report to this project's GitHub repo to report this: https://github.com/threeplanetssoftware/apple_cloud_notes_parser/issues
Updated AppleNoteStore object with 682 AppleNotes in 11 folders belonging to 1 accounts.
Adding the ZICNOTEDATA.ZPLAINTEXT and ZICNOTEDATA.ZDECOMPRESSEDDATA columns, this takes a few seconds
Traceback (most recent call last):
    8: from notes_cloud_ripper.rb:123:in `<main>'
    7: from notes_cloud_ripper.rb:123:in `each'
    6: from notes_cloud_ripper.rb:126:in `block in <main>'
    5: from notes_cloud_ripper.rb:126:in `open'
    4: from notes_cloud_ripper.rb:127:in `block (2 levels) in <main>'
    3: from /Users/ba/Downloads/apple_cloud_notes_parser-master/lib/AppleNoteStore.rb:530:in `generate_html'
    2: from /Users/ba/Downloads/apple_cloud_notes_parser-master/lib/AppleNoteStore.rb:530:in `each'
    1: from /Users/ba/Downloads/apple_cloud_notes_parser-master/lib/AppleNoteStore.rb:532:in `block in generate_html'
/Users/ba/Downloads/apple_cloud_notes_parser-master/lib/AppleNote.rb:354:in `generate_html': undefined method `primary_key' for nil:NilClass (NoMethodError)
rake aborted!
Command failed with status (1): [/Users/ba/.rbenv/versions/2.5.1/bin/ruby n...]
/Users/ba/Downloads/apple_cloud_notes_parser-master/Rakefile:7:in `block in <top (required)>'
Tasks: TOP => default => run
(See full trace by running task with --trace)

I ran the queries you asked for above:

First

SELECT ZICCLOUDSYNCINGOBJECT.Z_PK
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.ZNAME IS NOT NULL
Result: Z_PK
1

Second:

SELECT ZICCLOUDSYNCINGOBJECT.ZNAME, ZICCLOUDSYNCINGOBJECT.Z_PK, ZICCLOUDSYNCINGOBJECT.ZIDENTIFIER 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.Z_PK IN (1)
Result: ZNAME Z_PK ZIDENTIFIER
iCloud 1 9C572F73-7275-4B93-B434-975C27C96F2F
threeplanetssoftware commented 4 years ago

Thanks for the additional report! I'll see if this helps finish the bug off.

threeplanetssoftware commented 4 years ago

This may be related, but it actually failed on the folders, not the accounts. Could you please run these queries the same as the ones you did before?

First: 

SELECT ZICCLOUDSYNCINGOBJECT.Z_PK 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.ZTITLE2 IS NOT NULL

Second:

SELECT ZICCLOUDSYNCINGOBJECT.ZTITLE2, ZICCLOUDSYNCINGOBJECT.ZOWNER, 
ZICCLOUDSYNCINGOBJECT.Z_PK 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.Z_PK IN ([your Z_PKs])

Thank you for the help!

banj commented 4 years ago

All the thanks go to you! Thanks for puzzling this out -- I'm learning from it by tracing through your process.

First:

Z_PK |--| 2 4 164 169 177 264 474 536 794 1109 1969

Second:

SELECT ZICCLOUDSYNCINGOBJECT.ZTITLE2, ZICCLOUDSYNCINGOBJECT.ZOWNER, 
ZICCLOUDSYNCINGOBJECT.Z_PK 
FROM ZICCLOUDSYNCINGOBJECT 
WHERE ZICCLOUDSYNCINGOBJECT.Z_PK IN (2,4,164,169,177,264,474,536,794,1109,1969)

Result:

ZTITLE2 ZOWNER Z_PK
Notes 1 2
Recently Deleted 1 4
2017-18 Existential Lit 1 164
2017-18 Sophmore English 1 169
Beacon General 1 177
Family 1 264
Canisius 1 474
English Research 1 536
Digitizing Documents 1 794
2019-20 Scheduling 1 1109
2019-20 Existentialism 1 1969
threeplanetssoftware commented 4 years ago

I have been testing this with all of the test files I have sitting around for various iOS/Mac versions and cannot yet reproduce it. I believe based on your query results @banj that we are correctly identifying the version (iOS 13) and getting the correct information from ZICCLOUDSYNCINGOBJECT. I also believe that the issues reported by @dmd and @mshibo are all related to the same error, but can't make headway.

To better identify this, I just added some rudimentary logging in 8c327cc1e6155d180a90c528152d453b1ec78bfb. If you pull that down and run it again, it should create a log file in the output folder with a lot more information about each account, folder, and note it is working on. This will help identify if it is somehow the database being fingerprinted wrong (i.e. it dies on the first note), or something specifically weird about just one note. It should also just drop the offending note, letting you at least see the rest of them.

Could you try this version and let me know if it finishes the run and how many notes are parsed after it finishes? If the log doesn't contain anything sensitive, I'd appreciate seeing the relevant lines which start with "Rip Note", "Rip Account" and "Rip Folder".

Ultimately, I'm very puzzled by the inability to recreate this bug and if you have a file doing this which you feel comfortable sharing, or can provide instructions for how to reproduce, I'd love to add it to my tests.

Thank you for your patience.

dmd commented 4 years ago

Here you go: https://gist.github.com/43c468414bb25b259eecad3e413c5042

threeplanetssoftware commented 4 years ago

Thank you, that's really interesting as it confirms the folder value was missing. I'm unsure how to get Notes to add a note without a valid folder so while this may help with mocking up an example, it doesn't yet give us a great understanding of why it happens. Some questions that may help in recreating this:

  1. Is this note in a folder shared by another user/account, or did you create it?
  2. Is this note in a folder that was deleted or moved?
  3. Was this note saved on the device you dumped the database from? Or was it synced from another location?
  4. Was this device (or wherever the note was created) ever updated from an older version of iOS? There is a different column specifically in the iOS 11 version (Z_11FOLDERS) which I'd be interested if you saw it in your ZICCLOUDSYNCINGOBJECT schema.

Depending on your comfort level, I would love to see the non-BLOB results of these two SQLite queries, with headers. The first will select everything from the ZICCLOUDSYNCINGOBJECT table for that specific note (including things like the note title) and the second will select everything from the ZICNOTEDATA table for that specific note (excluding the note's text if you do not include BLOB data):

SELECT ZICCLOUDSYNCINGOBJECT.* FROM ZICCLOUDSYNCINGOBJECT WHERE ZICCLOUDSYNCINGOBJECT.Z_PK=19

SELECT ZICNOTEDATA.* FROM ZICNOTEDATA WHERE ZICNOTEDATA.ZNOTE=19

Separate from trying to recreate this, can you please confirm that the program actually finished this time, skipping the offending note?

Thank you for taking the time to help troubleshoot this!

dmd commented 4 years ago

I'm a little confused because you keep talking about iOS. I'm not interacting with my phone in any way for this; I'm using:

ruby notes_cloud_ripper.rb -m /Users/dmd/Library/Group\ Containers/group.com.apple.notes/

Separate from trying to recreate this, can you please confirm that the program actually finished this time, skipping the offending note?

Not sure what you mean by "finished". The program produces no output in the output folder other than debug_log.txt and copying in the NoteStore sqlite db.

sqlite> SELECT ZICCLOUDSYNCINGOBJECT.* FROM ZICCLOUDSYNCINGOBJECT WHERE ZICCLOUDSYNCINGOBJECT.Z_PK=19;
19|9|2|0|0|0|0|1|||18||||||||||||||||||||||||||0|0|||0||1||8|||||||||||||||||||||||||||||||||||||||||-541232580||||||A9224A0C-ADF8-480D-9AA0-46A6C1A3C86F||||||||||||||||||||||||||||||||||||||||||bplist00�TX$versionY$archiverT$topX$objects|||||||||||||||
sqlite> SELECT ZICNOTEDATA.* FROM ZICNOTEDATA WHERE ZICNOTEDATA.ZNOTE=19;
8|15|1|19|||

I do not have any folders at all:

image

banj commented 4 years ago

I ran this and had a look at the log and then checked that note in the database and found that ZICNOTEDATA.ZDATA was NULL, I found that interesting. The note id was 305, I ran the query you asked for:

Z_PK Z_ENT Z_OPT ZCRYPTOITERATIONCOUNT ZISPASSWORDPROTECTED ZMARKEDFORDELETION ZMINIMUMSUPPORTEDNOTESVERSION ZNEEDSINITIALFETCHFROMCLOUD ZNEEDSTOBEFETCHEDFROMCLOUD ZNEEDSTOSAVEUSERSPECIFICRECORD ZCLOUDSTATE ZACCOUNT ZCHECKEDFORLOCATION ZFILESIZE ZHANDWRITINGSUMMARYVERSION ZHASMARKUPDATA ZIMAGECLASSIFICATIONSUMMARYVERSION ZIMAGEFILTERTYPE ZOCRSUMMARYVERSION ZORIENTATION ZSECTION ZLOCATION ZMEDIA ZNOTE ZNOTEUSINGTITLEFORNOTETITLE ZPARENTATTACHMENT ZAPPEARANCETYPE ZSCALEWHENDRAWING ZVERSION ZVERSIONOUTOFDATE ZATTACHMENT ZSTATE ZACCOUNT1 ZTYPE ZACCOUNT2 ZATTACHMENT1 ZATTACHMENTVIEWTYPE ZISPINNED ZLEGACYNOTEWASPLAINTEXT ZNOTEHASCHANGES ZPAPERSTYLETYPE ZPREFERREDBACKGROUNDTYPE ZACCOUNT3 ZFOLDER ZNOTEDATA ZTITLESOURCEATTACHMENT ZISHIDDENNOTECONTAINER ZSORTORDER ZOWNER ZACCOUNTTYPE ZDIDCHOOSETOMIGRATE ZDIDFINISHMIGRATION ZDIDMIGRATEONMAC ZSTOREDATASEPARATELY ZACCOUNTDATA ZCUSTOMNOTESORTTYPEVALUE ZFOLDERTYPE ZIMPORTEDFROMLEGACY ZACCOUNT4 ZPARENT ZCREATIONDATE ZCROPPINGQUADBOTTOMLEFTX ZCROPPINGQUADBOTTOMLEFTY ZCROPPINGQUADBOTTOMRIGHTX ZCROPPINGQUADBOTTOMRIGHTY ZCROPPINGQUADTOPLEFTX ZCROPPINGQUADTOPLEFTY ZCROPPINGQUADTOPRIGHTX ZCROPPINGQUADTOPRIGHTY ZDURATION ZMODIFICATIONDATE ZORIGINX ZORIGINY ZPREVIEWUPDATEDATE ZSIZEHEIGHT ZSIZEWIDTH ZHEIGHT ZMODIFIEDDATE ZSCALE ZWIDTH ZSTATEMODIFICATIONDATE ZMODIFICATIONDATEATIMPORT ZCREATIONDATE1 ZFOLDERMODIFICATIONDATE ZLASTNOTIFIEDDATE ZLASTVIEWEDMODIFICATIONDATE ZLEGACYMODIFICATIONDATEATIMPORT ZMODIFICATIONDATE1 ZCUSTOMNOTESORTTYPEMODIFICATIONDATE ZDATEFORLASTTITLEMODIFICATION ZPARENTMODIFICATIONDATE ZIDENTIFIER ZPASSWORDHINT ZZONEOWNERNAME ZADDITIONALINDEXABLETEXT ZFALLBACKSUBTITLEIOS ZFALLBACKSUBTITLEMAC ZFALLBACKTITLE ZHANDWRITINGSUMMARY ZIMAGECLASSIFICATIONSUMMARY ZOCRSUMMARY ZREMOTEFILEURLSTRING ZSUMMARY ZTITLE ZTYPEUTI ZURLSTRING ZUSERTITLE ZDEVICEIDENTIFIER ZCONTENTHASHATIMPORT ZFILENAME ZLEGACYCONTENTHASHATIMPORT ZLEGACYIMPORTDEVICEIDENTIFIER ZLEGACYMANAGEDOBJECTIDURIREPRESENTATION ZSELECTEDINKCOLORSTRING ZSELECTEDINKIDENTIFIER ZSNIPPET ZTHUMBNAILATTACHMENTIDENTIFIER ZTITLE1 ZACCOUNTNAMEFORACCOUNTLISTSORTING ZNESTEDTITLEFORSORTING ZNAME ZUSERRECORDNAME ZTITLE2 ZASSETCRYPTOINITIALIZATIONVECTOR ZASSETCRYPTOTAG ZCRYPTOINITIALIZATIONVECTOR ZCRYPTOSALT ZCRYPTOTAG ZCRYPTOWRAPPEDKEY ZENCRYPTEDVALUESJSON ZSERVERRECORDDATA ZSERVERSHAREDATA ZUNAPPLIEDENCRYPTEDRECORD ZUSERSPECIFICSERVERRECORDDATA ZMERGEABLEDATA ZFALLBACKIMAGECRYPTOINITIALIZATIONVECTOR ZFALLBACKIMAGECRYPTOTAG ZMARKUPMODELDATA ZMERGEABLEDATA1 ZMETADATADATA ZCRYPTOMETADATAINITIALIZATIONVECTOR ZCRYPTOMETADATATAG ZENCRYPTEDMETADATA ZMETADATA ZLASTNOTIFIEDTIMESTAMPDATA ZLASTVIEWEDTIMESTAMPDATA ZREPLICAIDTOUSERIDDICTDATA ZCRYPTOVERIFIER ZMERGEABLEDATA2
305 9 2 0 0 0 0 1 (NULL) (NULL) 305 (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) 0 0 (NULL) (NULL) 0 (NULL) 1 (NULL) 101 (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) -541232580 (NULL) (NULL) (NULL) (NULL) (NULL) 0A894BA1-C86C-4EEC-929E-3C48B65740A5 (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) {"type":"Buffer","data":[98,112,108,105,115,116,48,48,212,1,2,3,4,5,6,7,84,88,36,118,101,114,115,105,111,110,89,36,97,114,99,104,105,118,101,114,84,36,116,111,112,88,36,111,98,106,101,99,116,115,18,0,1,134,160,95,16,15,78,83,75,101,121,101,100,65,114,99,104,105,118,101,114,223,16,38,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,46,49,46,46,47,53,46,46,46,46,46,59,60,46,46,46,46,46,66,46,68,46,47,46,46,46,74,53,47,46,47,46,80,81,47,46,95,16,22,84,111,109,98,115,116,111,110,101,100,80,117,98,108,105,99,75,101,121,73,68,115,95,16,25,72,97,115,85,112,100,97,116,101,100,80,97,114,101,110,116,82,101,102,101,114,101,110,99,101,95,16,19,67,104,97,105,110,80,114,111,116,101,99,116,105,111,110,68,97,116,97,93,75,110,111,119,110,84,111,83,101,114,118,101,114,95,16,17,68,105,115,112,108,97,121,101,100,72,111,115,116,110,97,109,101,89,66,97,115,101,84,111,107,101,110,95,16,16,87,97,110,116,115,67,104,97,105,110,80,67,83,75,101,121,91,82,101,99,111,114,100,67,116,105,109,101,90,82,111,117,116,105,110,103,75,101,121,95,16,18,80,114,111,116,101,99,116,105,111,110,68,97,116,97,69,116,97,103,95,16,38,80,114,101,118,105,111,117,115,80,114,111,116,101,99,116,105,111,110,68,97,116,97,69,116,97,103,70,114,111,109,85,110,105,116,84,101,115,116,95,16,18,67,111,110,102,108,105,99,116,76,111,115,101,114,69,116,97,103,115,95,16,26,80,114,101,118,105,111,117,115,80,114,111,116,101,99,116,105,111,110,68,97,116,97,69,116,97,103,90,82,101,99,111,114,100,84,121,112,101,95,16,19,67,114,101,97,116,111,114,85,115,101,114,82,101,99,111,114,100,73,68,95,16,15,80,97,114,101,110,116,82,101,102,101,114,101,110,99,101,89,83,104,97,114,101,69,116,97,103,88,80,67,83,75,101,121,73,68,92,90,111,110,101,105,115,104,75,101,121,73,68,95,16,32,77,117,116,97,98,108,101,69,110,99,114,121,112,116,101,100,80,117,98,108,105,99,83,104,97,114,105,110,103,75,101,121,84,69,84,97,103,95,16,22,80,114,101,118,105,111,117,115,83,104,97,114,101,82,101,102,101,114,101,110,99,101,95,16,16,77,111,100,105,102,105,101,100,66,121,68,101,118,105,99,101,94,80,114,111,116,101,99,116,105,111,110,68,97,116,97,95,16,17,85,115,101,76,105,103,104,116,119,101,105,103,104,116,80,67,83,94,83,104,97,114,101,82,101,102,101,114,101,110,99,101,83,85,82,76,95,16,22,67,104,97,105,110,80,97,114,101,110,116,80,117,98,108,105,99,75,101,121,73,68,95,16,24,76,97,115,116,77,111,100,105,102,105,101,100,85,115,101,114,82,101,99,111,114,100,73,68,91,82,101,99,111,114,100,77,116,105,109,101,95,16,21,87,97,110,116,115,80,117,98,108,105,99,83,104,97,114,105,110,103,75,101,121,95,16,22,90,111,110,101,80,114,111,116,101,99,116,105,111,110,68,97,116,97,69,116,97,103,89,87,97,115,67,97,99,104,101,100,95,16,15,67,104,97,105,110,80,114,105,118,97,116,101,75,101,121,90,80,101,114,109,105,115,115,105,111,110,88,82,101,99,111,114,100,73,68,95,16,24,72,97,115,85,112,100,97,116,101,100,83,104,97,114,101,82,101,102,101,114,101,110,99,101,95,16,23,80,114,101,118,105,111,117,115,80,97,114,101,110,116,82,101,102,101,114,101,110,99,101,128,0,8,128,0,9,128,0,128,0,8,128,9,128,0,128,0,128,0,128,0,128,0,128,1,128,11,128,0,128,0,128,0,128,0,128,0,128,19,128,0,128,18,128,0,8,128,0,128,0,128,0,128,16,128,9,8,128,0,8,128,0,16,1,128,2,8,128,0,175,16,20,85,86,87,94,95,101,102,103,109,112,116,119,123,124,128,129,130,134,135,136,85,36,110,117,108,108,95,16,17,78,111,116,101,95,85,115,101,114,83,112,101,99,105,102,105,99,211,88,89,90,91,92,93,86,36,99,108,97,115,115,90,82,101,99,111,114,100,78,97,109,101,86,90,111,110,101,73,68,128,8,128,3,128,4,95,16,140,78,111,116,101,95,85,115,101,114,83,112,101,99,105,102,105,99,58,58,95,51,52,101,99,99,98,101,100,48,101,49,51,49,99,52,99,52,99,98,100,48,98,56,54,101,101,57,55,57,53,55,50,47,80,114,105,118,97,116,101,58,58,78,111,116,101,115,58,58,95,51,52,101,99,99,98,101,100,48,101,49,51,49,99,52,99,52,99,98,100,48,98,56,54,101,101,57,55,57,53,55,50,58,58,48,65,56,57,52,66,65,49,45,67,56,54,67,45,52,69,69,67,45,57,50,57,69,45,51,67,52,56,66,54,53,55,52,48,65,53,211,96,97,88,98,99,100,88,90,111,110,101,78,97,109,101,89,111,119,110,101,114,78,97,109,101,128,5,128,6,128,7,85,78,111,116,101,115,95,16,16,95,95,100,101,102,97,117,108,116,79,119,110,101,114,95,95,210,104,105,106,107,90,36,99,108,97,115,115,110,97,109,101,88,36,99,108,97,115,115,101,115,94,67,75,82,101,99,111,114,100,90,111,110,101,73,68,162,106,108,88,78,83,79,98,106,101,99,116,210,104,105,110,111,90,67,75,82,101,99,111,114,100,73,68,162,110,108,210,113,88,114,115,87,78,83,46,116,105,109,101,35,65,192,70,99,48,185,88,16,128,10,210,104,105,117,118,86,78,83,68,97,116,101,162,117,108,211,88,89,90,91,121,122,128,8,128,12,128,13,95,16,16,95,95,100,101,102,97,117,108,116,79,119,110,101,114,95,95,211,96,97,88,125,126,100,128,14,128,15,128,7,92,95,100,101,102,97,117,108,116,90,111,110,101,95,16,16,95,95,100,101,102,97,117,108,116,79,119,110,101,114,95,95,211,88,89,90,91,132,122,128,8,128,17,128,13,95,16,16,95,95,100,101,102,97,117,108,116,79,119,110,101,114,95,95,84,104,97,107,117,83,50,110,51,0,8,0,17,0,26,0,36,0,41,0,50,0,55,0,73,0,152,0,177,0,205,0,227,0,241,1,5,1,15,1,34,1,46,1,57,1,78,1,119,1,140,1,169,1,180,1,202,1,220,1,230,1,239,1,252,2,31,2,36,2,61,2,80,2,95,2,115,2,130,2,134,2,159,2,186,2,198,2,222,2,247,3,1,3,19,3,30,3,39,3,66,3,92,3,94,3,95,3,97,3,98,3,100,3,102,3,103,3,105,3,107,3,109,3,111,3,113,3,115,3,117,3,119,3,121,3,123,3,125,3,127,3,129,3,131,3,133,3,135,3,137,3,138,3,140,3,142,3,144,3,146,3,148,3,149,3,151,3,152,3,154,3,156,3,158,3,159,3,161,3,184,3,190,3,210,3,217,3,224,3,235,3,242,3,244,3,246,3,248,4,135,4,142,4,151,4,161,4,163,4,165,4,167,4,173,4,192,4,197,4,208,4,217,4,232,4,235,4,244,4,249,5,4,5,7,5,12,5,20,5,29,5,31,5,36,5,43,5,46,5,53,5,55,5,57,5,59,5,78,5,85,5,87,5,89,5,91,5,104,5,123,5,130,5,132,5,134,5,136,5,155,5,160,0,0,0,0,0,0,2,1,0,0,0,0,0,0,0,137,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,164]} (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) (NULL) {"type":"Buffer","data":[]} {"type":"Buffer","data":[10,6,10,4,8,0,16,2,26,38,50,36,10,16,10,2,48,2,18,2,48,1,26,6,10,4,8,0,16,2,10,16,10,2,48,3,18,2,48,1,26,6,10,4,8,0,16,1,26,45,106,43,8,1,26,39,8,0,18,35,34,33,95,51,52,101,99,99,98,101,100,48,101,49,51,49,99,52,99,52,99,98,100,48,98,56,54,101,101,57,55,57,53,55,50,26,48,106,46,8,1,26,42,8,0,18,38,34,36,49,52,52,48,56,68,68,65,45,48,51,50,53,45,52,69,57,50,45,66,49,57,54,45,50,69,49,49,53,54,66,70,68,65,56,55,26,48,106,46,8,1,26,42,8,0,18,38,34,36,67,67,49,56,70,67,57,51,45,49,56,52,66,45,52,53,67,69,45,66,68,49,52,45,49,67,55,65,56,50,55,56,57,66,57,69,34,4,115,101,108,102,42,23,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,78,83,78,117,109,98,101,114,42,23,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,78,83,83,116,114,105,110,103,42,21,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,78,83,85,85,73,68,42,22,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,67,82,84,117,112,108,101,42,40,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,67,82,82,101,103,105,115,116,101,114,77,117,108,116,105,86,97,108,117,101,76,101,97,115,116,42,35,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,67,82,82,101,103,105,115,116,101,114,77,117,108,116,105,86,97,108,117,101,42,21,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,67,82,84,114,101,101,42,25,99,111,109,46,97,112,112,108,101,46,67,82,68,84,46,67,82,84,114,101,101,78,111,100,101,42,23,99,111,109,46,97,112,112,108,101,46,110,111,116,101,115,46,67,82,84,97,98,108,101,42,23,99,111,109,46,97,112,112,108,101,46,110,111,116,101,115,46,73,67,84,97,98,108,101,50,16,204,24,252,147,24,75,69,206,189,20,28,122,130,120,155,158]} (NULL) (NULL)

Second:

Z_PK Z_ENT Z_OPT ZNOTE ZCRYPTOINITIALIZATIONVECTOR ZCRYPTOTAG ZDATA
101 15 1 305 (NULL) (NULL) (NULL)

Edit to add: Other than the log and the copy of the SQLite Database, the output of running the program is empty. Edit2: The process was actually ongoing and I didn't realize. It failed with this error:

/Users/ba/Downloads/apple_cloud_notes_parser-master/lib/AppleNoteStore.rb:546:in `block in rip_note': undefined method `node_id' for #<AppleNote:0x00007f81c0097128> (NoMethodError)
Did you mean?  note_id
threeplanetssoftware commented 4 years ago

@banj Thank you for the second edit. I just pushed a commit to fix that. Hopefully it will complete now.

@dmd My apologies, I keep referring to iOS as this project stemmed from mobile forensics, so most of the testing and development has been aimed at the mobile versions of the app. I believe technically your notes are all in one folder, called 'Notes', with ID 3, per your gist. This is created by default, hence why I'm struggling to recreate this. I will play more on the MacOS version itself, thanks to your insight. With both of you having notes with null data, I might be able to force an example.

Thank you!

threeplanetssoftware commented 4 years ago

Would either of you be able to confirm if the current version complete successfully? It should drop out these notes which don't have an account or folder, but normal ones should be included in the output. Please let me know if there are further errors, I just don't want to close it without positive confirmation.

Thank you!

dmd commented 4 years ago

Success! Thank you so much!

threeplanetssoftware commented 4 years ago

Thank you for the confirmation @dmd! I'll close this issue now, but should you still see issues, please let me know.