Segmentation fault in drat list

akira-okumura commented 2 years ago

I got a segmentation fault in drat list for one of directories.

$ sudo ./drat list --container /dev/disk2s2 --volume 1 --path /Nagoya\ 2021-.photoslibrary/resources/derivatives
Opening `/dev/disk2s2` in read-only mode ... OK.
Determining block size ... 4096 bytes.

Finding most recent container superblock:
- Reading block 0x0 ... validating ... OK.
- Loading checkpoint descriptor area ... OK.
- Searching checkpoint descriptor area ... found most recent container superblock at index 159, its XID is 0x4921b.
Finding container's omap tree:
- Container's omap has Physical OID 0x624395.
- Reading block 0x624395 ... validating ... OK.
- Container's omap tree has Physical OID 0x624484.
- Reading block 0x624484 ... validating ...OK.

Finding volume 1's superblock:
- Volume 1 has Virtual OID 0x402 ... maps to block 0x624353 with XID 0x4921b.
- Reading block 0x624353 ... validating ... OK.
- Volume name: G-DRIVE
Finding volume's omap tree:
- Volume's omap has Physical OID 0x5d6493.
- Reading block 0x5d6493 ... validating ... OK.
- Volume's omap tree has Physical OID 0x5d7962.
- Reading block 0x5d7962 ... validating ... OK.

Finding volume's filesystem tree:
- Filesystem tree has Virtual OID 0x404 ... maps to block 0x5df33f.
- Reading block 0x5df33f ... validating ... OK.

Navigating to path `/Nagoya 2021-.photoslibrary/resources/derivatives` ... its FSOID is 0x6793a5.
Finding records for FSOID 0x6793a5 ... OK.
zsh: segmentation fault  sudo ./drat list --container /dev/disk2s2 --volume 1 --path

Here is the output of the parent directory.

$ sudo ./drat list --container /dev/disk2s2 --volume 1 --path /Nagoya\ 2021-.photoslibrary/resources/           
Opening `/dev/disk2s2` in read-only mode ... OK.
Determining block size ... 4096 bytes.

Finding most recent container superblock:
- Reading block 0x0 ... validating ... OK.
- Loading checkpoint descriptor area ... OK.
- Searching checkpoint descriptor area ... found most recent container superblock at index 159, its XID is 0x4921b.
Finding container's omap tree:
- Container's omap has Physical OID 0x624395.
- Reading block 0x624395 ... validating ... OK.
- Container's omap tree has Physical OID 0x624484.
- Reading block 0x624484 ... validating ...OK.

Finding volume 1's superblock:
- Volume 1 has Virtual OID 0x402 ... maps to block 0x624353 with XID 0x4921b.
- Reading block 0x624353 ... validating ... OK.
- Volume name: G-DRIVE
Finding volume's omap tree:
- Volume's omap has Physical OID 0x5d6493.
- Reading block 0x5d6493 ... validating ... OK.
- Volume's omap tree has Physical OID 0x5d7962.
- Reading block 0x5d7962 ... validating ... OK.

Finding volume's filesystem tree:
- Filesystem tree has Virtual OID 0x404 ... maps to block 0x5df33f.
- Reading block 0x5df33f ... validating ... OK.

Navigating to path `/Nagoya 2021-.photoslibrary/resources/` ... its FSOID is 0x678eb0.
Finding records for FSOID 0x678eb0 ... OK.
Filesystem object has 8 records, as follows:
- INODE
- XATTR || name = com.apple.quarantine
- DIR REC || Dirctry || target ID = 0x678eb1 || name = renders
- DIR REC || Dirctry || target ID = 0x6793a5 || name = derivatives
- DIR REC || Dirctry || target ID = 0x67cedf || name = streams
- DIR REC || Dirctry || target ID = 0x67cee0 || name = journals
- DIR REC || Dirctry || target ID = 0x67cf02 || name = cpl
- DIR REC || Dirctry || target ID = 0x67cf0f || name = caches

Please let me know if I can check anything more.

akira-okumura commented 2 years ago

Another seg fault.

$ sudo ./drat list --container /dev/disk2s2 --volume 1 --path /Nagoya\ 2021-.photoslibrary/originals/E                                          
Password:
Opening `/dev/disk2s2` in read-only mode ... OK.
Determining block size ... 4096 bytes.

Finding most recent container superblock:
- Reading block 0x0 ... validating ... OK.
- Loading checkpoint descriptor area ... OK.
- Searching checkpoint descriptor area ... found most recent container superblock at index 159, its XID is 0x4921b.
Finding container's omap tree:
- Container's omap has Physical OID 0x624395.
- Reading block 0x624395 ... validating ... OK.
- Container's omap tree has Physical OID 0x624484.
- Reading block 0x624484 ... validating ...OK.

Finding volume 1's superblock:
- Volume 1 has Virtual OID 0x402 ... maps to block 0x624353 with XID 0x4921b.
- Reading block 0x624353 ... validating ... OK.
- Volume name: G-DRIVE
Finding volume's omap tree:
- Volume's omap has Physical OID 0x5d6493.
- Reading block 0x5d6493 ... validating ... OK.
- Volume's omap tree has Physical OID 0x5d7962.
- Reading block 0x5d7962 ... validating ... OK.

Finding volume's filesystem tree:
- Filesystem tree has Virtual OID 0x404 ... maps to block 0x5df33f.
- Reading block 0x5df33f ... validating ... OK.

Navigating to path `/Nagoya 2021-.photoslibrary/originals/E` ... 
get_fs_records: File-system root B-trees don't have fixed size keys and values ... do they?
its FSOID is 0x67f2d2.
Finding records for FSOID 0x67f2d2 ... 
get_fs_records: File-system root B-trees don't have fixed size keys and values ... do they?
OK.
zsh: segmentation fault  sudo ./drat list --container /dev/disk2s2 --volume 1 --path

jivanpal commented 2 years ago

I think this is related to whatever caused the segfault in #30, and that my claim there that it was due to the user trying to recover a directory was incorrect. Curiously, in both cases, the segfault message does not show the entire command that was run. My first thought is an issue with argument parsing, though Drat's output seems to show that argparsing occurred correctly. Finding records for FSOID 0x6793a5 ... OK. immediately followed by a segfault suggests to me that the output of get_fs_records() is malformed. I will have to explore further.

In the meantime, can you try listing the items directly using their FSOIDs, 0x6793a5 and 0x67f2d2, with the --fsoid option rather than --path? I expect the same segfaults, but perhaps we'll get something different.

If you are willing to provide a dump of the disk, that will allow me to take a look at the data structures more closely, though i understand if you don't want to do that — I really need to create a privacy policy sometime!

akira-okumura commented 2 years ago

$ sudo ./drat list --container /dev/disk2s2 --volume 1 --fsoid 0x6793a5                                         
Opening `/dev/disk2s2` in read-only mode ... OK.
Determining block size ... 4096 bytes.

Finding most recent container superblock:
- Reading block 0x0 ... validating ... OK.
- Loading checkpoint descriptor area ... OK.
- Searching checkpoint descriptor area ... found most recent container superblock at index 159, its XID is 0x4921b.
Finding container's omap tree:
- Container's omap has Physical OID 0x624395.
- Reading block 0x624395 ... validating ... OK.
- Container's omap tree has Physical OID 0x624484.
- Reading block 0x624484 ... validating ...OK.

Finding volume 1's superblock:
- Volume 1 has Virtual OID 0x402 ... maps to block 0x624353 with XID 0x4921b.
- Reading block 0x624353 ... validating ... OK.
- Volume name: G-DRIVE
Finding volume's omap tree:
- Volume's omap has Physical OID 0x5d6493.
- Reading block 0x5d6493 ... validating ... OK.
- Volume's omap tree has Physical OID 0x5d7962.
- Reading block 0x5d7962 ... validating ... OK.

Finding volume's filesystem tree:
- Filesystem tree has Virtual OID 0x404 ... maps to block 0x5df33f.
- Reading block 0x5df33f ... validating ... OK.

Finding records for FSOID 0x6793a5 ... OK.
zsh: segmentation fault  sudo ./drat list --container /dev/disk2s2 --volume 1 --fsoid 0x6793a5

$ sudo ./drat list --container /dev/disk2s2 --volume 1 --fsoid 0x67f2d2
Opening `/dev/disk2s2` in read-only mode ... OK.
Determining block size ... 4096 bytes.

Finding most recent container superblock:
- Reading block 0x0 ... validating ... OK.
- Loading checkpoint descriptor area ... OK.
- Searching checkpoint descriptor area ... found most recent container superblock at index 159, its XID is 0x4921b.
Finding container's omap tree:
- Container's omap has Physical OID 0x624395.
- Reading block 0x624395 ... validating ... OK.
- Container's omap tree has Physical OID 0x624484.
- Reading block 0x624484 ... validating ...OK.

Finding volume 1's superblock:
- Volume 1 has Virtual OID 0x402 ... maps to block 0x624353 with XID 0x4921b.
- Reading block 0x624353 ... validating ... OK.
- Volume name: G-DRIVE
Finding volume's omap tree:
- Volume's omap has Physical OID 0x5d6493.
- Reading block 0x5d6493 ... validating ... OK.
- Volume's omap tree has Physical OID 0x5d7962.
- Reading block 0x5d7962 ... validating ... OK.

Finding volume's filesystem tree:
- Filesystem tree has Virtual OID 0x404 ... maps to block 0x5df33f.
- Reading block 0x5df33f ... validating ... OK.

Finding records for FSOID 0x67f2d2 ... 
get_fs_records: File-system root B-trees don't have fixed size keys and values ... do they?
OK.
zsh: segmentation fault  sudo ./drat list --container /dev/disk2s2 --volume 1 --fsoid 0x67f2d2

How do I dump the disk data?

akira-okumura commented 2 years ago

$ sudo lldb --file ./drat                                                          
(lldb) target create "./drat"
Current executable set to './drat' (x86_64).
(lldb) run list --container /dev/disk2s2 --volume 1 --fsoid 0x6793a5
Process 17964 launched: '/Users/oxon/drat/drat' (x86_64)
Opening `/dev/disk2s2` in read-only mode ... OK.
(snip)
Finding records for FSOID 0x6793a5 ... OK.
Process 18598 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000100002acf drat`cmd_list(argc=8, argv=0x00007ffeefbffae8) at list.c:457:5
   454      printf("OK.\n");
   455  
   456      size_t num_records = 0;
-> 457      for (j_rec_t** cursor = fs_records; *cursor; cursor++) {
   458          num_records++;
   459      }
   460      printf("Filesystem object has %zu records, as follows:\n", num_records);
Target 0: (drat) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000100002acf drat`cmd_list(argc=8, argv=0x00007ffeefbffae8) at list.c:457:5
    frame #1: 0x0000000100001589 drat`main(argc=8, argv=0x00007ffeefbffae8) at drat.c:74:12
    frame #2: 0x00007fff618d93d5 libdyld.dylib`start + 1
    frame #3: 0x00007fff618d93d5 libdyld.dylib`start + 1
(lldb) print fs_records
(j_rec_t **) $0 = 0x0000000000000000

$ sudo lldb --file ./drat
(lldb) target create "./drat"
Current executable set to './drat' (x86_64).
(lldb) run list --container /dev/disk2s2 --volume 1 --fsoid 0x67f2d2
Process 17982 launched: '/Users/oxon/drat/drat' (x86_64)
Opening `/dev/disk2s2` in read-only mode ... OK.
Determining block size ... 4096 bytes.
(snip)
Finding records for FSOID 0x67f2d2 ... 
get_fs_records: File-system root B-trees don't have fixed size keys and values ... do they?
OK.
Process 18586 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000100002acf drat`cmd_list(argc=8, argv=0x00007ffeefbffae8) at list.c:457:5
   454      printf("OK.\n");
   455  
   456      size_t num_records = 0;
-> 457      for (j_rec_t** cursor = fs_records; *cursor; cursor++) {
   458          num_records++;
   459      }
   460      printf("Filesystem object has %zu records, as follows:\n", num_records);
Target 0: (drat) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000100002acf drat`cmd_list(argc=8, argv=0x00007ffeefbffae8) at list.c:457:5
    frame #1: 0x0000000100001589 drat`main(argc=8, argv=0x00007ffeefbffae8) at drat.c:74:12
    frame #2: 0x00007fff618d93d5 libdyld.dylib`start + 1
    frame #3: 0x00007fff618d93d5 libdyld.dylib`start + 1
(lldb) print fs_records
(j_rec_t **) $0 = 0x0000000000000000

akira-okumura commented 2 years ago

I use origin/main on macOS 10.14.6.

$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

jivanpal commented 2 years ago

How do I dump the disk data?

The B-tree data structures are spread across the partition, but usually almost all of them are within the first 2%–5% of it. Email the output of diskutil apfs list to me at jivan.pal@gmail.com and I can tell you how to send me that data securely and directly over the internet using dd.

akira-okumura commented 2 years ago

diskutil cannot find /dev/disk2 while drat can see it.

$ diskutil apfs list                                                   
APFS Containers (2 found)
|
+-- Container disk1 BC604240-7DA3-4989-84CD-815F1557DE68
|   ====================================================
|   APFS Container Reference:     disk1
|   Size (Capacity Ceiling):      2000662327296 B (2.0 TB)
|   Capacity In Use By Volumes:   149109809152 B (149.1 GB) (7.5% used)
|   Capacity Not Allocated:       1851552518144 B (1.9 TB) (92.5% free)
|   |
|   +-< Physical Store disk0s2 566440DB-A992-4B4E-9784-8EED5259D020
|   |   -----------------------------------------------------------
|   |   APFS Physical Store Disk:   disk0s2
|   |   Size:                       2000662327296 B (2.0 TB)
|   |
|   +-> Volume disk1s1 EDD6A644-9D0A-4D3C-9550-A3E6DD64925A
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s1 (No specific role)
|   |   Name:                      Macintosh HD (Case-insensitive)
|   |   Mount Point:               /
|   |   Capacity Consumed:         145031593984 B (145.0 GB)
|   |   FileVault:                 Yes (Unlocked)
|   |
|   +-> Volume disk1s2 F5A9062C-601F-43FE-A082-D0CBBA5358CE
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s2 (Preboot)
|   |   Name:                      Preboot (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         45965312 B (46.0 MB)
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s3 599BCC60-1FDF-495C-9A66-9C61A6A216BF
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk1s3 (Recovery)
|   |   Name:                      Recovery (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         510476288 B (510.5 MB)
|   |   FileVault:                 No
|   |
|   +-> Volume disk1s4 6F6BF631-63B6-4FEE-BC84-5CBD615C610B
|       ---------------------------------------------------
|       APFS Volume Disk (Role):   disk1s4 (VM)
|       Name:                      VM (Case-insensitive)
|       Mount Point:               /private/var/vm
|       Capacity Consumed:         3221258240 B (3.2 GB)
|       FileVault:                 No
|
+-- Container ERROR -69808
    ======================
    APFS Container Reference:     disk3
    Size (Capacity Ceiling):      ERROR -69620
    Capacity In Use By Volumes:   ERROR -69620
    Capacity Not Allocated:       ERROR -69620
    |
    +-< Physical Store disk2s2 E2F68F91-FFB7-4CEB-9314-49918264922C
    |   -----------------------------------------------------------
    |   APFS Physical Store Disk:   disk2s2
    |   Size:                       6000965369856 B (6.0 TB)
    |
    +-> No Volumes

jivanpal commented 2 years ago

Thanks, I have sent an email to your GitHub listed email address with further instructions.

akira-okumura commented 2 years ago

Navigating to path `/Nagoya 2021-.photoslibrary/originals/E` ... 
get_fs_records: File-system root B-trees don't have fixed size keys and values ... do they?
its FSOID is 0x67f2d2.
Finding records for FSOID 0x67f2d2 ... 
get_fs_records: File-system root B-trees don't have fixed size keys and values ... do they?
OK.
zsh: segmentation fault  sudo ./drat list --container /dev/disk2s2 --volume 1 --path

The path /Nagoya 2021-.photoslibrary/originals/E can be read by Disk Drill preview, which I have not purchased yet.

jivanpal / drat

Segmentation fault in drat list #35