Please help! Unknown error when apfs-inspect

[zhanchengqian]

Hi,

My file system got damaged when I was trying to copy large trunk of files from the drive (apfs encrypted MacBook Pro ssd, I put it inside a third-party ssd-to-usb external case). When I was copying, suddenly it says the directory doesn’t exist, and soon I found out the folder I was copying disappeared.

I tried to do ls in terminal, but no result, it disappeared. Then that night, I downloaded disk drill. Disk drill shows the missing folder but after I recovered the files I found out 50% of the files are garbled, not opening. The other 50% is okay, but anything before November 2017 is non openable.

Then I ran across this GitHub page and I tried to use apfs-inspect and apfs-read, but no success. I think both of these returned ABORT unknown error. I don’t know what happened because the message offers little detail.

I mean these files are not something too important, but are memorable things that I kept during college. I really want to recover these data. The garbled portion is around 35GB. Please help me diagnose the issue. Is it possible to get more detail of why apfs-inspect failed?

Also just FYI, the recovered non-openable files are the same size as the good files. I also have very small portion of the garbled data on another drive (known to be good and can be opened), like 1GB out of the garbled 35GB. I compared the folders that are in common. for example, file A (from the other drive) is 35MB, and the garbled file A (what I recovered) is also 35MB. I compared the hex, the hex has nothing in common, totally garbled in the recovered file.

I was reading the drive on another Mac machine, High Sierra, and the ssd is connected thru ssd-usb3.0 external dongle. This MacBook ssd is extracted out of my MacBook, since it got damaged in a recent incident.

Please help me! I don’t know what caused the data damage, and I don’t understand why Disk Drill recovered some good files and some garbled files. Also, why the apfs-inspect and apfs-read didn’t work at all? I will provide screenshots if needed.

Let me know what I should do next.

Thanks, God bless you all

[zhanchengqian]

[zhanchengqian]

also, when I ran fsck it says Object map is invalid.

[jivanpal]

Output of diskutil list and diskutil apfs list would be helpful just to make sure we're not missing anything trivial.

There are a couple of possible (but exceedingly unlikely) error conditions for fopen() that I didn't handle, but have now added code for. Can you checkout the unstable branch, recompile, and try again? That may give us some info as to why you can't open /dev/disk4s1.

The actual "ABORT: Unknown error" message you are seeing appears when fopen() cannot open the file (/dev/disk4s1) for some unknown reason. I don't know what to do with that info (or rather, lack of info). My best guess is that your SATA-to-USB adapter (or something along the way, such as your USB controller/bus) is not working properly, and it may have caused the damage in the first place. Can you attach the drive directly to a machine's SATA port and try again?

I've been meaning to sort out this project for a long time now, but hey, life gets in the way. Hopefully I'll have a week or two in November to dedicate to working on this.

[zhanchengqian]

Thanks for the reply. I tried the unstable branch (on another mac), and still got the "ABORT unknown error". I assume that maybe it has something to do with the USB adapter. The thing is Apple uses its own proprietary hardware interface, so I have no way of directly attach the SSD to a machine's SATA port. It means I can only access it thru an adapter of some sort. I will try to get another adapter from another brand, and see if any thing different will appear.

Also, here are the outputs from diskutil: (shows as disk3s1 on this machine)

Thanks once again for your help, hope all the best!

[zhanchengqian]

I know it's kinda dumb but I ran the repair in disk utility and it gave me this. I don't know whether this would give some more info or not. But I wanna post anyway.

I also tried to unmount other volumes on the same SSD, and I also tried to do this in recovery mode. It always give me the same error message.

[jivanpal]

here are the outputs from diskutil: (shows as disk3s1 on this machine)

disk3s1 is not the APFS partition/container itself; it is the Data volume within your APFS container. The container itself is disk2s2 (see "Physical store" in the output of both commands). Try that first.

I ran the repair in disk utility and it gave me this.

This is telling you that a repair cannot be done whilst you are running macOS normally, since one of the volumes is mounted. I would clone the partition or whole disk to another disk (in case the repair actually worsens things) and then attempt repair from macOS Recovery. hold CMD+R when booting, then open Terminal and find the physical store ID via diskutil list (e.g. as disk2s2 is currently). Unmount all mounted volumes belonging to the APFS container with diskutil unmountDisk diskXsY, and then run e.g fsck_apfs -y /dev/rdiskXsY, where X and Y are from the physical store ID. Note, that's rdisk in the latter command, not just disk.

[zhanchengqian]

Hi jivanpal, thanks for the reply.

Now I have a new adapter, and now I understand that it's actually drive disk2s2. Here is what happened when I ran diskutil verifyVolume:

~ diskutil verifyVolume /dev/disk2s2
Started file system verification on disk2s2
Verifying storage system
Performing fsck_apfs -n -x /dev/disk2s2
Checking the container superblock
Checking the EFI jumpstart record
Checking the space manager
Checking the space manager free queue trees
Checking the object map
error: (oid 0xd55eb) om: invalid o_xid (0x6a1c29)
error: (oid 0xd55eb) om: invalid o_type (0x40000003, expected 0x4000000b)
error: verification/reading of the omap object failed: Illegal byte sequence
The volume /dev/disk2s2 could not be verified completely
Storage system check exit code is 8
Error: -69716: Storage system verify or repair failed
Underlying error: 8

Then I ran these commands, for you reference. (I am a CS grad student, but please forgive me that I don't have a good understanding of what's going on in the Apple APFS system, please point out any problem here)

➜  bin sudo ./apfs-list /dev/disk2s2 0 /Users/Encholl
Opening file at `/dev/disk2s2` in read-only mode ... OK.
Simulating a mount of the APFS container.
Validating checksum of block 0x0 ... OK.
Locating the checkpoint descriptor area:
- Its length is 280 blocks.
- It is contiguous.
- The address of its first block is 0x10647c.
Loading the checkpoint descriptor area into memory ... OK.
Locating the most recent well-formed container superblock in the checkpoint descriptor area:
- It lies at index 201 within the checkpoint descriptor area.
- The corresponding checkpoint starts at index 200 within the checkpoint descriptor area, and spans 2 blocks.

Loading the corresponding checkpoint ... OK.
- There are 68 checkpoint-mappings in this checkpoint.

Reading the Ephemeral objects used by this checkpoint ... OK.
Validating the Ephemeral objects ... OK.
The container superblock states that the container object map has Physical OID 0xd6f38.
Loading the container object map ... OK.
Validating the container object map ... OK.
Reading the root node of the container object map B-tree ... OK.
Validating the root node of the container object map B-tree ... OK.
The container superblock lists 5 APFS volumes, whose superblocks have the following Virtual OIDs:
- 0x403
- 0x54961
- 0x54963
- 0x5ee18
- 0x3cc230

Reading the APFS volume superblocks ... OK.
Validating the APFS volume superblocks ... OK.

 Volume list
================
 0: Macintosh HD - 数据
 1: Preboot
 2: Recovery
 3: VM
 4: Macintosh HD
The volume object map has Physical OID 0xd6edc.
Reading the volume object map ... OK.
Validating the volume object map ... OK.
Reading the root node of the volume object map B-tree ... OK.
Validating the root node of the volume object map B-tree ... OK.
The file-system tree root for this volume has Virtual OID 0x404.
Looking up this Virtual OID in the volume object map ... corresponding block address is 0x6250.
Reading ... validating ... FAILED.
Going back to look at the previous checkpoint instead.
➜  bin

[zhanchengqian]

The output when running apfs-inspect:

inspect-output.txt

Also I tried to read somewhere around 0xa4bf0 ~ 0xa4bf2, which is the address I found some of the missing files via Disk Drill. And I got these two results, and it seems something wrong happened around these addresses, I assume?

➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf0

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf0 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf0:
--------------------------------------------------------------------------------
Stored checksum:    0xbbfcbad22880ea15
OID:                0x5a217bf9d6cd0011
XID:                0x71f860f74621dcab
Storage type:       Ephemeral
Type flags:         Non-persistent (should never appear on disk --- if it does, file a bug against the APFS implementation that created this object)
Type:               Unknown type (0x%08x) --- perhaps this type was introduced in a later version of APFS than that published on 2019-02-27.
Subtype:            Unknown subtype (0x00003c3f) --- perhaps this subtype was introduced in a later version of APFS than that published on 2019-02-27.

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf1

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf1 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf1:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin

[zhanchengqian]

Running the fsck gives me this:

fsck_apfs -y -x /dev/rdisk3s1
Checking the container superblock.
Checking the EFI jumpstart record.
Checking the space manager.
Checking the space manager free queue trees.
Checking the object map.
error: (oid 0xd55eb) om: invalid o_xid (0x6a1c29)
error: (oid 0xd55eb) om: invalid o_type (0x40000003, expected 0x4000000b)
error: verification/reading of the omap object failed: Illegal byte sequence
The volume /dev/rdisk3s1 could not be verified completely.

In all, I know there is /Users/Encholl folder somewhere near 0xa4bf2, (I got that address from Disk Drill), and what kind of "next step" should I take? I hope I've provided enough info for you to find what the problem is. I will provide more info as you request. I lack some knowledge about this kind of file system. I will be very grateful if you could give me some more insight into how I could recover some of the files near that address in the drive.

Thanks for your help again, I think I cannot find another knowledgeable guy like you on the Internet. Hope you have a great day!

[jivanpal]

Your volume's object map block is at address 0xd6edc. This block only stores metadata about the object map; it isn't the object map B-tree itself. However, it tells you where the B-tree is. Reading that block with apfs-read should reveal the address of the object map B-tree's root node. Substitute the address you find for <omap root node address> in the commands below.

---

We want to use apfs-recover to grab your files from /Users/Encholl. Unfortunately, apfs-recover always traverses from / down the tree, but the address that virtual OID 0x404 (the root of the volume) resolves to is an invalid block, so apfs-recover will currently encounter an error because it cannot find /.

We can either:

(a) Check whether virtual OID 0xa4bf2 is really /Users/Encholl, after which we can modify apfs-recover to scan from there rather than from /; or

(b) Fix the blocks which represent the filesystem root, /, /Users, and /Users/Encholl, then use apfs-recover in its current form.

---

For (a), the way forward is:

1. Check block 0xa4bf2 with apfs-read. Please provide this output, as it seems to be missing from the output you pasted in your previous comment.
2. If it really is a filesystem B-tree node, check its contents with apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 <omap root node address>.
3. If it really contains entries from /Users/Encholl, modify apfs-recover to fetch files using this info.
4. Use apfs-recover to grab your files.

---

For (b), the way forward would be:

1. Correct the block with virtual OID 0x404, which represents the root of your Data volume. This block is distinct from the root of your filesystem, /; in fact, this block lists the virtual OID of / (usually 0x2) and another entity called private-data (usually 0x1).
2. Correct the blocks which represent /, /Users, and /Users/Encholl, if necessary.
3. Use apfs-recover to grab your files from /Users/Encholl.

Ideally, steps (1) and (2) would be automated, but these tools are not sophisticated enough to do this (implementing this is the main thing I would like to work on right now), as this can be quite case-specific. The apfs-explore-* commands allow you to explore B-trees (object maps and filesystem records), and apfs-search allows you to scan the disk to look for blocks with certain properties (e.g. if you are hunting for a specific dentry or virtual OID) in order to find the relevant info to do these steps manually instead. Manual fixing of blocks is done with the aid of apfs-modify (use with extreme caution, it WILL irrevocably overwrite data on disk). This whole process can be extremely time-consuming.

The first thing to do would be to see whether a valid instance of the block with virtual OID 0x404 can be found. You can explore the object map B-tree in order to look for the physical address of blocks with a given virtual OID. Do this with apfs-explore-omap-tree /dev/disk2s2 <omap root node address>.

---

Best of luck!

[zhanchengqian]

Thanks for the reply, here is the 0xa4bf2 apfs-read result:

➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf2

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf2 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf2:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.

[zhanchengqian]

To be specific, I tried to read multiple addresses, and arrived at these results:

0xa4bf0 (checksum OID XID looks good) 0xa4bf1 (checksum OID XID All zeroes) 0xa4bf2 (checksum OID XID All zeroes) ... ... 0xd4155 (checksum OID XID All zeroes) 0xd4156 (checksum OID XID All zeroes) 0xd4157 (checksum OID XID looks good)

By "looks good" I mean they have some values instead of zeroes.

[zhanchengqian]

Here is the object map info:

➜  bin sudo ./apfs-read /dev/disk2s2 0xd6edc

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xd6edc ... validating ... OK.

Details of block 0xd6edc:
--------------------------------------------------------------------------------
Stored checksum:    0x5184bf83edf6462b
OID:                0xd6edc
XID:                0x6a1c6d
Storage type:       Physical
Type flags:         (none)
Type:               Object map
Subtype:            (none/invalid)
Flags:
- No flags are set.
Object mappings tree:
- Storage type:         Physical
- Type flags:           (none)
- Type:                 B-tree (root node)
- Object ID:            0xd6ef8
Snapshots tree:
- Storage type:         Physical
- Type flags:           (none)
- Type:                 B-tree (root node)
- Object ID:            0x0
- Number of snapshots:  0 snapshots
- Latest snapshot XID:  0x0
In-progress revert:
- Minimum XID:          0x0
- Maximum XID:          0x0
--------------------------------------------------------------------------------

END: All done.
➜  bin

[zhanchengqian]

So from the following result, I assume that 0xa4bf2 is not really the address I want? It does not bring any useful result and thus it seems it's not the file-system tree root node.

➜  bin sudo ./apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 0xd6ef8

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading the file-system tree root node (block 0xa4bf2) ... validating ... FAILED.

Reading the object map root node (block 0xd6ef8) ... validating ... OK.

Node details:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)
Flags:                          (none)
Number of child levels:         0
Number of keys in this node:    0
Location of table of contents:
- Offset from start of node data area:  0x0 = 0
- Length (bytes):                       0x0 = 0
Location of key–value shared free space:
- Offset from start of keys area:       0x0 = 0
- Length (bytes):                       0x0 = 0
--------------------------------------------------------------------------------

Node has 0 entries, as follows:
Assertion failed: (node->btn_nkeys > 0), function main, file src/apfs-explore-fs-tree.c, line 160.
[1]    20545 abort      sudo ./apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 0xd6ef8

[zhanchengqian]

output relates to 0x404:

➜  bin sudo ./apfs-read /dev/disk2s2 0x404
Password:

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0x404 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0x404:
--------------------------------------------------------------------------------
Stored checksum:    0x3a177b0ee6eef00c
OID:                0x44cd78635880773e
XID:                0x3657f74039de0215
Storage type:       Ephemeral
Type flags:         (none)
Type:               Unknown type (0x%08x) --- perhaps this type was introduced in a later version of APFS than that published on 2019-02-27.
Subtype:            Unknown subtype (0x00000cf8) --- perhaps this subtype was introduced in a later version of APFS than that published on 2019-02-27.

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin sudo ./apfs-explore-omap-tree /dev/disk2s2 0x404

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0x404 ... validating ... FAILED.

Object map trees don't have variable size keys and values ... do they?
➜  bin

[zhanchengqian]

Some info may be useful, related to the o-map

omap-info.txt

So if I understand correctly, the FS tree root has virtual id 0x404, and it resolves to a physical block address which is invalid. And my object map's physical block address is at 0xd6ef8. Upon further inspection, the 0xa4bf2 address does not give me any useful information here. I'm kind of confused. It seems that I can read the object map (which is still kind of good for now), but I do not know there /Users/Encholl is. Then what should I do next? From the object map that I have here, how do I get a human-friendly directory structure here? I don't really know how to get the correct physical block address for the /Users/Encholl. 😣

Please give some insights, thank you!

P.S. in case this might give some more info, other folders under /User are all good (visible from Finder), such as /User/aaa & /User/bbb, and /System /Applications /Library are all good (visible from Finder). The only missing folder is /User/Encholl

Also I don't really want to do apfs-modify, since it might override some data. I really prefer to go for the a) method you mentioned above. Thanks!

[jivanpal]

here is the 0xa4bf2 apfs-read result [...] To be specific, I tried to read multiple addresses, and arrived at these results [...]

This shows that this range of blocks (0xa4bf1 – 0xd4156) has been zeroed out.

I assume that 0xa4bf2 is not really the address I want?

Correct, that block is useless, as there is no data there; it has been zeroed out.

---

Here is the object map info:

This shows that 0xd6ef8 is the address of the Omap B-tree's root node (see "Object mappings tree, Object ID").

output relates to 0x404: sudo ./apfs-read /dev/disk2s2 0x404

0x404 is a virtual object ID; it is not an address on disk (which the APFS spec refers to as physical object ID). apfs-read cannot tell you anything about virtual OID 0x404 without you knowing the address of an instance of the virtual object with that OID, which is what the omap tree allows you to find.

sudo ./apfs-explore-omap-tree /dev/disk2s2 0x404

apfs-explore-omap-tree is to be used as follows: apfs-explore-omap-tree <container> <omap root node address>

In your case, do apfs-explore-omap-tree /dev/disk2s2 0xd6ef8, and the contents of the Omap B-tree's root node will be shown to you. From there, you can traverse the tree manually by entering the index of a node in order to find entries for the virtual OID you are looking for (0x404).

Some info may be useful, related to the o-map: omap-info.txt

Great! This is what I was referring to just above. After entering 0 at the first list, you descended the first node, which revealed the following:

Child node resides at adress 0xd6f24. Reading ... validating ... OK.

Node has 141 entries, as follows:
-   0:  OID =     0x404  ||  XID =  0x6a1c6d
-   1:  OID =     0x47f  ||  XID =  0x3010f7
[...]
Choose an entry [0-140]: _

At this point, enter 0 again to descend the node labelled 0x404, which will take you further down the tree, closer to where the leaf nodes for virtual OID 0x404 are. Given that 0x404 is the label on the node, you should just have to enter 0 a few more times until you see all the leaf nodes we're interested in.

Definitely seems like a good idea to let the user just specify the virtual OID and automatically traverse the tree for them; I'll add that feature to the roadmap.

---

So if I understand correctly, the FS tree root has virtual id 0x404, and it resolves to a physical block address which is invalid.

We have not yet resolved virtual OID 0x404 to an address on disk (a.k.a. physical OID). You must descend the tree as described just above in order to find such an address (or addresses).

Also I don't really want to do apfs-modify, since it might override some data. I really prefer to go for the a) method you mentioned above. Thanks!

Unfortunately, (a) is no longer an option, since we now know that block 0xa4bf2 is completely gone. Of course I understand not wanting to overwrite data on the problematic disk. I would advise that you clone the disk/partition (e.g. using dd) before doing any further work. That way, we can revert to the current state if things go awry.

You should probably also add an entry to your working environment's /etc/fstab to prevent it from mounting the drive/partition with write privileges, because if we make alterations to the partition but they are not sufficient for the APFS driver to act on the disk properly, the system will almost certainly rewrite blocks that we just modified, as well as creating new APFS transactions, which will just put us back at square one.

I am planning on solving this issue of needing to modify the underlying disk by using FUSE or a custom filesystem driver to allow mounting a problematic APFS container virtually with the aid of a supplemental file (sort of a config file) that specifies the contents of problematic blocks that we wish to replace ad hoc. In this way, the underlying disk would not need to be modified. It may be worth waiting for this feature, as it is near the top of my list, but there is no ETA on this.

[zhanchengqian]

Thanks for the reply,

I will do a backup of the drive soon. (I've already done the /etc/fstab thing)

Btw, when I dd the drive to an disk image, and maybe later restore it back to the drive, is that a bit-to-bit restoration? Or maybe I should get another 500GB drive and use disk utility's restore feature to bit-to-bit copy the whole thing to the new drive? (this way I will need to get another 500GB SSD for $100, but everything should be precisely copied)

So I did what you told me, I inputed 0 all the way and got these: omap-info-more.txt

In the end, I got 60 entries, which shows a list of actual OIDs. Does these represent some sort of file directory?

Lastly, you mentioned Correct, that block is useless, as there is no data there; it has been zeroed out. , does this mean that these are index, and these indexes got wiped, but the actual file they point to, are still in the drive itself? Or does it mean that, these blocks are wiped, and the related files are lost forever?

P.S. (another edit) I was reading this post the other day, (Post link) You mentioned except for the particular leaf node that contains the file-system records for /Users/jivan This seems quite interesting. I assume that in my case, there is no way to find the /Users/Encholl ? Anyways, since you mentioned that we need to do the method b), then I will prepare for it. I feel like it's the block that represents /Users/Encholl got wiped, which results the fact that I cannot see the folder in the Finder. So we need to reconstruct the blocks, right?

I really appreciate your help here. I know sometimes my questions can be a bit silly. Thanks for you patience. I want to get you a couple cups of coffee/tea if I can get anything useful out of this drive. 😎

[jivanpal]

Btw, when I dd the drive to an disk image, and maybe later restore it back to the drive, is that a bit-to-bit restoration?

Yeah, you could dd the partition/disk to a file on another drive (disk image), or dd the entire drive to another drive (disk clone). Either will suffice, as the disk image/clone's contents will be identical in both cases.

maybe I should get another 500GB drive and use disk utility's restore feature[?]

I'm not familiar with that feature of Disk Utility, so can't confirm whether that actually does a bit-for-bit copy or not.

So I did what you told me, I inputed 0 all the way and got these.

Great, that shows all the entries in the leftmost leaf node of the Omap B-tree. Unfortunately, in this case, it seems that all the of target blocks no longer have the expected data; all the (   NO) entries show that the OID and XID stored in the target blocks don't match up with the OID and XID in the corresponding Omap entry. We will have to resort to apfs-search to find particular dentries by scanning the entire partition. We can usually narrow down the search space to a small set of blocks near the front of the partition, since the addresses at which dentries are stored are usually in that region. This will increase the speed of subsequent searches if we need to do more than one search, which is very likely. In my experience, such a reduced search takes a few minutes and yields good results.

[Do] these represent some sort of file directory?

No, the filesystem (FS) B-tree stores the dentries (directory entries). Each object in the filesystem (e.g. a directory or file) has a corresponding set of entries ("dentries" or "FS records") in the FS B-tree. The file extent records map each filepath to a Virtual OID. The Omap B-tree then maps each (Virtual OID, XID) pair to a Physical OID (an address on disk).

Lastly, you mentioned, "Correct, that block is useless, as there is no data there; it has been zeroed out." Does this mean that these are index, and these indexes got wiped, but the actual file they point to, are still in the drive itself?

Yes, it is just the references to the objects' locations on disk (in particular, the Physical OIDs that would be stored in the Omap entries) that we haven't got. The FS B-tree entries are hopefully still present, but we are yet to try looking for those directly; we were first hoping to find good Omap entries that would tell us where the FS B-tree nodes are, but no such luck here. If the Omap B-tree was intact, then we could explore the FS B-tree using apfs-explore-fs-tree — we can't do that currently, because the nodes of the FS B-tree are referenced using Virtual OIDs, which it is the Omap's job to resolve to Physical OIDs, but the Omap is in a bad state.

The actual file extents for your files (the blocks which store the contents of your files) are most likely fine. However, due to the fragmented nature of APFS (that is, files are rarely stored in contiguous disk address ranges), tools such as TestDisk likely won't recover your files in a useful manner unless they happen to be stored contiguously (e.g. because a contiguous set of disk addresses were free when the file was written, and the file has not been modified since then). Finding/reconstructing the FS B-tree and Omap B-tree, where possible, is the key to getting your files back.

---

You mentioned, "except for the particular leaf node that contains the file-system records for /Users/jivan".

I got unlucky in that the FS B-tree node for /Users/jivan was corrupted, but lucky in that only certain small sets of blocks were zeroed out in my case, so the dentries for e.g. /Users/jivan/Documents were still present on disk and could be found with the aid of apfs-search. I then used the knowledge of where those dentries were to reconstruct the corrupted node which stored the dentries for /Users/jivan.

I really appreciate your help here. I know sometimes my questions can be a bit silly. Thanks for you patience. I want to get you a couple cups of coffee/tea if I can get anything useful out of this drive. 😎

Not at all, more than happy to help! Just strapped for time nowadays. Feel free to donate via PayPal, I'd greatly appreciate whatever you're willing to part with. 🙂

---

I'm busy for the rest of this week, then hopefully will be working on this project next week. I will probably be cleaning up the Git repo and then working on apfs-search, which should help you complete your next steps. If you can bear with until then (21 November), then we can probably continue with your issue. I may open a support forum for this project so that GitHub can be used exclusively for bug reports, and then move this conversation there.

[zhanchengqian]

Thanks for the reply,

Now after reading your reply here, I sort of understand the big picture of what's going on here. So we must find a way to restore the FS B-tree and the Omap B-tree in order to restore the file directory. However, due to the fragmented nature of APFS (that is, files are rarely stored in contiguous disk address ranges), tools such as TestDisk likely won't recover your files in a useful manner this explains why I used Disk Drill to recover files but only got 5% usable data, and the rest 95% is all garbled stuff.

Just wondering, what if the FS B-tree and Omap B-tree are both heavily damaged? Will this mean that there is no other way to recover any file in this case? I hope at least parts of the FS B-tree survives so that we can trace some files down.

I just made a small donation to your Paypal. (not too much but shows my respect) I hope you can work on the project during your spare time, since I believe this project can save many people's data in the future. And I will donate later, if everything goes well.

I will be busy moving to a new place in the next week. So don't worry about it, I will be waiting for you to work on the apfs-search feature. And I will also find some time to clone the drive. (Btw if I clone this drive to drive-B, your tool should work exactly the same on drive-B, right? I don't really want to wipe this particular drive, so I may clone it to drive-B and use your tool on drive-B and wipe drive-B if anything goes wrong. This way, I am more comfortable, I guess)

Wish all the best to you, and wish you have some time in the next few weeks to work on this project.

[jivanpal]

Just wondering, what if the FS B-tree and Omap B-tree are both heavily damaged? Will this mean that there is no other way to recover any file in this case?

Yes, but it seems very unlikely that this is the case. A common issue when filesystems such as NTFS and EXT4 become corrupted is that no valid superblocks and dentry tables can be found, but with APFS's copy-on-write behaviour, this is much more unlikely because multiple instances of each object from different points in time are present on the disk in different locations.

I just made a small donation to your Paypal.

Thank you very much! I saw it this morning 😊

And I will also find some time to clone the drive. (Btw if I clone this drive to drive-B, your tool should work exactly the same on drive-B, right? I don't really want to wipe this particular drive, so I may clone it to drive-B and use your tool on drive-B and wipe drive-B if anything goes wrong. This way, I am more comfortable, I guess)

That sounds good. Yes, the tool will work even in that case, as there is nothing special about where the data is stored, only that it is stored in data structures of the form described in the spec. There is no practical difference between a block device abstracted to a file like /dev/disk2s2, and a regular file like ~/Desktop/apfs.img; only the contents are important.

[zhanchengqian]

Hi jivanpal,

I was quite busy during the past few weeks. I cloned the drive to another external drive using dd. (Took around 10 hours) And I notice that you added apfs-modify. My question would be, what should I do next? Could you please provide me with some information about the next steps? Any advice would be greatly appreciated.

Best regards.

[jivanpal]

Hi, I am steadily finding time to continue working on this project. I am current consolidating/refactoring the different tools into one program, then I should be working on search, and probably also modify so that it doesn't actually modify the disk contents, but instead virtually modifies them to achieve the same effect. This should provide you with an easier/safer way to recover your data. With my schedule being what it is, I expect there won't be much progress before the new year.

[zhanchengqian]

Hi jivanpal,

Glad to hear back from you. I fully understand that you can be quite busy during this time of the year. And I really appreciate what you've added to the project during the past few weeks. Just wondering, is there any trial-and-error that I can do, using what we've got here? I don't want to just sit here and wait 😎. Since I've already cloned the drive, I guess I can just play around with it and see if it works? What do you think?

Best regards.

[zhanchengqian]

Happy new year jivanpal!

I hope everything goes well with you. I was kind of busy during the past few weeks and didn't check back at this project. I recently tried a couple things I found on GitHub, some of which appears too simple to solve my problem. It just appears to be that your tool (with some future development) will come very close to repair my corrupted drive. (And you might be the only person on this planet that's willing to help me at this moment 😁) I always blame myself for accidentally corrupt this drive (and not backing it up beforehand), whenever I think of it. It's been at the back of my mind, from time to time.

I did not quite understand how the corruption happened in the first place, though, since I've always had the impression that Apple builds its OS and file system in a very robust way. But I do think, Apple should come up with some repair tool regarding the APFS format since I've seen quite a lot corruption cases on the Internet recently.

I fully understand that you can be quite busy with your daily work & other things, just as I do. Hopefully, you can find some time to work on this project this year (maybe some development on the omap-repair part?) I will keep supporting you, since you're kind and also quite knowledgeable. And personally I don't really want to let go those data 😭

Oh btw, do you have a timeline for developments of this project this year? Any reply will be greatly appreciated (haven't heard back from you for a while, haha) Again, happy new year and hope all the best!

Ah forgot to mention, I did a small donation to your PayPal, please check it. Thanks! (Maybe get some snacks or a cup of tea, haha.)

Best regards.

[jivanpal]

Thank you very much for your recent donation, I did receive it! I will be resuming work on this tomorrow, spending a couple hours every day on the project, so progress should be steady henceforth. I will probably publicise a Trello board soon with a timeline for features and fixes.

Happy new year!

[zhanchengqian]

Glad to hear back from you. Hopefully everything goes well with you! I was reading Apple's APFS doc this morning and getting some ideas about APFS as a whole. Just one quick question for you, last time I was using DiskDrill, and it would list all directory just fine, but the files it extracted are almost all garbage (but with correct file size). In this case, I would assume it's DiskDrill's implementation that caused this problem? Do you think your command line tool can extract files just fine? Or do you think we should reconstruct the omap and mount the disk in Finder so that we can copy files from Finder that way?

When thinking about it, I feel like the reason why files from recent time (late 2017 - now) are recovered correctly, whereas files from earlier time (2012 - mid 2017) are recovered incorrectly, might due to the fact that maybe I upgraded the OS sometime in late 2017 and the OS converted the HFS disk to APFS disk. And maybe DiskDrill's implementation is able to deal with files that are written in native APFS form, rather than dealing with files that got converted from HFS to APFS? That's my guess, what do you think?

Best regards.

[jivanpal]

last time I was using DiskDrill, and it would list all directory just fine, but the files it extracted are almost all garbage (but with correct file size). In this case, I would assume it's DiskDrill's implementation that caused this problem?

If it can list directories and filenames properly, then their encryption implementation is at least working in that regard. I would assume the issue is with file data decryption (though that seems unlikely), but I'm afraid you'd have to ask the Disk Drill team for support with that issue. It could simply be that your file data has been clobbered.

maybe DiskDrill's implementation is able to deal with files that are written in native APFS form, rather than dealing with files that got converted from HFS to APFS?

I highly doubt this; to my knowledge, file data and metadata has always been stored in the same way, with the only exception being the way that filenames are stored — j_drec_key_t is the older structure, j_drec_hashed_key_t is the newer structure, but implementations with support for the latter should support both.

[zhanchengqian]

Thanks for your reply, according to their website:

Why can’t I open some recovered files? The sad fact of the matter is that some recovered files are corrupt. Disk Drill does its best to the collect the pieces of a file and reassemble it, but sometimes it just doesn’t work. A piece of the file may have been overwritten or resided in a section of the disk with bad sectors. When a file is missing a part, it is called “corrupt” and often it will not open in its native application.

I understand that some parts of the disk might be corrupt beyond repair, but I don't think over 90% of the data are all corrupt since you also mentioned that it's copy-on-write kinda deal. I mean it's possible that 10% of the data got totally messed up, but when 90% of the files are non-recoverable, I hope at least the data is fine and it's something wrong with DiskDrill.

What do you think? Would your apfs-recover maybe get some useful data out of it? (Maybe there's something wrong with DiskDrill and it overwrites parts of my disk when scanning it? This would be my concern at this point. Uh, I don't know, dude.)

Best regards. Hope all the best!

[zhanchengqian]

Oh one more thing here, I have a small portion of files that I have copied years before the incident, and after comparing the two files (one I stored elsewhere, and the other one is recovered using DiskDrill), I found the compositions of the files are wildly different. Here are two files (one good and one bad) for your inspection: two files.zip

Maybe somehow the extraction process got mislead and got offset by a few blocks? I don't know.

Here is the full output of apfs-inspect: (I notice that last time I didn't post the full inspect result, because my terminal caps at 1000 lines) apfs-inspect-full.txt