Sorting root directory on exFAT devices causes corruption

Dsibe commented 1 month ago

Hi! I've encountered an issue related to sorting items on exFAT file systems, specifically on the root directory. After calling the .sort() method on the root of an exFAT device, it enters "write-protected" mode, essentially becoming unusable.

Problem description

When sorting the root directory of an exFAT-formatted drive using the .sort() method, the drive becomes write-protected.
After sorting, it's impossible to create new folders, rename files, delete or create new files, or modify existing ones. This behavior is reflected across all the system, and persists after reboot.
The drive becomes completely read-only.

Ways to lift write protection

I found the following methods can lift this write-protected mode:

Formatting the volume
Running the "Error checking" tool from Windows Explorer
Running CHKDSK /F in the command prompt

I found these methods to NOT help with the write-protected mode (all of those are commonly recommended when you have issues related to 'locking' or making your devices read-only):

Using diskpart and executing attributes clear readonly.
Changing HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\StorageDevicePolicies key in Registry.

Minimum reproducible example

Create a partition or insert a USB device/SDcard/whatever and assign a drive letter to it.
- In my example, I'm using a 1GB partition of my internal HDD with drive letter H:.
Format the drive to exFAT using the following command:
```
format H: /FS:exFAT /V:Test /Q /Y
```
(I use this command as Windows sometimes doesn't show exFAT as an option when using "Disk Management" or formatting from Explorer.)

Run this program:

import os
os.environ["FATTOOLS_DEBUG"] = "15" # enable debug

import traceback
from FATtools.Volume import vopen, vclose

# This assumes that 'H:' is a freshly formatted, empty drive.
# Let's create two random files and then "sort" the root.

with open("H:\\1.txt", "w") as f:
    f.write("1")

with open("H:\\2.txt", "w") as f:
    f.write("2")

# Run the sorting
root = vopen("H:", "r+b")
root.sort()
vclose(root)

# Now, attempt any write operation...

try:
    with open("H:\\test.txt", "w") as f:
        f.write("test")
except:
    print("Error writing.")
    traceback.print_exc()
# This will raise: PermissionError: [Errno 13] Permission denied: 'H:\\1.txt'

try:
    os.rename("H:\\1.txt", "H:\\a.txt")
except:
    print("Error renaming.")
    traceback.print_exc()
# This will raise: PermissionError: [WinError 19] The media is write protected: 'h:\\1.txt' -> 'H:\\a.txt'

Running this code, I get the following output (with DEBUG enabled): see this gist

Observations

This issue only occurs when sorting items on the root of the drive.
Sorting items in subfolders (e.g., H:\example_folder) does not trigger this issue (by using root.opendir followed by a .sort()).
Running chkdsk H: /f resolves the write-protection. This suggests that the issue might be related to corruption of the file system or root directory, see output below:

>> chkdsk H: /f
The type of the file system is exFAT.
Volume Serial Number is 9225-3A34
Windows is verifying files and folders...
Volume label is Test.
Volume label is Test.
The upcase file content is incorrect.
The upcase file content is incorrect.
Corruption was found while examining files and directories.
Corruption was found while examining the volume bitmap.
File and folder verification is complete.

Windows has made corrections to the file system.
No further action is required.

    981824 KB total disk space.
       192 KB in 6 files.
       224 KB in 7 indexes.
         0 KB in bad sectors.
        32 KB in use by the system.
    981376 KB available on disk.

     32768 bytes in each allocation unit.
     30682 total allocation units on disk.
     30668 allocation units available on disk.

Testing Environment

I've made several tests on different physical devices to ensure this isn't device-specific:

Internal HDD
Internal SSD
USB flash drive
SD card (via card reader)

Tests were run on two devices (laptop & PC, both running Windows 10). All physical device types showed the same issue. I also used different Python versions - 3.8, 3.10, 3.12.

Thank you for creating such an advanced and useful project! I hope it's possible to find a solution to this bug. Please let me know if you need any additional information from me; I'll be glad to help!

maxpat78 commented 1 month ago

Just drafted the v. 1.0.39 release, that fixes such bug. Thank you!

Dsibe commented 1 month ago

Thank you very much for the swift response and release! I have completed all the tests and am glad to say that everything worked successfully with no issues. I truly appreciate your hard work. Have a great day!

maxpat78 / FATtools