Failing tests for FAT and NTFS

[importantchoice]

Some of us discovered a few problems while running the unittests. Under Debian and arch linux all FAT tests pass, but the NTFS test of MFT File Records failes with:

____________________________________________________________________ TestGetRecordOfFile.test_get_record_of_file ____________________________________________________________________

self = <test_ntfs.TestGetRecordOfFile object at 0x7f78e89e5fd0>, testfs_ntfs_stable1 = ['/tmp/tmpjej3b9pj/testfs-ntfs-stable1.dd']

    def test_get_record_of_file(self, testfs_ntfs_stable1):
        """
            Tests if the correct record is returned for
            the supplied name
            """
        with open(testfs_ntfs_stable1[0], 'rb') as fs:
            ntfs = NTFS(fs)
            assert ntfs.get_record_of_file('$MFT') == 0
            assert ntfs.get_record_of_file('$MFTMirr') == 1
            assert ntfs.get_record_of_file('$BadClus') == 8
            assert ntfs.get_record_of_file('another') == 64
>           assert ntfs.get_record_of_file('onedirectory/nested_directory') == 69
E           AssertionError: assert 71 == 69
E            +  where 71 = <bound method NTFS.get_record_of_file of <fishy.ntfs.ntfs_filesystem.ntfs.NTFS object at 0x7f78e89e5748>>('onedirectory/nested_directory')
E            +    where <bound method NTFS.get_record_of_file of <fishy.ntfs.ntfs_filesystem.ntfs.NTFS object at 0x7f78e89e5748>> = <fishy.ntfs.ntfs_filesystem.ntfs.NTFS object at 0x7f78e89e5748>.get_record_of_file

tests/test_ntfs.py:69: AssertionError

Under ubuntu there are issues with the FAT tests but NTFS tests pass.

I was not able to reproduce the FAT test failures under the latest ubuntu version.

[importantchoice]

My first guess would be, that the mkfs command behaves differently on our distos. My fsstat output of the generated NTFS filesystem:

$ fsstat utils/testfs-ntfs-stable1.dd
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: NTFS
Volume Serial Number: 011B15140ECCD48F
OEM Name: NTFS    
Version: Windows XP

METADATA INFORMATION
--------------------------------------------
First Cluster of MFT: 4
First Cluster of MFT Mirror: 1249
Size of MFT Entries: 1024 bytes
Size of Index Records: 4096 bytes
Range: 0 - 74
Root Directory: 5

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 4096
Total Cluster Range: 0 - 2498
Total Sector Range: 0 - 19998

$AttrDef Attribute Values:
$STANDARD_INFORMATION (16)   Size: 48-72   Flags: Resident
$ATTRIBUTE_LIST (32)   Size: No Limit   Flags: Non-resident
$FILE_NAME (48)   Size: 68-578   Flags: Resident,Index
$OBJECT_ID (64)   Size: 0-256   Flags: Resident
$SECURITY_DESCRIPTOR (80)   Size: No Limit   Flags: Non-resident
$VOLUME_NAME (96)   Size: 2-256   Flags: Resident
$VOLUME_INFORMATION (112)   Size: 12-12   Flags: Resident
$DATA (128)   Size: No Limit   Flags: 
$INDEX_ROOT (144)   Size: No Limit   Flags: Resident
$INDEX_ALLOCATION (160)   Size: No Limit   Flags: Non-resident
$BITMAP (176)   Size: No Limit   Flags: Non-resident
$REPARSE_POINT (192)   Size: 0-16384   Flags: Non-resident
$EA_INFORMATION (208)   Size: 8-8   Flags: Resident
$EA (224)   Size: 0-65536   Flags: 
$LOGGED_UTILITY_STREAM (256)   Size: 0-65536   Flags: Non-resident

For FAT12:

$  fsstat utils/testfs-fat12-stable1.dd 
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT12

OEM Name: mkfs.fat
Volume ID: 0xb67f3a84
Volume Label (Boot Sector): NO NAME    
Volume Label (Root Directory):
File System Type Label: FAT12   

Sectors before file system: 0

File System Layout (in sectors)
Total Range: 0 - 1999
* Reserved: 0 - 0
** Boot Sector: 0
* FAT 0: 1 - 2
* FAT 1: 3 - 4
* Data Area: 5 - 1999
** Root Directory: 5 - 36
** Cluster Area: 37 - 1996
** Non-clustered: 1997 - 1999

METADATA INFORMATION
--------------------------------------------
Range: 2 - 31926
Root Directory: 2

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 2048
Total Cluster Range: 2 - 491

FAT CONTENTS (in sectors)
--------------------------------------------
41-44 (4) -> EOF
45-60 (16) -> EOF
61-76 (16) -> EOF
77-80 (4) -> EOF
81-84 (4) -> EOF
85-88 (4) -> EOF
89-92 (4) -> EOF
93-96 (4) -> EOF

For FAT16:

$ fsstat utils/testfs-fat16-stable1.dd 
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT16

OEM Name: mkfs.fat
Volume ID: 0xb6837c7e
Volume Label (Boot Sector): NO NAME    
Volume Label (Root Directory):
File System Type Label: FAT16   

Sectors before file system: 0

File System Layout (in sectors)
Total Range: 0 - 49999
* Reserved: 0 - 3
** Boot Sector: 0
* FAT 0: 4 - 55
* FAT 1: 56 - 107
* Data Area: 108 - 49999
** Root Directory: 108 - 139
** Cluster Area: 140 - 49999

METADATA INFORMATION
--------------------------------------------
Range: 2 - 798278
Root Directory: 2

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 2048
Total Cluster Range: 2 - 12466

FAT CONTENTS (in sectors)
--------------------------------------------
144-147 (4) -> EOF
148-163 (16) -> EOF
164-179 (16) -> EOF
180-183 (4) -> EOF
184-187 (4) -> EOF
188-191 (4) -> EOF
192-195 (4) -> EOF
196-199 (4) -> EOF

For FAT32:

$ fsstat utils/testfs-fat32-stable1.dd 
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT32

OEM Name: mkfs.fat
Volume ID: 0xb6b0c82c
Volume Label (Boot Sector): NO NAME    
Volume Label (Root Directory):
File System Type Label: FAT32   
Next Free Sector (FS Info): 1200
Free Sector Count (FS Info): 548792

Sectors before file system: 0

File System Layout (in sectors)
Total Range: 0 - 549999
* Reserved: 0 - 31
** Boot Sector: 0
** FS Info Sector: 1
** Backup Boot Sector: 6
* FAT 0: 32 - 575
* FAT 1: 576 - 1119
* Data Area: 1120 - 549999
** Cluster Area: 1120 - 549999
*** Root Directory: 1120 - 1127

METADATA INFORMATION
--------------------------------------------
Range: 2 - 8782086
Root Directory: 2

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 4096
Total Cluster Range: 2 - 68611

FAT CONTENTS (in sectors)
--------------------------------------------
1120-1127 (8) -> EOF
1128-1135 (8) -> EOF
1136-1151 (16) -> EOF
1152-1167 (16) -> EOF
1168-1175 (8) -> EOF
1176-1183 (8) -> EOF
1184-1191 (8) -> EOF
1192-1199 (8) -> EOF
1200-1207 (8) -> EOF

@timbuntu could you compare my results with yours and post them if they differ?

[timbuntu]

For NTFS and FAT12 my fsstat output only differed from yours in the Volume Serial Number, but FAT16 and FAT32 differed quite a bit.

FAT16:

$ fsstat utils/testfs-fat16-stable1.dd
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT16

OEM Name: mkfs.fat
Volume ID: 0x123bd7b5
Volume Label (Boot Sector): NO NAME    
Volume Label (Root Directory):
File System Type Label: FAT16   

Sectors before file system: 0

File System Layout (in sectors)
Total Range: 0 - 49999
* Reserved: 0 - 0
** Boot Sector: 0
* FAT 0: 1 - 52
* FAT 1: 53 - 104
* Data Area: 105 - 49999
** Root Directory: 105 - 136
** Cluster Area: 137 - 49996
** Non-clustered: 49997 - 49999

METADATA INFORMATION
--------------------------------------------
Range: 2 - 798326
Root Directory: 2

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 2048
Total Cluster Range: 2 - 12466

FAT CONTENTS (in sectors)
--------------------------------------------
141-144 (4) -> EOF
145-160 (16) -> EOF
161-176 (16) -> EOF
177-180 (4) -> EOF
181-184 (4) -> EOF
185-188 (4) -> EOF
189-192 (4) -> EOF
193-196 (4) -> EOF

FAT32:

$ fsstat utils/testfs-fat32-stable1.dd
FILE SYSTEM INFORMATION
--------------------------------------------
File System Type: FAT32

OEM Name: mkfs.fat
Volume ID: 0x12827e8e
Volume Label (Boot Sector): NO NAME    
Volume Label (Root Directory):
File System Type Label: FAT32   
Next Free Sector (FS Info): 1186
Free Sector Count (FS Info): 548800

Sectors before file system: 0

File System Layout (in sectors)
Total Range: 0 - 549999
* Reserved: 0 - 31
** Boot Sector: 0
** FS Info Sector: 1
** Backup Boot Sector: 6
* FAT 0: 32 - 568
* FAT 1: 569 - 1105
* Data Area: 1106 - 549999
** Cluster Area: 1106 - 549993
*** Root Directory: 1106 - 1113
** Non-clustered: 549994 - 549999

METADATA INFORMATION
--------------------------------------------
Range: 2 - 8782310
Root Directory: 2

CONTENT INFORMATION
--------------------------------------------
Sector Size: 512
Cluster Size: 4096
Total Cluster Range: 2 - 68612

FAT CONTENTS (in sectors)
--------------------------------------------
1106-1113 (8) -> EOF
1114-1121 (8) -> EOF
1122-1137 (16) -> EOF
1138-1153 (16) -> EOF
1154-1161 (8) -> EOF
1162-1169 (8) -> EOF
1170-1177 (8) -> EOF
1178-1185 (8) -> EOF
1186-1193 (8) -> EOF

And well, I am still on Ubuntu 14.04, so maybe that could cause it? Also, did you run the NTFS test on the system that you did the fsstat on? So does it fail even though the test filesystems seem to be identical in their structure? If that is the case, doesn't it have to do with the behaviour while copying the test files to the filesystem?

[timbuntu]

Ok, so I did install an Ubuntu 17.10 VM, and there the FAT tests work and the NTFS test fails the same way as on Jenkins.

So the problem is indeed me being still on Ubuntu 14.04, which seems to use different default settings for FAT16 and FAT32 in the mkfs command.

I also looked into how the files are copied onto the filesystem and it is indeed done in a different order on both systems, which explains why the NTFS tests fail.

Ubuntu 14.04:

$ cp -rv fs-files/* mount-fs
»fs-files/another“ -> »mount-fs/another“
»fs-files/areallylongfilenamethatiwanttoreadcorrectly.txt“ -> »mount-fs/areallylongfilenamethatiwanttoreadcorrectly.txt“
»fs-files/long_file.txt“ -> »mount-fs/long_file.txt“
»fs-files/no_free_slack.txt“ -> »mount-fs/no_free_slack.txt“
»fs-files/onedirectory“ -> »mount-fs/onedirectory“
»fs-files/onedirectory/nested_directory“ -> »mount-fs/onedirectory/nested_directory“
»fs-files/onedirectory/nested_directory/royce.txt“ -> »mount-fs/onedirectory/nested_directory/royce.txt“
»fs-files/onedirectory/areallylongfilenamethatialsowanttoreadassoonaspossible.txt“ -> »mount-fs/onedirectory/areallylongfilenamethatialsowanttoreadassoonaspossible.txt“
»fs-files/onedirectory/afileinadirectory.txt“ -> »mount-fs/onedirectory/afileinadirectory.txt“
»fs-files/testfile.txt“ -> »mount-fs/testfile.txt“

Ubuntu 17.10:

$ cp -rv fs-files/* mount-fs
'fs-files/another' -> 'mount-fs/another'
'fs-files/areallylongfilenamethatiwanttoreadcorrectly.txt' -> 'mount-fs/areallylongfilenamethatiwanttoreadcorrectly.txt'
'fs-files/long_file.txt' -> 'mount-fs/long_file.txt'
'fs-files/no_free_slack.txt' -> 'mount-fs/no_free_slack.txt'
'fs-files/onedirectory' -> 'mount-fs/onedirectory'
'fs-files/onedirectory/afileinadirectory.txt' -> 'mount-fs/onedirectory/afileinadirectory.txt'
'fs-files/onedirectory/areallylongfilenamethatialsowanttoreadassoonaspossible.txt' -> 'mount-fs/onedirectory/areallylongfilenamethatialsowanttoreadassoonaspossible.txt'
'fs-files/onedirectory/nested_directory' -> 'mount-fs/onedirectory/nested_directory'
'fs-files/onedirectory/nested_directory/royce.txt' -> 'mount-fs/onedirectory/nested_directory/royce.txt'
'fs-files/testfile.txt' -> 'mount-fs/testfile.txt'

[timbuntu]

I did change that one test to only use NTFS system files now, they are defined to be at fixed positions in the mft all the time, so there should be no further problems with that test.

[importantchoice]

I never would have guessed that this is about different shell or cp behaviour. But I think we might encounter further problems with it. E.g. when unittesting the ntfs slack space module. So we should try to "unify" this behaviour. If the different order is caused by the shell globbing, maybe we could pre-sort the files. If cp itself is the problem, then we could do it in a loop to ensure the correct order...

Regarding FAT: The first difference in the FAT16 output is the reserved sector count, that is increased in my version (maybe to have no non-clustered sectors left). So, providing the -R flag to mkfs.fat might fix that. The FAT32 output seems more stange to me. Why should the FAT size be increased? Either this was a bug in older mkfs.fat implementations, or the generated image files differ in their size(?)

Oh and could anyone with failing FAT tests post the output of pytest? I don't remember it exactly, but I think there was an error in FAT12 tests, right?

[importantchoice]

With commit d0377b1 the creation of FAT16 images should be stabelized across our different mkfs.vfat versions. Commit 87853cf replaces the 'unpredictable' recursive copy of cp with a deterministic order. @timbuntu could you check if this really fixes any unittest issue? And if there are failures left, post the pytest output?

[importantchoice]

with commit #8 this should be fixed.