f3probe different result on USB and non-USB

[Massimo-B]

Recently there was a change to allow non-USB devices to use f3probe, by using --reset-type=2. This is from what I understand.

However the result on the old reliable USB reader says counterfeit (/dev/sdg), while the internal SD reader says "real thing" (/dev/mmcblk0).

Build revision:

commit b7aa61bf154b238962b2d55a29bad19ca4ba60a7 (HEAD -> master, origin/master, origin/HEAD)
Author: Michel Machado <michel@digirati.com.br>
Date:   Fri Dec 22 14:16:03 2017 -0500

# ./f3probe /dev/sdg 
F3 probe 7.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

WARNING: Probing normally takes from a few seconds to 15 minutes, but
         it can take longer. Please be patient.

Probe finished, recovering blocks... Done

Bad news: The device `/dev/sdg' is a counterfeit of type limbo

You can "fix" this device using the following command:
f3fix --last-sec=8388607 /dev/sdg

Device geometry:
             *Usable* size: 4.00 GB (8388608 blocks)
            Announced size: 59.62 GB (125042688 blocks)
                    Module: 64.00 GB (2^36 Bytes)
    Approximate cache size: 0.00 Byte (0 blocks), need-reset=no
       Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 1.02s

# ./f3probe --reset-type=2 /dev/mmcblk0 
F3 probe 7.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

WARNING: Probing normally takes from a few seconds to 15 minutes, but
         it can take longer. Please be patient.

Probe finished, recovering blocks... Done

Good news: The device `/dev/mmcblk0' is the real thing

Device geometry:
             *Usable* size: 59.62 GB (125042688 blocks)
            Announced size: 59.62 GB (125042688 blocks)
                    Module: 64.00 GB (2^36 Bytes)
    Approximate cache size: 0.00 Byte (0 blocks), need-reset=no
       Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 14'17"

If the USB reader is really wrong and the SD card it not a counterfeit, then I may have returned some sane SD cards in the past...

In order to check the result, I'm going to run a f3write on the internal reader...

[Massimo-B]

I've written and read ~21GB successfully, so at least the usable 4GB counterfeit is wrong. I could write the whole card, but I like to avoid as this even reduces lifetime again:

./f3read /mnt/dummy
F3 read 7.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2097152/        0/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2097152/        0/      0/      0
Validating file 8.h2w ... 2097152/        0/      0/      0
Validating file 9.h2w ... 2097152/        0/      0/      0
Validating file 10.h2w ... 2097152/        0/      0/      0
Validating file 11.h2w ... 2097152/        0/      0/      0
Validating file 12.h2w ... 2097152/        0/      0/      0
Validating file 13.h2w ... 2097152/        0/      0/      0
Validating file 14.h2w ... 2097152/        0/      0/      0
Validating file 15.h2w ... 2097152/        0/      0/      0
Validating file 16.h2w ... 2097152/        0/      0/      0
Validating file 17.h2w ... 2097152/        0/      0/      0
Validating file 18.h2w ... 2097152/        0/      0/      0
Validating file 19.h2w ... 2097152/        0/      0/      0
Validating file 20.h2w ... 2097152/        0/      0/      0
Validating file 21.h2w ... 2097152/        0/      0/      0
Validating file 22.h2w ...  239161/        0/      0/      0

  Data OK: 21.11 GB (44279353 sectors)
Data LOST: 0.00 Byte (0 sectors)
           Corrupted: 0.00 Byte (0 sectors)
    Slightly changed: 0.00 Byte (0 sectors)
         Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 6.17 MB/s

I was wondering before because the shop claimed that it is definitly no counterfeit and recommended FAT32 as usual. This time I tried on the internal reader with a mkfs.vfat and f3write/read while the USB reader that claimed to see a counterfeit also had f3read failures but I was using f2fs there... So finally I'm not sure if the USB reader has issues with this card while it was fine with other cards, or if really the f2fs is the culprit which is even close to vfat, other than ext4 without journal for instance.

[AltraMayor]

When a system call returns an error, f3 doesn't have a way to know who is responsible for the failure: the computer, RAM memory, the kernel, the PCI bus, the USB reader, the memory card, or a combination of them. Given that f3 is meant to test memory cards, f3 implicitly assumes that the memory card is the only possible culprit.

The report of # ./f3probe /dev/sdg clearly found an error. But given that the card worked on another reader, it means that the USB reader or the USB port is the likely culprit; assuming that the issue is in the hardware. There's also a third possibility: when a reader warms too much the card being tested, it can make the card fail; see #37 for a reference on this issue. As pointed out in the previous paragraph, the issue could be in the kernel (e.g. f2fs) as well.

If you really want to catch the culprit in this case, you'll need to test many combinations to isolate the failing part. I should point out that this kind of failures are rare, or rarely reported, so I don't have tips to offer here.

Good luck!

[Massimo-B]

Ignoring the reduction of lifetime I ran a complete test on the sane reader, all sane. I re-ran the complete write/read test on the USB reader and there I see that f3read at least has a different result than f3probe on the same reader. I used the same vfat filesystem as for the sane test:

f3read reported

  Data OK: 17.13 GB (35923289 sectors)
Data LOST: 41.65 GB (87349735 sectors)
           Corrupted: 3.50 KB (7 sectors)
    Slightly changed: 0.00 Byte (0 sectors)
         Overwritten: 41.65 GB (87349728 sectors)
Average reading speed: 7.70 MB/s

while f3probe said this:

Probe finished, recovering blocks... Done

Bad news: The device `/dev/sdf' is a counterfeit of type limbo

You can "fix" this device using the following command:
f3fix --last-sec=8388607 /dev/sdf

Device geometry:
             *Usable* size: 4.00 GB (8388608 blocks)
            Announced size: 59.62 GB (125042688 blocks)
                    Module: 64.00 GB (2^36 Bytes)
    Approximate cache size: 0.00 Byte (0 blocks), need-reset=no
       Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 923.0ms

Curiously the f3probe returns immediately while on the sane reader it runs several minutes. It can't have done something seriously in less than 1sec. Afaik it backups some sectores, uses these for a small writing/reading test and then restores those sectors, regardless of the filesystem.

Does using f3probe on non-USB with --reset-type=2 have any impact? When is the real reset required? I guess the reset of f3probe replaces the recommended disconnecting of the card between f3write and f3read?

[AltraMayor]

The different results are just a reflection of what the two different tests are doing. f3write/f3read is checking whatever they can write and read; it doesn't matter if the good blocks are in a continuous region. While f3probe is trying to identify a continuous region when the drive is fake or damaged.

f3probe can do meaningful tests under 1s. f3probe works in a progressive way, the more a drive behaves as a real drive, the harder the tests become. So the sane reader is just triggering the harder tests, while the failing reader is quickly failing the simpler tests.

Does using f3probe on non-USB with --reset-type=2 have any impact?

It might slow down the test.

When is the real reset required?

Currently, the different reset methods only impact the performance of the tests.

I guess the reset of f3probe replaces the recommended disconnecting of the card between f3write and f3read?

Those are not the same thing. f3probe's resets are dealing with a possible cache inside the drive, whereas the recommended disconnection of the drive between f3write and f3read deals with the block cache of the operating system.

[unfa]

@AltraMayor - isn't f3read and f3write using syncronised I/O to avoid using OS disk cache? Or is it something different?

[AltraMayor]

f3write/f3read is not using synchronized I/O but the system call posix_fadvise(). But given that it's just an advice, the kernel is free to ignore it. That's why unplugging and plugging back is still recommended.

[wdthompson]

Some of the OLDER or cheaper usb hubs (but this is years ago now) have trouble with SDHC, cards ove 4 GB Also, I have found that older usb hubs often had early fail, better now