Formatted flash drive not detected by Windows 8.1

zauberparacelsus commented 3 years ago

I ran format-udf on a flash drive of mine, and loaded some data onto it. It reads fine on my Linux system (64bit Manjaro Linux), but on my Windows system (64bit Windows 8.1 Pro), it does not get automatically mounted on bootup or when I remove and re-insert the flash drive.

Running gdisk -l on my flash drive returns the following output:

GPT fdisk (gdisk) version 1.0.5

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present

***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. 
***************************************************************

Warning! Main partition table overlaps the first partition by 34 blocks!
You will need to delete this partition or resize it in another utility.

Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.
Disk /dev/sde: 485228544 sectors, 231.4 GiB
Model: Ultra           
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 410064B0-8EA4-406E-AD44-504336157E5E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 485228510
Partitions will be aligned on 2048-sector boundaries
Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name

JElchison commented 3 years ago

Hi @zauberparacelsus,

Since Windows is a closed-source operating system, these issues are often really hard to diagnose. What you're experiencing may very well be a limitation of Windows 8.1. (FYI, Windows 8.1 reached the end of Mainstream Support on January 9, 2018, and will reach end of Extended Support on January 10, 2023.)

If you're willing to do the legwork, I'm happy to dig in with you. Questions that may surface relevant info:

is the issue reproducible?
- what are the exact environment/steps that i can use to repro it?
- how many times have you attempted these exact steps?
are you using a 4K (Advanced Format) drive?
does the issue still occur if you use a different flash drive?
does the issue still occur if you use a hard drive instead of a flash drive?
does the issue still occur if you use a different version of Windows (like Windows 10)?
does the issue still occur if you use a different distro of Linux to run format-udf?
does the issue still occur if you use OS X to run format-udf?
what version of udftools are you using?
what version of format-udf are you using?

also, please post output from each of the following:

the actual format-udf command you used
sudo fdisk -l
- is your drive in the list?
sudo blkid -c /dev/null /dev/sdX, where X is the letter associated with your drive

You should also know that using gdisk may itself engander what format-udf seeks to accomplish. For more info, see the README, particularly the text about implementing a fake partition table.

zauberparacelsus commented 3 years ago

The environment was 64bit Manjaro Linux, using kernel version 5.7.19-2.

As far as steps, I just unmounted the flash drive, ran ./format-udf.sh /dev/sde ZAUBERFLASH, then disconnected and reconnected it. Doing everything again on the flash drive yields the following output:

[zauber@zauber-desktop ~]$ ./format-udf.sh /dev/sde ZAUBERFLASH
[+] Validating arguments...
[+] Testing dependencies...
[+] Looking for drive detail tool... using /usr/bin/blockdev
[+] Looking for drive listing tool... using /usr/bin/blockdev
[+] Looking for drive info tool... using /usr/bin/lsblk
[+] Looking for drive summary tool... using /usr/bin/blkid
[+] Looking for unmount tool... using /usr/bin/umount
[+] Looking for UDF tool... using /usr/bin/mkudffs
[+] Detecting logical block size...
[*] Detected logical block size of 512
[+] Validating detected logical block size...
[+] Detecting physical block size...
[*] Detected physical block size of 512
[+] Validating detected physical block size...
[+] Validating file system block size...
[*] Using file system block size of 512
[+] Detecting total size...
[*] Detected total size of 248437014528
[+] Validating detected total size...
[+] Gathering drive information...
/dev/sde: UUID="5fdf469229ac82f7" LABEL="ZAUBERFLASH" BLOCK_SIZE="512" TYPE="udf" PTTYPE="dos"
Ultra           
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0    248437014528   /dev/sde
rw   256   512  4096          0    248437014528   /dev/sde1
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Unmounting device...
umount: /dev/sde: not mounted.
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18175 s, 11.5 MB/s
[+] Formatting /dev/sde ...
filename=/dev/sde
label=ZAUBERFLASH
uuid=5fdf47f2e624ffca
blocksize=512
blocks=485228544
udfrev=2.01
start=0, blocks=64, type=ERASE 
start=64, blocks=13, type=VRS 
start=77, blocks=19, type=ERASE 
start=96, blocks=16, type=MVDS 
start=112, blocks=16, type=ERASE 
start=128, blocks=16, type=LVID 
start=144, blocks=112, type=ERASE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=485228024, type=PSPACE 
start=485228281, blocks=6, type=ERASE 
start=485228287, blocks=1, type=ANCHOR 
start=485228288, blocks=96, type=ERASE 
start=485228384, blocks=16, type=RVDS 
start=485228400, blocks=143, type=ERASE 
start=485228543, blocks=1, type=ANCHOR 
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes copied, 0.00737688 s, 2.2 kB/s
2+0 records in
2+0 records out
2 bytes copied, 8.8631e-05 s, 22.6 kB/s
[+] Successfully formatted /dev/sde: UUID="5fdf47f2e624ffca" LABEL="ZAUBERFLASH" BLOCK_SIZE="512" TYPE="udf" PTTYPE="dos"
Please disconnect/reconnect your drive now

I do not believe I am using a 4K drive, and based on the wikipedia article for Advanced Format, it looks like it only applies to magnetic disk drives, but the device in question is a USB flash drive.

I do not have a spare flash drive, or any devices running OS X. I also only have Windows 8.1 on my computer, and I don't use other Linux distros. Upgrading to Windows 10, wiping another device to install another Linux distro, or opening up my computer to install additional hardware go far above and beyond what I am willing to do. (plus I'm not even sure that my spare hard drives even work)

I am using version 2.2 of udftools, and version 1.8 of format-udf.

Running sudo fdisk -l, my flash drive shows up as two device, /dev/sde and /dev/sde1

Disk /dev/sde: 231.38 GiB, 248437014528 bytes, 485228544 sectors
Disk model: Ultra           
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start       End   Sectors   Size Id Type
/dev/sde1           0 485228543 485228544 231.4G  b W95 FAT32

Running blkid on both /dev/sde and /dev/sde1 gives the following output:

[zauber@zauber-desktop ~]$ sudo blkid -c /dev/null /dev/sde
/dev/sde: UUID="5fdf47f2e624ffca" LABEL="ZAUBERFLASH" BLOCK_SIZE="512" TYPE="udf" PTTYPE="dos"
[zauber@zauber-desktop ~]$ sudo blkid -c /dev/null /dev/sde1
/dev/sde1: UUID="5fdf47f2e624ffca" LABEL="ZAUBERFLASH" BLOCK_SIZE="512" TYPE="udf" PTTYPE="dos"

EDIT: I additionally tried to compile and install udfclient so I could try running udf-format with newfs_udf. However, the AUR pkgbuild for udfclient was both orphaned and out of date (0.8.9), and so I tried compiling it anyways, which failed.

JElchison commented 3 years ago

Hi @zauberparacelsus, thanks for the data! There's nothing obvious that sticks out to me. I've been unable to reproduce your situation, but I also don't have a Windows 8.1 machine available for testing.

I'll keep this issue open in case someone else can contribute another data point.

Thanks!

zauberparacelsus commented 3 years ago

Alright. For now I'm going to reformat my flash drive as exFat, but I'll give format-udf another spin if things change.

zauberparacelsus commented 3 years ago

Some additional info: I found that even with exFat, the drive still wasn't readable on windows. Acting on a hunch though, I unplugged it from the USB 3.0 hub it was on and instead plugged it in to a USB 2.x slot on the front of my computer, and then it just worked.

So, either exfat and udf have issues with USB 3.0, or there's something wrong with my computer's USB 3.0 controller.

EDIT: if the latter is the case, then I think that maybe Linux has better error-handling with USB 3.0 compared to Windows 8.1? I think the first is more likely though, as Windows had no trouble reading the flash drive in that USB 3.0 slot when it was formatted with fat32

JElchison / format-udf

Formatted flash drive not detected by Windows 8.1 #54