format-udf

Bash script to format a block device (hard drive or Flash drive) in UDF. The output is a drive that can be used for reading/writing across multiple operating system families: Windows, macOS, and Linux. This script should be capable of running in macOS or in Linux.

Features

• Formats a block device (hard drive or Flash drive) in Universal Disk Format (UDF)
  • UDF revision 2.01 used for maximal compatibility (see note on Linux support below)
• Resulting file system can be read/written across multiple operating system families (Windows, macOS, and Linux)
• Runs on any OS having a Bash environment
• Optionally wipes device before formatting
• Ability to override detected device block size
• Option to force non-interactive mode (useful for scripting)
• Writes a fake MBR for added compatibility on Windows (optionally disabled)

For the advanced user, format-udf is also capable of formatting a single existing partition, without modifying the partition table. Beware that using this method will render the newly formatted UDF partition unusable on macOS (but still usable on Linux and Windows). (See #24 for caveats.) Because of this limitation, the recommendation is to format the entire device.

Why?

format-udf was created to address some OS-specific quirks that prevent a naively-formatted UDF device from working across various operating systems. Here are some of the complicating factors, which format-udf aims to abstract away:

• Different operating systems support different versions of the UDF specification. Some OS versions only offer read-only support.
• Windows seems to only mount UDF devices if the file system block size matches the device's logical block size
• Different operating systems (like Windows XP) only attempt mounting UDF file systems with a hard-coded block size
• Windows does not support hard disks without a partition table. (This is strange because Windows does not apply the same limitation to flash drives.)
• macOS seems to only mount UDF file systems that utilize the full disk (not just a partition)

At first glance, these constraints appear to be in partial conflict. The solution, as suggested by Pieter, is to place a fake partition table (via MBR) in the first block of the drive, which lists a single entire-disk partition. This works because UDF (perhaps intentionally) doesn't utilize the first block. Unfortunately, there has been no easy way to do this, while juggling all of the other variables (such as device logical block size). format-udf writes such a fake MBR for added compatibility on Windows. If this is not what you desire, you can disable the MBR with -p none.

The goal of this project is to provide access to a cross-platform file system with modern features, in such a way that is:

1. Easy to use for the average user
2. Maximally compatible across operating systems
3. As compliant as is reasonable with the UDF specification
4. Maximally flexible to help users with uncommon needs

UDF OS Support

Not all operating systems support UDF. The following tables detail operating system support for UDF. Data was adapted from https://en.wikipedia.org/wiki/Universal_Disk_Format#Compatibility (as retrieved on 2017-06-16).

Natively Supported

Both read/write are supported unless otherwise listed below.

Supported with Third-Party Utilities

Not Supported

4K Drive Support

Not all operating systems support 4K drives (Advanced Format). If you operating system supports UDF, but not your 4K drive, you still may encounter issues using format-udf.

Windows 4K Drive Support

The following tables detail Windows support for 4K drives. Data was adapted from the Microsoft support policy for 4K sector hard drives in Windows (as retrieved on 2017-06-16). Overlaid into this table are testing results from the format-udf community. (Special thanks to @pali for his testing on XP.)

If you have conducted testing and would like to update this table to benefit future users of format-udf, please send a pull request. Please include a link to your raw data or testing results.

Environment

• Any OS having a Bash environment
• The following tools must be installed, executable, and in the PATH:
  • printf
  • xxd
  • One of the following: blockdev, ioreg
  • One of the following: blockdev, diskutil
  • One of the following: lsblk, diskutil
  • One of the following: umount, diskutil
  • One of the following: mkudffs, newfs_udf

Prerequisites

To install necessary prerequisites on Ubuntu:

sudo apt-get install udftools coreutils vim-common

Installation

format-udf is a self-contained script. Simply copy format-udf.sh to a directory of your choosing. Don't forget to make it executable:

chmod +x format-udf.sh

Usage

Bash script to format a block device (hard drive or Flash drive) in UDF.
The output is a drive that can be used for reading/writing across multiple
operating system families: Windows, macOS, and Linux.
This script should be capable of running in macOS or in Linux.

Usage:  ./format-udf.sh [-b BLOCK_SIZE] [-f] [-p PARTITION_TYPE] [-w WIPE_METHOD] device label
        ./format-udf.sh -v
        ./format-udf.sh -h

    -b BLOCK_SIZE
        Block size to be used during format operation.
        If absent, defaults to value reported by blockdev/diskutil.
        This is an expert-only option.  Please consult the README for details.

    -f
        Forces non-interactive mode.  Useful for scripting.
        Please use with caution, as no user confirmation is given.

    -h
        Display help information and exit.

    -p PARTITION_TYPE
        Partition type to set during format operation.
        Currently supported types include:  mbr, none
            mbr  - Master boot record (default)
            none - Do not modify partitions
        If absent, defaults to 'mbr'.
        See also:
            https://github.com/JElchison/format-udf#why

    -v
        Display version information and exit.

    -w WIPE_METHOD
        Wipe method to be used before format operation.
        Currently supported types include:  quick, zero, scrub
            quick - Quick method (default)
            zero  - Write zeros to the entire device
            scrub - Iteratively writes patterns on device
                    to make retrieving the data more difficult.
                    Requires 'scrub' to be executable and in the PATH.
                    See also http://linux.die.net/man/1/scrub
        If absent, defaults to 'quick'.
        Note:  'zero' and 'scrub' methods will take a long time.

    device
        Device to format.  Examples:
          * /dev/sdx   (Linux, where 'x' is a letter) or
          * /dev/diskN (macOS, where 'N' is a number)

    label
        Label to apply to formatted device.

Example:  ./format-udf.sh /dev/sdg "My UDF External Drive"

Example usage

On Ubuntu:

user@computer:~$ ./format-udf.sh /dev/sdg "My UDF External Drive"
[+] Validating arguments...
[+] Testing dependencies...
[+] Looking for drive detail tool... using /sbin/blockdev
[+] Looking for drive listing tool... using /sbin/blockdev
[+] Looking for drive info tool... using /bin/lsblk
[+] Looking for drive summary tool... using /sbin/blkid
[+] Looking for unmount tool... using /bin/umount
[+] Looking for UDF tool... using /usr/sbin/mkudffs
[+] Detecting logical block size...
[sudo] password for user:
[*] Detected logical block size of 512
[+] Validating detected logical block size...
[+] Detecting physical block size...
[*] Detected physical block size of 512
[+] Validating detected physical block size...
[+] Validating file system block size...
[*] Using file system block size of 512
[+] Detecting total size...
[*] Detected total size of 8019509248
[+] Validating detected total size...
[+] Gathering drive information...
/dev/sdg: LABEL="Old Drive" UUID="4843-D1BD" TYPE="vfat"

RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512   512          0      8019509248   /dev/sdg
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Unmounting device...
umount: /dev/sdg: not mounted
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.450716 s, 4.7 MB/s
[+] Formatting /dev/sdg ...
filename=/dev/sdg
label=My UDF External Drive
uuid=5e4924cc17b50769
blocksize=512
blocks=15663104
udfrev=2.01
start=0, blocks=64, type=ERASE 
start=64, blocks=13, type=VRS 
start=77, blocks=19, type=ERASE 
start=96, blocks=16, type=MVDS 
start=112, blocks=16, type=ERASE 
start=128, blocks=16, type=LVID 
start=144, blocks=112, type=ERASE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=15662590, type=PSPACE 
start=15662847, blocks=1, type=ANCHOR 
start=15662848, blocks=96, type=ERASE 
start=15662944, blocks=16, type=RVDS 
start=15662960, blocks=143, type=ERASE 
start=15663103, blocks=1, type=ANCHOR 
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes copied, 0.0037039 s, 4.3 kB/s
2+0 records in
2+0 records out
2 bytes copied, 3.7193e-05 s, 53.8 kB/s
[+] Successfully formatted /dev/sdg: UUID="5e4924cc17b50769" LABEL="My UDF External Drive" TYPE="udf" PTTYPE="dos"
Please disconnect/reconnect your drive now.

On macOS:

computer:~ user$ ./format-udf.sh /dev/disk4 "My UDF External Drive"
[+] Validating arguments...
[+] Testing dependencies...
[+] Looking for drive detail tool... using /usr/sbin/ioreg
[+] Looking for drive listing tool... using /usr/sbin/diskutil
[+] Looking for drive info tool... using /usr/sbin/diskutil
[+] Looking for drive summary tool... using (none)
[+] Looking for unmount tool... using /usr/sbin/diskutil
[+] Looking for UDF tool... using /sbin/newfs_udf
[+] Detecting logical block size...
[*] Detected logical block size of 512
[+] Validating detected logical block size...
[+] Detecting physical block size...
[+] Validating file system block size...
[*] Using file system block size of 512
[+] Detecting total size...
[*] Detected total size of 8019509248
[+] Validating detected total size...
[+] Gathering drive information...
/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *8.0 GB     disk4
   1:             Windows_FAT_32 Old Drive               8.0 GB     disk4s1
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Unmounting device...
Password:
Unmount of all volumes on disk4 was successful
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
2097152 bytes transferred in 3.113956 secs (673469 bytes/sec)
[+] Formatting /dev/disk4 ...
write to block device: /dev/disk4  last written block address: 15663103
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes transferred in 0.000615 secs (26021 bytes/sec)
2+0 records in
2+0 records out
2 bytes transferred in 0.000644 secs (3106 bytes/sec)
[+] Successfully formatted 
Please disconnect/reconnect your drive now.

Caveats

Block Size

If's extremely important that format-udf use the correct block size when formatting your drive. format-udf will attempt to detect and use the correct (logical) block size. If you know what you're doing, the format-udf -b BLOCK_SIZE option can be used to explicitly override the detected block size value.

If the wrong block size is used while formatting (i.e. one that doesn't match the logical block size of your drive), the resultant drive will likely have OS compatibility issues and suffer from non-optimal performance issues.

In the same way, it's just as important that the resultant drive be mounted using the correct block size. Many operating systems will only attempt one block size (usually whatever the mount utility defaults to). For example, In order to mount a UDF device, Windows seems to require that the UDF file system use a block size equal to the logical block size. If your block size isn't the OS's default, then auto-mounting likely will not work on your OS. While a small nuisance, manual mounting attempts should still succeed for nonstandard block sizes.

Example of how to manually mount on Linux:

$ mount -t udf -o bs=4096 /dev/sdX /mnt/mount-point

Example of how to manually mount on macOS:

$ sudo mount_udf -b 4096 /dev/diskN /Volumes/MountPoint

Sadly, anything with block size different than 512 doesn't seem to mount on Windows XP.

For more info, see #12, #13, #16, and #31.

Maximum UDF File System Capacity

The UDF format has a maximum of 2^32 blocks. With format-udf, these blocks equate to logical blocks.

• If your drive's logical block size is 512 bytes, then your maximum UDF file system capacity will be 2 TiB
• If your drive's logical block size is 4096 bytes, then your maximum UDF file system capacity will be 16 TiB

If your drive has capacity in excess of this maximum size, the extra capacity will not be used. This is a limitation of UDF itself.

For Best Results

For maximal OS compatibility, use format-udf on a device having a logical block size of 512 bytes. This will limit your total capacity to 2 TiB, but the resultant device should work on the most operating systems.

For maximal resultant UDF file system capacity, use use format-udf on a device having a logical block size of 4096 bytes. This will increase your total capacity (from 2 TiB) to 16 TiB, but will limit the number/types of operating systems that will be able to mount/read/write the resultant device. See compatibility tables above for more detail.

For a human-readable device label, use format-udf in one of the following configurations:

• Run format-udf on Linux
• Run format-udf on macOS, but modify the drive label using Linux or Windows

Miscellaneous Tips

• After installing GRUB2 on a partitionless drive, you can use fdisk to set the partition as active if your BIOS can't boot from partitionless drives. (Thanks to @tome- for the tip.)

Contributions

• Thanks to @walterav1984 for his contribution to add NVMe support

See Also

• Filesystems for portable disks
• Universal Disk Format on Wikipedia
• Wenguang's Introduction to Universal Disk Format (UDF)
• UDF Specifications
• Sharing a Hard/Flash Drive Across Windows, OS X, and Linux with UDF