search

JElchison / format-udf

Bash script to format a block device (hard drive or Flash drive) in UDF. The output is a drive that can be used for reading/writing across multiple operating system families: Windows, macOS, and Linux. This script should be capable of running in macOS or in Linux.
GNU General Public License v2.0
518 stars 48 forks source link

When formatted under Ubuntu MATE 15.04 - Mac OS X does not recognize disk format #16

Closed asmoporma closed 8 years ago

asmoporma commented 9 years ago

Formatted USB stick under raspberry pi running Ubuntu MATE 15.04. Formatting goes fine. When USB stick is inserted to Mac OS X 10.10 (yosemite), it does not contain filesystem. If USB stick is formatted on Mac, Ubuntu will recognize filesystem but shows label "MAC OS X ...". UDF is fine filesystem but there seems to be some inconsistency with tools and how labels are handled.

JElchison commented 9 years ago

hi, @asmoporma. thanks for your report.

i've seen OS X not auto-detect UDF drives before, though i have no explanation for why. what happens when you attempt to manually mount the UDF drive in OS X using the terminal?

more debugging info that might be helpful:

please post output from each of the following, on ubuntu:

also, please post output from each of the following, on OS X. note, these are linux commands, so they may need to be translated into their OS X counterparts:

this issue may be related to #12

as for the labeling inconsistency, please see #11

many thanks!

ksterker commented 8 years ago

Tried today with OS X 10.9.5 after formatting the external HDD under Ubuntu 14.04 yesterday and it's the same. The drive itself is recognized, but did not seem to contain a file system. Instead the system asked to format it. So I ended up running the script on OS X, and now the drive is usable on both Linux and OS X, as intended :-) - with the slight inconvenience of #11 .

I guess I could still provide the requested info from the Linux side, if you think that'll be useful. It's too late for most of the OS X stuff, now that the disk is already reformatted, however.

At any rate, nice utility. With the few issues hopefully sorted out, it'll be real helpful for preparing cross-platform storage!

S-C-Johnston commented 8 years ago

I've performed investigation on this task. I found it suspect that OS X couldn't handle udf partitions, as opposed to whole disk fake partitions. It can't, automatically, but it can if you do this manually using mount_udf -b (physical block size). For advanced format drives under linux, you must then also define the physical block size for the partition.

Test disk is a 1TB Seagate advanced format drive, so 4096 physical blocksize, 512 logical.

  1. I created a new GPT partition table for the drive using gparted under knoppix.
  2. Also using gparted, I created a new partition formatted as unused or free space about 378 GB in size, just as an arbitrary partial size for testing. It's also offset from the beginning of the disk a bit.
  3. Using format-udf, I formatted the partition: $ ./format-udf -p none sdd1 udf-test, adding the -p none flag. The script responded that the disk wouldn't be understandable by OS X.

Now the testing.

Almost forgot:

$ uname -a
Darwin Euclid.local 15.2.0 Darwin Kernel Version 15.2.0: Fri Nov 13 19:56:56 PST 2015; root:xnu-3248.20.55~2/RELEASE_X86_64 x86_64
$ mount
/dev/disk0s2 on / (hfs, local, journaled)
devfs on /dev (devfs, local, nobrowse)
map -hosts on /net (autofs, nosuid, automounted, nobrowse)
map auto_home on /home (autofs, automounted, nobrowse)
/dev/disk1s1 on /Volumes/System Reserved (ntfs, local, read-only, noowners)
/dev/disk3s1 on /Volumes/Schrodinger (exfat, asynchronous, local, noowners)
/dev/disk1s2 on /Volumes/Untitled (ntfs, local, noowners, nobrowse)
/dev/disk2s1 on ../udf-test (udf, local)

diskutil doesn't really know what to make of disk2, or disk2s1:

$ diskutil list /dev/disk2
/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk2
   1:           Linux Filesystem                         378.1 GB   disk2s1

$ diskutil info /dev/disk2
   Device Identifier:        disk2
   Device Node:              /dev/disk2
   Whole:                    Yes
   Part of Whole:            disk2
   Device / Media Name:      ST1000DM003-1ER162

   Volume Name:              Not applicable (no file system)

   Mounted:                  Not applicable (no file system)

   File System:              None

   Content (IOContent):      GUID_partition_scheme
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SATA
   SMART Status:             Verified

   Total Size:               1.0 TB (1000204886016 Bytes) (exactly 1953525168 512-Byte-Units)
   Volume Free Space:        Not applicable (no file system)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          No

   Solid State:              No
   Virtual:                  No
   OS 9 Drivers:             No
   Low Level Format:         Not supported

$ diskutil info /dev/disk2s1
   Device Identifier:        disk2s1
   Device Node:              /dev/disk2s1
   Whole:                    No
   Part of Whole:            disk2
   Device / Media Name:      Untitled 1

   Volume Name:              Not applicable (no file system)

   Mounted:                  Not applicable (no file system)

   File System:              None

   Partition Type:           Linux Filesystem
   OS Can Be Installed:      No
   Media Type:               Generic
   Protocol:                 SATA
   SMART Status:             Verified
   Disk / Partition UUID:    0AF10C07-0116-4611-B234-4BC3E5B7BD57

   Total Size:               378.1 GB (378077708288 Bytes) (exactly 738433024 512-Byte-Units)
   Volume Free Space:        Not applicable (no file system)
   Device Block Size:        512 Bytes

   Read-Only Media:          No
   Read-Only Volume:         Not applicable (no file system)

   Device Location:          External
   Removable Media:          No

   Solid State:              No

diskutil also lies, to some extent. It knows that the logical block size is 512 bytes, so that's what it reports, but you need the physical block size to mount properly. Trying to mount the partition with the logical block size diskutil reports (which is the default that the mount_ utils use) results in this:

$ sudo mount_udf -b 512 /dev/disk2s1 ../udf-test/
mount_udf: /dev/disk2s1 on ../udf-test/: Device not configured

However, using mount_udf with the physical block size works:

$ sudo mount_udf -b 4096 /dev/disk2s1 ../udf-test/
$ ls ../udf-test/
lost+found

Though many of the disk tools still don't know what to do with it, and report strange results. However, I had similar results when I used format-udf under OS X on a whole disk, so I think that's an OS X handling of udf issue, not a partition table issue. 

$ df -h ../udf-test/
Filesystem     Size   Used  Avail Capacity   iused    ifree %iused  Mounted on
/dev/disk2s1  352Gi   11Mi  352Gi     1% 18446744073617248021 92303597 922337203680862404608%   ../udf-test

Conclusion: advanced format drives are weird, OS X needs manual instructions to mount nonstandard udf partitions, physical block size is important.

JElchison commented 8 years ago

Since we haven't heard from @asmoporma since the ticket was opened about 14 months ago, I feel comfortable closing this issue with the clarifying text I've just added on the README. Essentially, there's not much that format-udf can do if OS X has trouble auto-mounting drives with certain block sizes.

Related discussion is still welcome in this thread. I'm always more that happy to revisit this issue if there's any way that format-udf can produce drives that are more compatible on target OSes (while respecting the device geometry, of course).