Support HFS+ Extended Attributes

[moon-chilled]

I'm installing xcode, as described in the README, but I get a bunch of messages like this:

cp: /Volumes/Xcode_7.3.1/Xcode.app/Contents/Developer/Documentation/DocSets/com.apple.adc.documentation.OSX.docset/Contents/Resources/Tokens/Objective-C/instp/NSLengthFormatter/unitStyle.xml: could not copy extended attributes to /Applications/Xcode.app/Contents/Developer/Documentation/DocSets/com.apple.adc.documentation.OSX.docset/Contents/Resources/Tokens/Objective-C/instp/NSLengthFormatter/unitStyle.xml: Bad file descriptor

/Applications/Xcode.app/Contents/Developer/Documentation/DocSets/com.apple.adc.documentation.OSX.docset/Contents/Resources/Tokens/Objective-C/instp/NSLengthFormatter/unitStyle.xml is created, and does have reasonable contents. What's going on?

[ahyattdev]

Extended Attributes are an HFS+ feature that Darling does not currently support because everything runs on the host's filesystem.

I am not sure why Xcode uses extended attributes here. It probably is meaningless. Anyway, no effect can be noticed now because the docsets can't be viewed until Xcode itself is working 😆.

[ahyattdev]

This is how Netatalk deals with HFS+ Extended Attributes

[LubosD]

When copying extattrs fails, cp tries to log that, which fails with EBADF (as we don't have syslogd) and then prints out the wrong error message to stderr.

[DuIslingr]

Could AFS cause issues with darling in the future? iOS 10.3 seems to have it and I imagine it will be present in all future mac versions. Asking here since its somewhat relevant I think.

[LubosD]

@DuIslingr No, I think AFS will be relevant only to darling-dmg (aka hdiutil).

[bugaevc]

It's APFS, actually

[ylluminate]

I think @lundman could have some useful insights on this area with his dealings with Extended Attribute handling on ZFS for macOS.

@lundman, what have been your experiences in this area as far as file system support goes on macOS and do you have any thoughts in going in the reverse direction here since the goal is being able to store these attributes on non-mac filesystems such as EXT3/4, ZFS, or any Linux FS?

(Heck, @brendonhumphrey and @ilovezfs might also be interested in chiming in.)

[lundman]

ext3/zfs already does xattrs (in style of forks), so unless you are going through a network filesystem to confuse things, you should be able to store them fine. One thing that differs between Linux/Unix and OsX is that OsX defines new vnops to handle xattrs, vnop_getxattr vnop_setxattr vnop_removexattr vnop_listxattr and corresponding libc syscalls. You'd have to map those to the openat style process.

Unless I completely misunderstood the question and answered something way deeper :)

[r-wilbur]

@ylluminate @lundman Did you folks make some progress towards supporting the extended attributes in the last 3 years? What is left to be done? Is there something I can do to help?

I'm getting a slightly different error message today. I rebuilt from a fresh clone of source yesterday and installed it. Today, I downloaded the OsX application[1] I hoped to run under darling as a .dmg[2], mounted it with Darling [~]$ hdiutil attach Downloads/macOS\ Mojave\ Patcher.dmg Darling [~]$ cp -r /Volumes/macOS\ Mojave\ Patcher/macOS\ Mojave\ Patcher.app /Applications/ [for each file I get a message of the following form] cp: /Volumes/macOS Mojave Patcher/macOS Mojave Patcher.app/Contents/Resources/Quartz.framework/Frameworks/QuartzFilters.framework/Resources/sv.lproj/QFilterView.strings: could not copy extended attributes to /Applications/macOS Mojave Patcher.app/Contents/Resources/Quartz.framework/Frameworks/QuartzFilters.framework/Resources/sv.lproj/QFilterView.strings: Operation not supported on socket

References: [1] http://dosdude1.com/mojave/ [2] https://gateway.pinata.cloud/ipfs/Qmds4TpDcAXszxMMACdBYK7XA8ih2kBoiuV2ZWejebgzgY/macOS%20Mojave%20Patcher.dmg

[LubosD]

This is of no concern - it's not a fatal error, just a warning.

[r-wilbur]

@LubosD What are the 'extended attributes' that APFS uses but Darling can't presently manage?

My application for Darling is to try and create a bootable USB memory stick containing an installer for macOS Mojave (10.14) and a program to patch that installation. I have formatted a partition on the memory stick as OS X Extended (Journaled) (using a friend's mac, before the arrival of the pandemic), but now am interested in running a macOS program[*] under Darling (on my Linux computer) which, if successful, will make a bootable image on that partition containing the macOS Mojave installer and a program to patch the Mojave installation so that it will boot on my late-2009 mac mini.

If I can help get Darling to the place where this works, it seems to me that Darling wins some new compatibility and functionality and I win when I can patch my mac mini to boot macOS Mojave so I can run XCode 11+ for iOS development.

(Does this possibly belong under a different bug/issue with the title 'Support APFS Extended Attributes'?) Reference: [*] http://dosdude1.com/mojave/#instructions

[r-wilbur]

In my understanding this involves copying files from darling filesystem to the USB memory stick partition (APFS) and getting the attributes correct so that the mac mini bootROM will recognize and boot from that partition. Also so it is valid under macOS Mojave, which is what I believe it will boot into.

It looks like there is an invocation of 'chmod' by the application which is currently unsupported in darling as I get the error message: usage: chmod [-fhv] [-R [-H | -L | -P]] [-a | +a | =a [i][# [ n]]] mode|entry file ... chmod [-fhv] [-R [-H | -L | -P]] [-E | -C | -N | -i | -I] file ...

Running the application also involves some GUI support, part of which is yet to be implemented in darling. macOS_Mojave_Patcher_notes.txt macOS_Mojave_Patcher.log

[r-wilbur]

@LubosD et al: This is the only darling issue listed on the github page for folks interested in making their first contribution to darling[*], to wit: " Contribute to darlinghq/darling

Make your first contribution to this repository by tackling one of the issues listed below.

Each issue displayed here is a "good first issue," selected for its relative approachability for first-time contributors. "

Reference: [*] https://github.com/darlinghq/darling/contribute

[LubosD]

There's no APFS support in Darling. If we're talking about the source DMG, then it must be HFS+.

Darling does support xattrs as of now, this includes HFS-specific attributes such as com.apple.FinderInfo. The problem is the namespaces feature used on Linux, which places additional restrictions on xattr names. So all xattrs starting with com.apple. should be translated to user.com.apple. during setxattr.c/fsetxattr.c/removexattr.c and back again in listxattr.c/getxattr.c.

This is relatively easy to do, but since it commonly only affects "Finder info" (e.g. pixel coordinates where to show this file in Finder) or potentially the resource fork (which is used only with Carbon apps, which we don't support yet), it's not a big deal.

Whatever issues you're having, they're really not related to the errors you're mentioning.

[r-wilbur]

@LubosD: I stand corrected! The filesystem is indeed HFS+ on the USB memory stick and I wasn't clear on the equivalence between 'Mac OS Extended' and HFS+. So it looks like if we can map the HFS+ xattrs from the DMG filesystem to Linux filesystem in darling and then map back out to HFS+ filesystem on the USB memory stick the xattrs should successfully survive the whiplash.

I implemented the translation in darling/src/kernel/emulation/linux/xattr/ new files: translate_namespace.{c|h} changed fgetxattr.c, flistxattr.c, fremovexattr.c, fsetxattr.c, getxattr.c, listxattr.c, removexattr.c, setxattr.c I have rebuilt and am going to test to see how it works.

darling "com.apple.FinderInfo" -> linux "user.com.apple.FinderInfo", linux "user.com.apple.FinderInfo" -> darling "com.apple.FinderInfo"

The translation amounts to darling "xattr_name" -> linux "user.xattr_name" linux "user.xattr_name" -> darling "xattr_name"

For any xattr_name not beginning with "user.": linux "xattr_name" -> darling "xattr_name"

[r-wilbur]

Private conversation from taken Discord (with small edits). r-wilbur Sunday 11 Jul 2021 at 3:05 PM The implementation of xattr translation works! In the process of getting it to work I found and fixed a bug in src/libc/gen/fts.c and semantics changed slightly in src/external/file_cmds/ls/ls.c Also fixed some bugs in src/kernel/emulation/linux/stat/fstatfs.c and statfs.c Since I have to determine whether or not the file is in a filesystem provided by darling/Mac OS in order to know how to treat the extended attribute names (translate between Linux form and OSX form), I end up calling statfs64 (when supplied filename) or fstatfs64 (when only a file descriptor is available) for every file. The statfs64 and fstatfs64 calls end up doing a string search through the mount points in the Linux /proc/mounts in order to figure out the filesystem info. This is a lot of work and slows down the operation of cp a bit when you are trying to copy the extended attributes. I was thinking of optimizations that could alleviate a lot of the repeated string searches. If it is a safe assumption that all the files in a particular path are in the same filesystem, I would be happy to implement a hash table cache in the xattr code of filesystem identification results for paths which would allow xattr code to only need to call statfs64 or fstatfs64 once for each path considered, cache the result, and subsequently look it up in the cache. The cached value would be a boolean (is_darling_fs) so the keys would take more space than the data! My problem presently is that I hadn't had the time to formulate my question and pose it to anyone, till now. I filed a bug for the issues in statfs64 and fstatfs64, #946. I haven't filed a bug against fts or ls, yet. — @facekapow Sunday 11 Jul 2021 at 11:02 PM that sounds great! about the bugfix in libc (fts.c) and the semantics change in ls: are these absolutely necessary for your xattr translation work, or are they unrelated and you just happened to stumble into them and fix them? i ask because in Darling we try to avoid modifying Apple's open source code as much as possible. this is because we try to emulate macOS as accurately as possible, so we want to keep all the same bugs as well.

a cache sounds like a good idea, but it could quickly lead to working with stale information (e.g. a mount point is unmounted but the cache, working with old information, still says it refers to a different filesystem). better to be a little slower but more accurate. — r-wilbur Monday 12 Jul 2021 at 12:03 AM Regarding the change in fts.c: the extant code checked the return value of getattrlistbulk, if -1 then it assumed the call was not supported and tried a different way to get the information. When I changed the prototype of getattrlistbulk to match that in the XNU header file and the return value from -1 to -ENOTSUP, things that used the fts code started failing in weird ways. 'ls' was generating segmentation violations. I had thought that -ENOTSUP was a preferable way to signal an unsupported feature than -1 (== -ENOPERM). That can be reverted with no hard feelings but I should make a note of why the magic value of -1 needs to be returned (or, alternatively, break down and implement getattrlistbulk). The other changes in fts.c consisted of dealing more carefully with deleted pointers (setting them to NULL since we check them for non-NULL before using them) and dealing carefully with bogus data that came from using an unimplemented call (NULL pointers and (unsigned)0 lengths that were then decremented and used as the size for allocation)!

Regarding the change in ls: listxattr is called with a NULL attribute buffer pointer for a file if ls is invoked with '-@' option to list extended attributes, the return value is supposed to be the required buffer size to accommodate the extended attribute information. If this is a darling/Mac OSX filesystem then the size will be the exact size needed because no translation of the extended attribute names is needed. In other words, all is well. If, on the other hand, this is a Linux filesystem, the size returned by the Linux system call will be larger than or equal to that required for the OSX extended attribute information because some of the names may need translating from the Linux extended attribute namespace if it has 'user.osx.' prepended to the name. As far as buffer space requirements this does not present a problem. But later on in the ls.c code it expects the extended attribute buffer to use the whole size returned in that first call.

The way I dealt with that was to translate the names in the buffer and then return the translated size. All I had to do to the code in ls.c was update xattr_size by setting it equal to the return value of the second invocation of listxattr. In order to obviate that need, I will have to check whether the filesystem is darling/Mac OSX and if not, and the attribute buffer pointer is NULL, then create a buffer, pass it in, do the translation, destroy the buffer, and return the translated size. The other changes in the ls code were simply stability improvements like checking for NULL after allocation and before using pointers. — facekapow Tuesday 13 Jul 2021 at 10:41 AM if u had to modify the ls code to be compatible with your changes, then perhaps your implementation is missing something in order to be compatible with the way it works on a real macOS installation. from what u said above, it sounds like you need to calculate the correct unprefixed size on the first call. — r-wilbur Tuesday 13 Jul 2021 at 10:44 AM I've coded around the issues in libc/gen/fts.c and external/file_cmds/ls/{ls.c|print.c} and dropped all those changes to libc and ls. I rebuilt and tested with no problems. I also took the opportunity to improve the architecture a little bit and it is not as slow--I think about where it used to be. I changed the listxattr and flistxattr implementations to normal linux syscalls if no name translation needed. If translation needed, then I make a syscall to determine what size buffer Linux will need, allocate the buffer and make the syscall to get the names in the buffer, translate them, then report the actual size. It works without any change to any other code (aside from the fixes to fstatfs/statfs detailed in bug #946). — facekapow Tuesday 13 Jul 2021 at 10:50 AM nice! in that case, you can prob open up a PR now (if you think it's ready; sounds to me like it is) — r-wilbur Tuesday 13 Jul 2021 at 10:55 AM I was thinking that I would first pursue the fixes to fstatfs/statfs on bug #946 as those are critical to the correct operation of the xattr translation (determining whether translation needed for a particular filesystem).