nirbheek
commented
6 years ago Open nirbheek opened 7 years ago
nirbheek
commented
6 years ago 
raimue
commented
6 years ago It would be helpful to have a way to get the filename extension a shared_module() gets. Then we could pass this to the program via some -DPLUGIN_SUFFIX or a config.h and use this macro instead of relying on G_MODULE_SUFFIX from glib, which is currently wrong on macOS (see https://github.com/mesonbuild/meson/issues/3053#issuecomment-365005193).
With such a function to get the default filename extension, it would continue to work even if meson later decides to change it.
raimue
commented
6 years ago On a closer look, the .dylib suffix might actually be wrong as the files have a different filetype in the Mach-O header. otool -hv on a shared module shows "BUNDLE" and on a shared library "DYLIB".
GNU libtool has always used ".so" on Darwin/macOS for shared modules, so it might make sense to just keep it that way.
albfan
commented
5 years ago @nirbheek, maybe related https://gitlab.gnome.org/GNOME/gitg/issues/145#note_352754. Although homebrew tries to mimic linux on mac, so using so as suffix could be a false positive here.
albfan
commented
5 years ago Opps. That was on purpose: https://gitlab.gnome.org/GNOME/glib/merge_requests/280/diffs
neverpanic
commented
1 year ago macOS itself uses three possible formats for loadable modules, .so, .bundle, or no extension in subfolder of a folder named ${name}.bundle.
For example, there are loadable modules for the SASL authentication framework in /usr/lib/sasl2 on macOS Ventura, which use .so. Note that this location (and all the others I will look at) is protected by system integrity protection, which means it cannot be edited by users, even with root privileges. All these files, consequently, are directly from Apple.
$ ls -lash /usr/lib/sasl2/
total 1256
0 drwxr-xr-x 22 root wheel 704B Apr 1 18:46 .
0 drwxr-xr-x 32 root wheel 1.0K Apr 1 18:46 ..
48 -rwxr-xr-x 1 root wheel 164K Apr 1 18:46 apop.so
40 -rwxr-xr-x 1 root wheel 164K Apr 1 18:46 atoken.so
216 -rwxr-xr-x 1 root wheel 344K Apr 1 18:46 dhx.so
104 -rwxr-xr-x 1 root wheel 231K Apr 1 18:46 digestmd5WebDAV.so
48 -rwxr-xr-x 1 root wheel 165K Apr 1 18:46 libanonymous.2.so
48 -rwxr-xr-x 1 root wheel 165K Apr 1 18:46 libcrammd5.2.so
112 -rwxr-xr-x 1 root wheel 231K Apr 1 18:46 libdigestmd5.2.so
80 -rwxr-xr-x 1 root wheel 217K Apr 1 18:46 libgssapiv2.2.0.18.so
0 lrwxr-xr-x 1 root wheel 21B Apr 1 18:46 libgssapiv2.2.so -> libgssapiv2.2.0.18.so
80 -rwxr-xr-x 1 root wheel 198K Apr 1 18:46 libntlm.so
48 -rwxr-xr-x 1 root wheel 165K Apr 1 18:46 libplain.2.so
48 -rwxr-xr-x 1 root wheel 165K Apr 1 18:46 login.so
56 -rwxr-xr-x 1 root wheel 166K Apr 1 18:46 mschapv2.so
40 -rwxr-xr-x 1 root wheel 164K Apr 1 18:46 oauthbearer.so
0 drwxr-xr-x 5 root wheel 160B Apr 1 18:46 openldap
40 -rwxr-xr-x 1 root wheel 164K Apr 1 18:46 plain-clienttoken.so
40 -rwxr-xr-x 1 root wheel 165K Apr 1 18:46 pwauxprop.so
56 -rwxr-xr-x 1 root wheel 167K Apr 1 18:46 shadow_auxprop.so
56 -rwxr-xr-x 1 root wheel 165K Apr 1 18:46 smb_ntlmv2.so
96 -rwxr-xr-x 1 root wheel 202K Apr 1 18:46 srp.soAs previously outlined above by @raimue, these files all have a filetype of MH_BUNDLE (0x8), not MH_DYLIB (0x6), see also /Library/Developer/CommandLineTools/SDKs/MacOS.sdk/usr/include/mach-o/loader.h:
$ otool -hv /usr/lib/sasl2/atoken.so /usr/lib/libgmalloc.dylib
/usr/lib/sasl2/atoken.so:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL 0x00 BUNDLE 16 1136 NOUNDEFS DYLDLINK TWOLEVEL
/usr/lib/libgmalloc.dylib:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL 0x00 DYLIB 16 1616 NOUNDEFS DYLDLINK TWOLEVEL NO_REEXPORTED_DYLIBS APP_EXTENSION_SAFEFor other loadable modules, e.g. those associated with perl, Apple uses .bundle:
$ otool -hv /System/Library/Perl/5.34/darwin-thread-multi-2level/auto/Hash/Util/Util.bundle
/System/Library/Perl/5.34/darwin-thread-multi-2level/auto/Hash/Util/Util.bundle:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL 0x00 BUNDLE 14 1336 NOUNDEFS DYLDLINK TWOLEVELAdditionally, Apple also uses .bundle for directories that contain (similar to how .app bundles are structured) the loadable module at Contents/MacOS/$name, e.g. in /System/Library/Security/ldapl.bundle:
$ otool -hv ./Security/ldapdl.bundle/Contents/MacOS/ldapdl
./Security/ldapdl.bundle/Contents/MacOS/ldapdl:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL 0x00 BUNDLE 19 1752 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAKSome old documents from Apple (that I could no longer find on the Apple website) suggest that .bundle is the convention for file types that use MH_BUNDLE as header:
The MH_BUNDLE file type is the type typically used by code that you load at runtime (typically called bundles or plug-ins). By convention, the file name extension for this format is .bundle.
I ran a quick check for all files that claim to be of type MH_BUNDLE in /usr/lib, /usr/libexec and /System/Library, and could not find a single .dylib file, which suggests that the behavior currently used by meson is wrong. Meson should thus either use .so, or .bundle, and given that using .bundle would break existing code that worked when autoconf build systems were used, I would recommend using .so.
For your own analysis, the command I used to locate all bundle files was
find /usr/lib /usr/libexec /System/Library -type f -exec bash -c 'for file; do if otool -hv "$file" |& grep -q BUNDLE; then echo "$file"; fi; done' {} +Edit: My search did find three .dylib files that are bundles:
/System/Library/Frameworks/OpenCL.framework/Versions/A/Libraries/AMDil.dylib
/System/Library/Frameworks/CoreImage.framework/Versions/A/Frameworks/libWrapGL.dylib
/System/Library/Frameworks/CoreImage.framework/Versions/A/Frameworks/libWrapGLES.dylibI think those are the exception, not the rule.
neverpanic
commented
1 year ago CMake also uses .so, btw: https://gitlab.kitware.com/cmake/cmake/-/blob/master/Modules/Platform/Darwin.cmake#L47
Midar
commented
2 months ago I would recommend going with the bundle directory structure as that is the only thing that can be code signed. However, these have a different structure for iOS and macOS. There is also some discussion in https://github.com/audacious-media-player/audacious/issues/1364 which references this issue.
mohd-akram
commented
2 months ago Not using .so breaks any project using libtool for modules. Just ran into this with PulseAudio.
Midar
commented
2 months ago .so is the extension for ELF shared objects. macOS/iOS does not support those, so it is the wrong extension.
neverpanic
commented
2 months ago
.sois the extension for ELF shared objects. macOS/iOS does not support those, so it is the wrong extension.
This isn't correct. There's plenty of prior art of using .so on macOS for Mach-O loadable modules as outlined above in https://github.com/mesonbuild/meson/issues/1160#issuecomment-1511157623, for example for the SASL loadable modules shipped by Apple.
That being said, there are also plenty of loadable modules in .dylib files on macOS, so this seems like a bug in libtool that libtool should fix.
Midar
commented
2 months ago
.sois the extension for ELF shared objects. macOS/iOS does not support those, so it is the wrong extension.This isn't correct. There's plenty of prior art of using
.soon macOS for Mach-O loadable modules as outlined above in #1160 (comment), for example for the SASL loadable modules shipped by Apple.
Sorry, but this is just plain wrong. Yes, Python ships as part of macOS. And yes, Python got it wrong. Does Apple shipping 3rd party software that gets it wrong suddenly make it right? I doubt so. Please provide a link to the Apple documentation that states that .so is an acceptable extension for Mach-O (spoiler alert: it's not).
That being said, there are also plenty of loadable modules in
.dylibfiles on macOS, so this seems like a bug in libtool that libtool should fix.
I agree on that.
The correct format for extensions on macOS and iOS is bundles, though. See https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/AboutBundles/AboutBundles.html and https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/BundleTypes/BundleTypes.html#//apple_ref/doc/uid/10000123i-CH101-SW32
raimue
commented
2 months ago At least the problem has been mititgated in the glib world with the changes to the g_module_open() API only taking the basename without file extension. Depending on the platform, it will now try possible combinations with and without the "lib" prefix and different file extensions (thank you @nirbheek): https://gitlab.gnome.org/GNOME/glib/-/merge_requests/2950
For over 20 years, software ported to macOS has just used the .so file extension for compatibility. Even if Apple had intended that software should be using .bundle, the .so filename extension was adopted by the community as the de facto standard. In fact, Apple states at the very top of the previously linked bundle documentation:
Note: Although bundles are one way of packaging executable code, they are not the only way that is supported. UNIX shell scripts and command-line tools do not use the bundle structure, neither do static and dynamic shared libraries.
For sure in other places Apple's documentation will also always tell you that applications are being shipped as .app bundles. But I assume you also do not expect meson to always create .app bundles instead of plain executable files? No, the exception for command-line tools applies to this case.
A .bundle is a directory structure that requires more files than what is now being done with plain files. The ported software will never use anything from the Info.plist, why should developers be forced to create it?
To follow up on your earlier argument, codesign works just fine with a MH_BUNDLE file. There is no strict requirement to wrap inside a .bundle directory structure just for code signing. You are aware that code signing also works for plain executable files without an .app bundle? So why do you think that a .bundle is required to codesign a loadable module? The signature will be attached directly to the Mach-O file.
I think the only case in which a bundle directory structure is strictly required would be for provisioning profiles. But as far as I am aware, these only exist for the whole program anyway and there is no way to have provisioning profiles or entitlements on a loadable module, as these are applied to the whole program. https://developer.apple.com/documentation/xcode/signing-a-daemon-with-a-restricted-entitlement
Midar
commented
2 months ago The macOS and iOS developer community has most certainly not adopted .so as an extension. Some lazy programmers coming from other UNIX systems and not properly porting their software to macOS have. But from my understanding, Meson wants to be a build system that supports platforms properly instead of trying to use a hammer to make all OSes behave as closely as possible as Linux.
The argument with .app is a little bit weak, as that's for graphical apps. Command line apps, as shipped by Apple, are not .apps. So there is precedent in first party software not being an .app. But there is absolutely zero precedent for first party software using .so or using .bundle on anything that is not a directory.
The situation is bad enough as it is with software not supporting macOS / iOS natively. Let's not contribute to this by Meson requiring software to not support macOS / iOS properly. There is a native way to handle plugins, let's use it. Generating an Info.plist from Meson would be a possibility, too, if that is your concern.
raimue
commented
2 months ago You already have to write custom scripts to create an .app bundle anyway (see #48). Nobody stops you from creating a .bundle directory structure after meson has built the loadable module. You can rename a foobar.so to a foobar.bundle/Contents/foobar, if you prefer. Then you also need to adapt the dlopen() code to make it find the loadable module. But I still do not see the benefit? Why would you make it more complicated than it has to be?
After all there is no technical reason to prefer a specific file extension for loadable modules. It all just comes down to finding a convention that both build system and the code loading the module will agree upon.
To find a common ground, I think we all agree that .dylib was not the best choice. Changing the default of .dylib to .so would allow some projects to drop conditional code setting the name_suffix property to .so. Using .so would be in line with other build tools (in particular libtool and cmake) that settled on using .so for loadable modules on macOS. But of course such a change would most probably also break other software that relied on the wrong .dylib default that was kept in meson for so long.
Midar
commented
2 months ago You already have to write custom scripts to create an
.appbundle anyway (see #48). Nobody stops you from creating a.bundledirectory structure after meson has built the loadable module. You can rename afoobar.soto afoobar.bundle/Contents/foobar, if you prefer. Then you also need to adapt thedlopen()code to make it find the loadable module. But I still do not see the benefit? Why would you make it more complicated than it has to be?
By that logic, you would also be okay to just rename a .dylib to .framework and call it a day. Why lie about what something is via the file extensions? Of course you can name a file however you want. You can call it .txt if you want. But that doesn't make it correct.
After all there is no technical reason to prefer a specific file extension for loadable modules. It all just comes down to finding a convention that both build system and the code loading the module will agree upon.
Then let's use no extension -- because that is the only extension you can find for plugins themselves that are not in a bundle.
To find a common ground, I think we all agree that
.dylibwas not the best choice.
Certainly a 100x better choice than .so. At least it's not lying. It is a dynamic library of type MH_BUNDLE, after all.
Changing the default of
.dylibto.sowould allow some projects to drop conditional code setting thename_suffixproperty to.so.
Those projects should be fixed instead.
Using
.sowould be in line with other build tools (in particular libtool and cmake) that settled on using.sofor loadable modules on macOS.
Why are you so keen on breaking the ecosystem that already is quite broken even further? That bug should be fixed in libtool and cmake, not propagated to meson!
But of course such a change would most probably also break other software that relied on the wrong
.dylibdefault that was kept in meson for so long.
If you insist on doing something that is not how it is done on the platform, at least .dylib is the least terrible of the options.
But let me make another proposal: If we're going to lie and intentionally break the ecosystem anyway, why not .dll? That is at least consistent with what Windows does and everybody knows what a DLL is. And it certainly isn't more wrong than .so.
eli-schwartz
commented
2 months ago The correct file extension is, of course, "whichever one the software that loads it is looking for". Everything is just a convention, if you care enough to change it.
And the Apple convention is that different projects baked their own conventions for convenience. The Apple convention is also "for meson projects, use dylib", no? Changing the output filename runs the risk of breaking existing projects in the wild.
Perhaps the answer is to add a kwarg, similar to the way we already special-case darwin with darwin_versions:. For shared_modules it would be darwin_suffix. Unsure whether that's better than just conditional definitions.
I don't think it's possible to have a one-size-fits-all extension, given that there are always going to be projects that hardcode a filename to load rather than iterating over many possible formats, and given that even if projects use -DPLUGIN_EXT=' + shmod.full_path().split('.')[-1] there will still be projects where a plugin is built with meson for another project that isn't built with meson and may have any extension style, so users do have to unfortunately control that.
So the answer will always have to be "users are empowered to deal with the preexisting world they have to live in".
Currently we default to
dylib. This can be changed with thename_suffixkwarg, but perhaps our default should besoorbundle. This needs investigation.Continued from https://github.com/mesonbuild/meson/pull/1126 and https://github.com/mesonbuild/meson/issues/1112