Help with building on nixpkgs for macOS

[pcasaretto]

Operating System

MacOS

What's the issue you encountered?

(As this is not a "bug", I've removed the [Bug] tag from the title.)

Hi there! Recently discovered imhex and was blown away. Thank you for producing such a great tool and making it free and open source. I manage my machine (a Mac) using nix and I've been trying to contribute back to the community by adding missing darwin software for nixpkgs. When attempting to build imhex inside a nixpkgs environment I'm facing the following error:

/tmp/nix-build-imhex-1.33.2.drv-7/source/lib/external/pattern_language/lib/source/pl/core/evaluator.cpp:1071:18: error: no member named 'uncaught_exceptions' in namespace 'std'
 1071 |         if (std::uncaught_exceptions() > 0)
      |             ~~~~~^

I've been learning c++ as I debug this issue and so far these are the things I've discovered.

• std::uncaught_exceptions is part of libc++ 17
• I've confirmed that it is included as a library when compiling evaluator.cpp with the following command coming from nixpkgs wrapping of clang++

  -isystem
  /nix/store/g91h1mvq0a3bplzlwg2rbz1d25mlc887-libcxx-17.0.6-dev/include/c++/v1

How can the issue be reproduced?

In an environment where nix is installed

1. Download my branch of nixpkgs pcasaretto/nixpkgs/imhex-darwin
2. nix-build . -A imhex

ImHex Version

1.33.2

ImHex Build Type

• [ ] Nightly or built from sources

Installation type

source

Additional context?

No response

[pcasaretto]

Alternatively, if this proves unfruitful, the guidelines for the nixpkgs community suggest offering a pre-built package.

[WerWolv]

Hey! I believe there already is a Nix flake file available for ImHex on the official Nix repo though I'm not sure if it supports macOS.

In any case, your build issues most likely stem from the fact that ImHex simply doesn't support Apple Clang. You need to download LLVM Clang using brew to do anything with ImHex. The reason for that is that Apple sadly leaves their toolchain severely outdated constantly and locks specific versions of it to specific versions of its OS which makes it really annoying to properly write software that supports a wide range of devices.

[pcasaretto]

though I'm not sure if it supports macOS.

exactly what I'm trying to remedy!

You need to download LLVM Clang using brew to do anything with ImHex.

nixpkgs provides that, here's some info for the environment where I'm currently building:

[nix-shell:~/src/github.com/pcasaretto/nixpkgs]$ which clang
/nix/store/i4pd9fw4x45kdn0c9v2xd1lkd7k6a4aw-clang-wrapper-17.0.6/bin/clang

[nix-shell:~/src/github.com/pcasaretto/nixpkgs]$ clang -v
HARDENING: disabled flags: pie fortify3 zerocallusedregs trivialautovarinit
HARDENING: Is active (not completely disabled with "all" flag)
HARDENING: enabling pic
HARDENING: enabling format
HARDENING: enabling stackprotector
HARDENING: enabling fortify
HARDENING: enabling strictoverflow
extra flags before to /nix/store/5wfzz9a0h284q33r0nb6djhinbb22wg0-clang-17.0.6/bin/clang:
  ''
original flags to /nix/store/5wfzz9a0h284q33r0nb6djhinbb22wg0-clang-17.0.6/bin/clang:
  -v
extra flags after to /nix/store/5wfzz9a0h284q33r0nb6djhinbb22wg0-clang-17.0.6/bin/clang:
  ''
clang version 17.0.6
Target: aarch64-apple-darwin23.5.0
Thread model: posix
InstalledDir: /nix/store/5wfzz9a0h284q33r0nb6djhinbb22wg0-clang-17.0.6/bin

[WerWolv]

Oh sorry. I guess I don't understand nix very well.

We use Clang 17.0.5 in the CI and that works fine. Could it be that we're missing some include for these functions and Nix's libc++ is just configured slightly differently? std::uncaught_exceptions() should be in the <exception> header

[pcasaretto]

Could it be that we're missing some include for these functions

How can I check that? Can you point me in the right direction?

[pcasaretto]

Adding an #include <exception> just after the include for <utility> advanced the build. The error changed to another file:

/Users/pcasaretto/src/github.com/pcasaretto/nixpkgs/source/lib/libimhex/include/hex/api/event_manager.hpp:127:51: error: no member named 'any_of' in namespace 'std'
  127 |                 const auto eventRegistered = std::any_of(begin, end, [&](auto &item) {
      |                                              ~~~~~^

[WerWolv]

Okay that definitely means I'm missing some includes in a few places. Do you have any test setup I can use to fix them? I have a mac but never used nix on it

[pcasaretto]

If you have a working nix installation you can download my branch of nixpkgs (pcasaretto/nixpkgs/imhex-darwin). A nix build . -A imhex on that folder will build and show the errors, however I'm using an alternate approach to facilitate testing. Here's the commands I'm using:

1. nix shell . -A imhex this sets up a bash with all dependencies satisfied
2. Inside that shell I issue nix specific commands to move forward with the build
3. runPhase unpackPhase
4. runPhase patchPhase
5. runPhase configurePhase
6. runPhase buildPhase

I then use code . to start a VSCode instance with all the correct paths to the tools required and edit/tweak the files and repeat the build command.

[WerWolv]

Hey, I tried setting it up now but nix doesn't seem to recognize any of those -A imhex arguments. I did a shallow clone of your repo's imhex-darwin branch and just ran the commands in there

[pcasaretto]

My bad! Those are the "new" commands. Please try again with nix-build and nix-shell instead of nix build and nix shell.

[WerWolv]

Got it to build, thanks! Okay the issue is that there's a missing #include <algorithm> in event_manager.hpp and a missing #include <exception> in evaluator.cpp. Both of these includes have been added since. The Nix package seems to build a really old version of ImHex so I suggest you update it to v1.35.4

[pcasaretto]

Updated! Now I'm getting an error mentioning zstd.

clang++: error: cannot compress debug sections (zstd not enabled) [-Werror,-Wdebug-compression-unavailable]
make[2]: *** [plugins/yara_rules/libromfs/generator/CMakeFiles/generator-yara_rules.dir/build.make:76: plugins/yara_rules/libromfs/generator/CMakeFile
s/generator-yara_rules.dir/source/main.cpp.o] Error 1

Using nix-shell I can see zstd is available:

$ which zstd
/nix/store/4k1x14dk2mn5v1kag22kzlxzr3n9ak36-zstd-1.5.6-bin/bin/zstd

And the build log shows it is found:

- Performing Test ZSTD_AVAILABLE_COMPILER                                                                                                            
-- Performing Test ZSTD_AVAILABLE_COMPILER - Success                                                                                                  
-- Performing Test ZSTD_AVAILABLE_LINKER                                                                                                              
-- Performing Test ZSTD_AVAILABLE_LINKER - Success                                                                                                    
-- Performing Test COMPRESS_AVAILABLE_COMPILER                                                                                                        
-- Performing Test COMPRESS_AVAILABLE_COMPILER - Success                                                                                              
-- Performing Test COMPRESS_AVAILABLE_LINKER                                                                                                          
-- Performing Test COMPRESS_AVAILABLE_LINKER - Success                                                                                                
Using Zstd compression for debug info because both compiler and linker support it                                                                     
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY                                                                                                     
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Failed                                                                                            
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY                                                                                              
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Failed                                                                                     
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR                                                                                                       
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Failed                                                                                              
-- Performing Test COMPILER_HAS_DEPRECATED                                                                                                            
-- Performing Test COMPILER_HAS_DEPRECATED - Failed                                                                                                   
-- Enabling decompression support using ZLIB (1.3.1)                                                                                                  
-- Found BZip2: /nix/store/6gl2073zk4b7nzf5idf82z739w4291yg-bzip2-1.0.8/lib/libbz2.dylib (found version "1.0.8")                                      
-- Looking for BZ2_bzCompressInit                                                                                                                     
-- Looking for BZ2_bzCompressInit - found                                                                                                             
-- Enabling decompression support using BZip2 ()                                                                                                      
-- Found LibLZMA: /nix/store/pl5mqfw4c0z3g56nid8xb90yx640vy4r-xz-5.4.6/lib/liblzma.dylib (found version "5.4.6")                                      
-- Enabling decompression support using LibLZMA ()                                                                                                    
-- Found ZSTD: /nix/store/38z0bsp6xw3vvqjd86z75nrsv4lf1bqs-zstd-1.5.6/lib/libzstd.dylib                                                               
-- Found Zstd: /nix/store/38z0bsp6xw3vvqjd86z75nrsv4lf1bqs-zstd-1.5.6/lib/libzstd.dylib

I've pushed my changes to that same branch.

[WerWolv]

That error is about compression of debug info. Seems like your version of clang doesn't support it. You can add -DIMHEX_COMPRESS_DEBUG_INFO=OFF to the cmake flags and it will stop trying it

[pcasaretto]

Got a build! But it resulted in this:

Looking at the MacOS build instructions I noticed the IMHEX_GENERATE_PACKAGE option. My guess is that would "complete" the bundle. When activating that I got the following errors:

warning: cannot resolve item '/usr/lib/libSystem.B.dylib'

  possible problems:
    need more directories?
    need to use InstallRequiredSystemLibraries?
    run in install tree instead of build tree?

-- 
warning: cannot resolve item '/usr/lib/libresolv.9.dylib'

  possible problems:
    need more directories?
    need to use InstallRequiredSystemLibraries?
    run in install tree instead of build tree?

-- 
warning: cannot resolve item '/usr/lib/libXplugin.1.dylib'

  possible problems:
    need more directories?
    need to use InstallRequiredSystemLibraries?
    run in install tree instead of build tree?

Nix builds are hermetic, so I thought to myself: "maybe I just need to include a MacOS SDK input and let CMake know". So I did that and set CMAKE_OSX_SYSROOT to the correct path:-DCMAKE_OSX_SYSROOT=/nix/store/fxpdfv1fkipfc66n972lvwhgc035bvs0-Ma cOSX-SDK-11.0.0.

Same error. Investigating further I noticed that that particular dylib is not present in that SDK bundle:

❯ fd libSystem /nix/store/fxpdfv1fkipfc66n972lvwhgc035bvs0-MacOSX-SDK-11.0.0/usr/
/nix/store/fxpdfv1fkipfc66n972lvwhgc035bvs0-MacOSX-SDK-11.0.0/usr/lib/libSystem.B.tbd
/nix/store/fxpdfv1fkipfc66n972lvwhgc035bvs0-MacOSX-SDK-11.0.0/usr/lib/libSystem.B_asan.tbd
/nix/store/fxpdfv1fkipfc66n972lvwhgc035bvs0-MacOSX-SDK-11.0.0/usr/lib/libSystem.tbd
/nix/store/fxpdfv1fkipfc66n972lvwhgc035bvs0-MacOSX-SDK-11.0.0/usr/lib/libSystem_asan.tbd

AND, it's not even present in my own system:

❯ fd libSystem /usr/lib

Resorting to Google, it seems the rabbit hole is deeper than I thought...

https://daiderd.com/2020/06/25/nix-and-libsystem.html

Including darwin.Libsystem to the buildInputs messes up the build horribly:

In file included from /tmp/nix-build-imhex-1.35.4.drv-6/source/lib/external/libwolv/libs/utils/source/utils/string.cpp:3:                             
In file included from /tmp/nix-build-imhex-1.35.4.drv-6/source/lib/external/libwolv/libs/utils/include/wolv/utils/string.hpp:3:
In file included from /tmp/nix-build-imhex-1.35.4.drv-6/source/lib/external/libwolv/libs/types/include/wolv/types.hpp:1:
/nix/store/g91h1mvq0a3bplzlwg2rbz1d25mlc887-libcxx-17.0.6-dev/include/c++/v1/cstdint:149:5: error: <cstdint> tried including <stdint.h> but didn't fin
d libc++'s <stdint.h> header.           This usually means that your header search paths are not configured properly.           The header search path
s should contain the C++ Standard Library headers before           any C Standard Library, and you are probably using compiler flags that make that   
        not be the case.
  149 | #   error <cstdint> tried including <stdint.h> but didn't find libc++'s <stdint.h> header. \
      |     ^
In file included from /tmp/nix-build-imhex-1.35.4.drv-6/source/lib/external/libwolv/libs/utils/source/utils/string.cpp:3:
In file included from /tmp/nix-build-imhex-1.35.4.drv-6/source/lib/external/libwolv/libs/utils/include/wolv/utils/string.hpp:5:
In file included from /nix/store/g91h1mvq0a3bplzlwg2rbz1d25mlc887-libcxx-17.0.6-dev/include/c++/v1/algorithm:1741:
/nix/store/g91h1mvq0a3bplzlwg2rbz1d25mlc887-libcxx-17.0.6-dev/include/c++/v1/cstddef:46:5: error: <cstddef> tried including <stddef.h> but didn't find
 libc++'s <stddef.h> header.           This usually means that your header search paths are not configured properly.           The header search paths
 should contain the C++ Standard Library headers before           any C Standard Library, and you are probably using compiler flags that make that    
       not be the case.
   46 | #   error <cstddef> tried including <stddef.h> but didn't find libc++'s <stddef.h> header. \
      |     ^
/nix/store/g91h1mvq0a3bplzlwg2rbz1d25mlc887-libcxx-17.0.6-dev/include/c++/v1/cstddef:59:9: error: no member named 'nullptr_t' in the global namespace
   59 | using ::nullptr_t;
      |       ~~^

Searching for Lybsystem in nixpkgs returns a lot of results. I'm looking at a few looking for inspiration.

[WerWolv]

Did you run make install after the build with IMHEX_GENERATE_PACKAGE=ON suceeded? Because that copies and sets up a lot of things needed for it to work. Also if you're on a M1 machine, you need to properly sign the bundle as well, otherwise the OS will just refuse to open it with the most obscure ass error messages they could have come up with

[pcasaretto]

Sorry, should have made it clearer. That output is part of the "installPhase" of nix build. The exact command run is

make -j10 SHELL=/nix/store/s30jrpgav677fpc9yvkqsib70xfmx7xi-bash-5.2p26/bin/bash gsettingsschemadir=/nix/store/zffq7ccnf0jgi02fps0k6jl678xq6mxw-imh
ex-1.35.4/share/gsettings-schemas/imhex-1.35.4/glib-2.0/schemas/ install

[pcasaretto]

I have the feeling that this is what is happening here: https://gitlab.kitware.com/cmake/cmake/-/issues/21569

[pcasaretto]

This didn't work https://github.com/WerWolv/ImHex/commit/e1085dc1f94ecd0db8fed066ac858ab58b703a4d

[pcasaretto]

Giving up. I'll work instead on adding an official pre-built binary to nixpkgs.

[pcasaretto]

Good news @WerWolv ! With a little help from the great people at nixpkgs/darwin-mantainers we've managed to push this forward. We have a working build that's looking really good. However, it's still missing some stuff inside the bundle. On the latest release there are 3 things that we're missing that I need your help with:

• fonts.hexpluglib
• ui.hexpluglib
• and the nodes dir

both hexpluglib files are also present in the plugins dir. Are these files inside Contents/MacOS needed/useful? regarding nodes, where does that come from?

[WerWolv]

That's great news! fonts.hexpluglib and ui.hexpluglib are basically plugins like the builtin, visualizer or hash plugin but they act as libraries for other plugins (so plugins can link to them). They are required and need to be in the plugins folder as well. I think macOS really isn't happy about this and tries to move them into the Frameworks folder or something but my packaging code moves them back afterwards.

The nodes folder comes from the ImHex-Patterns Repo and should be downloaded like the other folders there too

[pcasaretto]

Great. That matches our findings. We are almost there! Gonna go ahead and close this. Thank you.

[pcasaretto]

Hey WerWolv, it's me again 😅

When trying to compile for x86-64 darwin, we encountered an error. At first, it was just a misaligned SDK version, but as soon as I fixed that I was met with:

In file included from /tmp/nix-build-imhex-1.35.3.drv-0/ImHex-source-1.35.3/plugins/builtin/source/content/providers/process_memory_provider.cpp:13:  
In file included from /nix/store/2l9s8m6hb82d4qghp9bsqd65radv11q1-libSystem-11.0.0/include/mach/arm/vm_types.h:72:                                    
/nix/store/2l9s8m6hb82d4qghp9bsqd65radv11q1-libSystem-11.0.0/include/arm/_types.h:55:3: error: typedef redefinition with different types ('union __mbs
tate_t' vs 'union __mbstate_t')                                                                                                                       
   55 | } __mbstate_t;                                                                                                                                
      |   ^                                                                                                                                           
/nix/store/2l9s8m6hb82d4qghp9bsqd65radv11q1-libSystem-11.0.0/include/i386/_types.h:79:3: note: previous definition is here                            
   79 | } __mbstate_t;
      |   ^

I tracked down the import to this file https://github.com/WerWolv/ImHex/blob/de8465a8f4f4945d1c87a499872749b65e522e2d/plugins/builtin/source/content/providers/process_memory_provider.cpp#L13. It seemed strange that the code was importing an architecture-specific header. Changing this to use the "machine" headers, which choose the correct architecture on compile time (example) fixes the problem. What's extra strange is that your current release for x86 darwin is newer than the code that introduced this apparently problematic import. This makes me think that I'm missing something.

[pcasaretto]

Hey @WerWolv , just a friendly nudge here. I'd be happy to submit a PR with a fix if you could enlighten me as to how the recent builds dont trip on this.

[WerWolv]

Sorry, I didn't see your past message. The code you linked is absolutely not correct and will not work for x86 macs. I assume that the files are still available in the SDK somewhere so it manages to include them but the generated code will not work. The fix would be to move those includes (and the code depending on it) behind an #if defined(__aarch64__) / #endif and then providing separate x86 implementations guarded behind a #if defined(__x86_64__) / #endif. No idea if the rest of that class will work though on old macs, I have no real way of testing it.

[pcasaretto]

Forgive my ignorance, but isn't that what this header file is doing?

[pcasaretto]

Hey @WerWolv ! Took a stab at fixing it i #1910