Closed apfelkraut closed 2 years ago
If you’re building for yourself most of the diffs from BUILD.txt are not required because they are needed only for old deployment target (building for older macOS versions). Also you don’t have to build everything from source, so installation should be fast.
As you mentioned some packages are not detected by darktable when installed through homebrew. I have no idea why it’s that way, nor have any desire to investigate it. Without fixing these problems I would say homebrew shouldn’t be considered a supported way to build darktable.
So I see 2 ways forward:
1). document a simplified build using macports (minimal changes to packages, only works locally);
2). fix homebrew problems and then document a homebrew build.
One more note: in order to have homebrew supported we actually need an active darktable developer using macOS and homebrew. This is the only way to keep track of homebrew changes (e.g. bugs) and have build instructions up-to-date. Fixing mentioned problems with dependencies from homebrew would be a good sign that someone actually cares enough for this to work.
Right now enough attention is paid only to macports.
Thanks a lot for your feedback @parafin.
Just to add to the above description: When starting darktable I get the following output. Probably this is also not how it is supposed to be, especially the last line?
(process:90757): GLib-GObject-CRITICAL **: 21:12:22.607: g_object_set: assertion 'G_IS_OBJECT (object)' failed
(darktable:90757): GLib-GObject-WARNING **: 21:12:23.029: invalid cast from 'GtkMenuBar' to 'GtkWindow'
(darktable:90757): Gtk-CRITICAL **: 21:12:23.029: gtk_window_add_accel_group: assertion 'GTK_IS_WINDOW (window)' failed
[iop_lens]: could not load lensfun database in `~/bin/darktable-dev/share/lensfun/version_2'!
First three messages have been there since forever, no idea where they come from, but doesn’t look like it affects anything. Last one concerns lensfun, and I think you have an alpha release version of lensfun installed, which is not advisable. It’s better to downgrade to latest stable release. But error message itself is probably expected, because we also look for lensfun DB in the same prefix as darktable directory structure.
Thanks @parafin. The lensfun database at questions is via homebrew the same as via macports: 0.3.95.
In my case copying the database from its default location to my build directory resolves the issue: cp -R /usr/local/Cellar/lensfun/0.3.95_4/share/lensfun ~/bin/darktable-dev/share/
.
I'm guessing then 0.3.95-specific code for loading DB if flawed here: https://github.com/darktable-org/darktable/blob/c6e8ed5447d57ba22a146f6d890462a80e002b8e/src/iop/lens.cc#L1250-L1258
As I say it's not recommended for use and not very well tested.
Regarding the following not found optional package (see above):
* LLVM (required version >= 3.9)
I need to add homebrew's llvm to PATH:
export PATH="/usr/local/opt/llvm/bin:$PATH"
Regarding the following not found optional package (see above):
* LibSoup2
I need to add homebrew's libsoup@2 to PKG_CONFIG_PATH:
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:/usr/local/opt/libsoup@2/lib/pkgconfig"
Regarding the following not found optional package (see above):
* ICU
* OSMGpsMap
I need to add homebrew's icu4c to PKG_CONFIG_PATH:
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:/usr/local/opt/icu4c/lib/pkgconfig"
Interestingly this also helps osm-gps-map to be found by pkgconfig.
Regarding the other two packages - colord and imagemagick - as mentioned above, I now have some more understanding:
In sum my approach to build darktable using homebrew now looks like this and produces a nicely working darktable executable:
# Prerequisites
# - Currently works only on i386 (x86_64) or on arm (m1) using Rosetta.
# - Homebrew is running within the default paths and is up-to-date.
# If you have an arm (m1) Mac follow https://stackoverflow.com/a/68443301 to
# make homebrew build for i386. Before you run any steps below, verify that you
# are within your i386 environment.
# - darktable sources are available at ~/src/darktable, the binary should
# go to ~/bin/darktable-dev.
#
# (tested running macOS 12.1 on m1)
# Install dependencies
brew install adwaita-icon-theme cmake desktop-file-utils exiv2 gettext git glib gmic gphoto2 graphicsmagick gtk+3 gtk-mac-integration icu4c intltool iso-codes json-glib lensfun libavif libheif librsvg libsecret libsoup@2 little-cms2 lua libomp ninja openexr openjpeg osm-gps-map portmidi pugixml sdl2 cmocka curl perl jpeg llvm po4a
# Add LLVM to PATH
export PATH="$(brew --prefix)/opt/llvm/bin:$PATH"
# Add libsoup@2 to PKG_CONFIG_PATH
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$(brew --prefix)/opt/libsoup@2/lib/pkgconfig"
# Add ICU to PKG_CONFIG_PATH
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$(brew --prefix)/opt/icu4c/lib/pkgconfig"
# Build darktable
~/src/darktable/build.sh --install --build-type RelWithDebInfo --prefix ~/bin/darktable-dev
# Copy lensfun database
cp -R $(brew --prefix)/opt/lensfun/share/lensfun ~/bin/darktable-dev/share/
# Run darktable with dedicated cache and config directories (to leave your productive environment untouched)
GSETTINGS_SCHEMA_DIR=$(brew --prefix)/opt/glib/share/glib-2.0/schemas/ XDG_DATA_DIRS=$(brew --prefix)/share ~/bin/darktable-dev/bin/darktable --configdir ~/bin/darktable-dev/.config/darktable/ --cachedir ~/bin/darktable-dev/.cache/darktable/
As an amateur builder I am not seeing any more output or behaviour that worries me, or am I missing something? Thanks!
Great. What is required to compile darktable for m1 apple silicon instead of intel with rosetta?
@algreen05 see https://github.com/darktable-org/darktable/issues/7564
Danke. I read that thread and it appears they/you were unable to successfully compile darktable in Homebrew to run on Apple Silicon. Is there anything I/we can do further to make it work?
I would recommend to use the linked issue (https://github.com/darktable-org/darktable/issues/7564) to drive this further. All the details are there and some of the authors are also working with/on a homebrew-based build. I guess everyone is primarily interested in creating a working arm (m1) build, independent of what package manager is actually used.
Thanks for finding the solution for not detected libsoup, osm-gps-map and icu. I've added it to CI build in PR #10701. What about ISOBMFF support in exiv2 needed for CR3 support in darktable? Judging by the cmake output I see in CI builds homebrew doesn't ship exiv2 with that feature enabled...
Sorry, indeed it is missing. Will look into this and take care.
In addition downgrading lensfun to 0.3.2 is also still missing.
Regarding exiv2 with support of bmff:
To manually add it you need to modify /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/exiv2.rb
to include the changes as provided in https://github.com/darktable-org/darktable/blob/master/packaging/macosx/exiv2-BMFF.patch.
Specifically -DEXIV2_ENABLE_BMFF=ON
must be given as args within the install section.
You should then do a brew reinstall --build-from-source exiv2
to recompile it with the new option. Doublecheck via exiv2 --version --verbose --grep bmff
if your changes have been successful. If successful the output should be enable_bmff=1
.
Note that this change will be gone as soon as you update your homebrew (brew update
). With brew pin exiv2
you should be able to persist it.
To have bmff support activated by default I have created the following PR: https://github.com/Homebrew/homebrew-core/pull/92084.
Looks like progress on an M1 build has stalled. Is anyone still working on this?
If I am not mistaken, the path to all of the required libraries is set (for Apple) here:
In case of homebrew, the libraries reside in /opt/homebrew
(if you are on arm64/m1) or /usr/local
(if you are on i386/x86_64) and did not change the default installation path of homebrew. You can automatically get this path by running brew --prefix
or reading the environment variable HOMEBREW_PREFIX
.
It would be great if during configuration we could check if one is running homebrew. If so, CMAKE_PREFIX_PATH should be appended with the prefix of homebrew. The charm would be, that this would work independent of the architecture and even if you are running homebrew for both on the same machine. Homebrew will always return the right path depending on wether you are on arm64 or i386. In case it is not available, then one is probably running macports and it should go with the existing line to not mess with the currently working approach.
I am really no CMake expert, but a possible solution could look like:
if(DEFINED ENV{HOMEBREW_PREFIX})
message("-- Homebrew detected, setting CMAKE_PREFIX_PATH to $ENV{HOMEBREW_PREFIX}")
LIST(APPEND CMAKE_PREFIX_PATH $ENV{HOMEBREW_PREFIX})
else()
message("-- Homebrew not detected, setting CMAKE_PREFIX_PATH to prefer macports and/or user-installed libraries over system ones")
LIST(APPEND CMAKE_PREFIX_PATH /opt/local /usr/local)
endif()
What do you think? CC @parafin @MStraeten
With that change I was able to successfully build and run darktable on arm64 and i386. I had hoped that this would also omit the requirement to set the PKG_CONFIG_PATH
for libsoup@2 and icu4c manually (as described above), but it seems still to be required.
I think it’s fine. But see my previous comment: https://github.com/darktable-org/darktable/issues/10583#issuecomment-988603098
So I would rather see this as a pull request, together with maybe alternative BUILD.txt file based on homebrew instead of macports. And I think it makes sense to merge only after exiv2 issue is handled somehow. For M1 support there’s also lensfun issue as a blocker, and I don’t know the status of gtk-mac-integration there currently.
Thanks @parafin for your feedback!
I created PR https://github.com/darktable-org/darktable/pull/11130 with that changes. In addition it defines the Homebrew-specific path for llvm, icu4c and libsoup@2 which makes PR https://github.com/darktable-org/darktable/pull/10701 obsolete and brings any user closer in building darktable with Homebrew without setting any specific paths as originally suggested by me (see above).
Exiv2 and BMFF support is addressed. https://github.com/Homebrew/homebrew-core/pull/92084 was accepted and Homebrew now provides exiv2 with BMFF switched on by default.
Next task on the list is lensfun and support for its last stable release 0.3.2 in a Hombrew-based environment. If I understood correctly there is no way to get the by Homebrew currently only provided version 0.3.95 working due to https://github.com/lensfun/lensfun/issues/945 (darktable issue https://github.com/darktable-org/darktable/issues/2813)?
It works, but it breaks compatibility between lensfun versions, and so can and will break history stacks. Ideal solution would be a new lensfun release fixing this bug. More realistic is to stick with 0.3.2. The problem with 0.3.2 is that it doesn't build on M1 last time I checked, so some patch would be required. And I guess we're stuck with building it from the sources in this case. Also some users request newer lensfun due to new features it contain (new lens mathematical model or smth like that), so that's another downside with using stable lensfun release. It's a sad situation.
Forgive my ignorance, but isn't it better to have working build for M1 with newer version of lensfun than nothing? Arch based distributions have been using lensfun 0.3.95 for a long time as standard and no problem for me there. Actually I can convert and use adobe lcp profiles with this one and it's a nice benefit.
If you just use your build then you can do this in your own build. But darktable project has compatibility to old edits as an important design principle. As long as there’s no one contributing a conversion algorithm to upgrade old edits, it’s unlikely the new lensfun version will be supported.
@frgazor one principle of darktable is to assure that your image edits always look the same (no matter which version you used to do the edits). This currently cannot be assured when switching to lensfun 0.3.95, see https://github.com/darktable-org/darktable/issues/2813.
So if you start fresh with darktable and lensfun 0.3.95 you probably should be fine, as you have no edits done with an earlier version that is based on lensfun 0.3.2. Anyone else will not appreciate it.
There are basically two options:
yeah, like @apfelkraut said, it compiles just fine on M1/ARM. it's a bit of a pity that GitHub does not yet have M1 runners cuz then we could just build it there. also, i didn't have much time last months so no further progress from my side.
@frgazor so, i suppose the 1st option that @apfelkraut suggested may be the more promising one as it's probably sth you can do on your own and doesn't reqiure a deep understanding of lensfun.
This sounds promising. Will there, at some point, be a .dmg file anyone with an M1 can install to use Darktable on native apple silicon?
@algreen05 Maybe - I guess it comes down to finding the root cause why current DMG builds only work on some systems - see here https://github.com/darktable-org/darktable/issues/7564 for more context. I am kinda stuck there / lack time to dig deeper.
Well, thanks for all your work getting it this far. I will wait patiently until a reliable DMG build is available for M1. Am confident root causes and other issues will be discovered and resolved at some point this year.
I have tried three times to install Homebrew and compile/install Darktable and it never works. Is there a step-by-step guide for idiots like me on how to make this work?
not yet because there’s no maintainer for a homebrew based m1 build.
But @apfelkraut was able to compile it on his M1 mac. Didn't he have a process?
Building darktable on macOS using homebrew works for me without an issue. On Apple chips (arm) and on Intel chips (i386). Yet there is one significant limitation: it only works with lensfun 0.3.95 which is not backward compatible with any image that was edited based on lensfun 0.3.2 (all darktable releases so far). For details please see the discussion above.
A current step-by-step guide on how to compile darktable looks like this. Note this is not about making a DMG. You will need to run darktable from command line. Sources are residing in $HOME/src
and binaries in $HOME/bin
.
# 1. Get sources of darktable
mkdir ~/src
cd ~/src
git clone --recurse-submodules https://github.com/darktable-org/darktable.git
cd darktable
git submodule init
git submodule update
# 2. Make sure that homebrew is up-to-date
brew update
brew upgrade
# 3. Install dependencies via homebrew
brew install adwaita-icon-theme cmake desktop-file-utils exiv2 gettext git glib gmic gphoto2 graphicsmagick gtk+3 gtk-mac-integration icu4c intltool iso-codes json-glib lensfun libavif libheif librsvg libsecret libsoup@2 little-cms2 lua libomp ninja openexr openjpeg osm-gps-map portmidi pugixml sdl2 cmocka curl perl jpeg llvm po4a
# 4. Build darktable
~/src/darktable/build.sh --install --build-type RelWithDebInfo --prefix ~/bin/darktable-dev
# 5. Update and copy lensfun database (only required as we are kind of misusing darktable with lensfun 0.3.95)
lensfun-update-data
cp -R $(brew --prefix)/opt/lensfun/share/lensfun ~/bin/darktable-dev/share/
# Run darktable with dedicated cache and config directories (to leave your productive environment untouched)
GSETTINGS_SCHEMA_DIR=$(brew --prefix)/opt/glib/share/glib-2.0/schemas/ XDG_DATA_DIRS=$(brew --prefix)/share ~/bin/darktable-dev/bin/darktable --configdir ~/bin/darktable-dev/.config/darktable/ --cachedir ~/bin/darktable-dev/.cache/darktable/
@algreen05, you are welcome to share your error message(s) on why it never works. It could be easier for someone to help you ...
This works wonderfully. I had no error messages and figured out how to create a bash script file and move it to my dock. The speed of the M1 native app is fantastic compared to Rosetta.
Thank you!
To get lensfun 0.3.3 via homebrew I have just created a pull request: https://github.com/Homebrew/homebrew-core/pull/96709
By default homebrew now delivers lensfun 0.3.3 which is to my understanding backward compatible to the last stable release 0.3.2 and builds nicely also on M1.
(Just to note in case somebody still wants to continue working with the latest master branch of lensfun, so the current descendant of 0.3.95, you can do so with brew install --HEAD lensfun
).
Thanks for your great work, I was able to build and run darktable 3.9.0 natively with lensfun 0.3.3 from homebrew on M1 Max.
Hello, thank you for your work @apfelkraut
I managed to build, install and run the dev version on my M1 with the latest Monterey release. I'm gonna test perf differences between the native and emulated version.
Thanks again ! :)
Thanks a lot, @apfelkraut! On my M1 iMac this native build is much, much faster than the Rosetta emulation — though not when I use my default configuration. Some Terminal output seemed to predict trouble, but in fact everything runs smoothly:
(darktable:11461): Gtk-CRITICAL **: 14:22:22.704: gtk_window_add_accel_group: assertion 'GTK_IS_WINDOW (window)' failed
[dt_detect_cpu_features] Not implemented for this architecture.
[dt_detect_cpu_features] Please contribute a patch.
[dt_init] SSE2 instruction set is unavailable.
[dt_init] expect a LOT of functionality to be broken. you have been warned.
@apfelkraut Your instructions worked for me too! I'm going to work on copying over my working darktable data directories and work on it with this new version, and see how it goes.
Happy to hear!
@RhvdG the first line "Gtk-CRITICAL ..." is also shown when starting the official darktable package for Mac from command line ... to what I understood it should not worry us as darktable works as expected.
The other 4 lines are due to the different chip architecture ... e.g. SSE2 is an Intel specific instruction set. Naturally it is not available on non-Intel cpus.
On my M1 iMac this native build is much, much faster than the Rosetta emulation — though not when I use my default configuration
@RhvdG Could you please elaborate how you experience different speeds with different "configurations"? What configurations are you referring to?
I built darktable on my M1 Mac Mini using the fantastic instructions provided in this thread - but to be honest, I can't really detect any difference in performance vs. the official release. Are there any particular use cases that people can share that run faster with the ARM build?
@RhvdG Could you please elaborate how you experience different speeds with different "configurations"? What configurations are you referring to?
The final line in @apfelkraut's instructions runs darktable ‘with dedicated cache and config directories’. If I do that, the application is much faster. If instead I run it with the config directories I used before, with the Intel version, the speed gain is negligible. Does that answer your question? If not, let me know.
@RhvdG Could you please elaborate how you experience different speeds with different "configurations"? What configurations are you referring to?
The final line in @apfelkraut's instructions runs darktable ‘with dedicated cache and config directories’. If I do that, the application is much faster. If instead I run it with the config directories I used before, with the Intel version, the speed gain is negligible. Does that answer your question? If not, let me know.
Thanks for this observation. I can confirm this as well. Also, the Diffuse and Sharpen module in Darktable consumes significant processing power on my Macbook Air M1. If I use just one instance it results in image load times of 10+ seconds. If I do not use it, images load instantly.
run darktable ‘with dedicated cache and config directories’.
Interesting - that's how I've been running Darktable M1 build too, as per the instructions. Much faster for you, as you've said, but no faster for me when looking at general startup, image loads, basic edits incl. haze, crop, local contrast etc. Curious.
Presumably running without the configdir
and cachedir
parameters would run darktable normally - viz GSETTINGS_SCHEMA_DIR=$(brew --prefix)/opt/glib/share/glib-2.0/schemas/ XDG_DATA_DIRS=$(brew --prefix)/share ~/bin/darktable-dev/bin/darktable
- but that gives me a schema upgrade warning UI popup, which I hurriedly cancelled.
If I use just one instance vs. do not use it
@algreen05 Not sure what this means? Is this some parameter controlling cores allocated or something? I'd love to prove the M1 build is faster at something!
I mean one instance of Diffuse and Sharpen vs none. That module is processor intensive and slows down image work on my machine.
$ brew update && brew upgrade
$ brew install adwaita-icon-theme cmake desktop-file-utils exiv2 gettext git glib gmic gphoto2 graphicsmagick gtk+3 gtk-mac-integration icu4c intltool iso-codes json-glib lensfun libavif libheif librsvg libsecret libsoup@2 little-cms2 lua libomp ninja openexr openjpeg osm-gps-map portmidi pugixml sdl2 cmocka curl perl jpeg llvm po4a
ok
$ mkdir ~/src
$ cd ~/src
$ git clone --recurse-submodules https://github.com/darktable-org/darktable.git
ok
$ cd darktable
$ ./build.sh --install --build-type RelWithDebInfo --prefix ~/bin/darktable-dev
..
-- Checking for -std=c++14 support - works
CMake Error at cmake/modules/LibFindMacros.cmake:272 (message):
REQUIRED PACKAGE NOT FOUND
We could not find development headers for Glib. Do you have the necessary
dev package installed? This package is REQUIRED and you need to install it
or adjust CMake configuration in order to continue building darktable.
Relevant CMake configuration variables:
Glib_INCLUDE_DIR=<not found>
GlibConfig_INCLUDE_DIR=<not found>
Glib_LIBRARY=<not found>
You may use CMake GUI, cmake -D or ccmake to modify the values. Delete
CMakeCache.txt to discard all values and force full re-detection if
necessary.
Call Stack (most recent call first):
cmake/modules/FindGlib.cmake:40 (libfind_process)
src/CMakeLists.txt:296 (find_package)
How to change this configuration in cmake, nobody knows? CMakeError.log
@estaf1001 I assume that you are on recent state of the master branch? Could you try to re-run brew install glib
? What is the output?
Normally the build process should automatically find the dependencies that were installed via homebrew. This doesn't seem to work in your case. I can imagine that something went wrong with the second command (the very long string brew install ...
). So maybe you want to retry it? Watch out for any error message ...
I assume that you are on recent state of the master branch? Could you try to re-run brew install glib? What is the output?
% brew install glib
Running `brew update --preinstall`...
Warning: glib 2.72.1 is already installed, it's just not linked.
To link this version, run:
brew link glib
iron@estaf darktable %
The installation of each package go through. Yes, I will try it now.
This is primarily a request for help to find an easy and working way for newbies to build a full grown darktable from sources on macOS by providing dependencies via homebrew. If this is not considered off-topic at this place, I would propose to close it as soon as everything has been clarified.
For reasons of simplicity, I would like to limit this at the beginning to Intel-based (i386) Macs in contrast to M1-based (arm) Macs. Note that there is a dedicated issue to make this work for M1: https://github.com/darktable-org/darktable/issues/7564
In my case I am doing this on macOS 12.0.1 (Monterey).
When building for this platform you are basically left with two options:
The macOS community of darktable has put a tremendous effort in making it built and providing a very well-working DMG image based on macports. Thanks a lot, really highly appreciated!
Still, as newbie I am slightly overwhelmed by the number of steps to take and all the diffs that must be applied. In addition if you are used to a different package manager than macports, you normally do not want to switch (back).
I recently discovered that there is also the opportunity to build on basis of homebrew which I want to detail out subsequently.
Assuming that you have homebrew up and running, you first want to doublecheck if you have the latest packages installed:
You then need to install all the dependencies of darktable:
You then go to the root of the cloned darktable repository and build it. In my case I decided to put the binaries into ~/bin/darktable-dev:
If the built completed without any issue - like in my case - the final step is to run it. As I do not want to mess with my stable/productive darktable environment, I am passing dedicated/separate config and cache directories which are residing within the same directory:
Great! That was easy. I just did some basic tests and it seems to work.
Still I discovered some things where I am not sure how to take them or what they mean in the end = do I need them for the perfect darktable experience?
When reviewing the build log, I see the following messages:
That is strange. Beside Colord, homebrew includes these packages. Why have they not been found? I suspect that I need to somehow pass the correct library location to the build procedure. How do I do that?
When running
darktable --version
I also realize the following:Again, strange. Homebrew also offers imagemagick. Same question as above.