[Meta-bug][WONTFIX] Please resolve OSX packaging situation

[LebedevRI]

It is my current understanding that currently, darktable releases for OSX are being built by @parafin. It is my current understanding that the version of XCode used to do so is stuck at 12.4 (LLVM10-based), and it can not be upgraded further due to the fact that newer OSX versions do not support the particular hardware used to do these builds.

Although i have not personally had the need to use those builds, i would like to thank @parafin for contributing these builds for all that time!

That being said, this situation is an evolutionary deadlock. It, both, forces a large (larger than strictly necessary) set of compilers to be supported, and provides no way forward.

Each year, a new major GCC version is release, and two major releases of LLVM happen. If someone is serious about CI, that means they all need to be tested in CI. In rawspeed, that currently already amounts to a pretty large matrix: https://github.com/darktable-org/rawspeed/actions/runs/4069778116 ... and i'm not even covering a half of the configurations i want to cover. At some point, there will be more CI jobs than allowed by github actions.

There is no way forward here, because, obviously, e.g. newer C++ standard versions are not implemented in older versions of compilers, and if we can't change the minimal required version, then we are stuck with what we've got.

cc @parafin @MStraeten @johnny-bit

[parafin]

Several months ago I've outlined the way forward as I see it: https://github.com/darktable-org/darktable/pull/11579#issuecomment-1140500390

Someone has to take responsibility and basically take over my duty as macOS maintainer.

[johnny-bit]

In terms of linux compiller support I'd be for dropping old Ubuntu LTS in favour of keeping int only the most recent one. Ubuntu LTS has 2 year release cycle, which slows us down in terms of compillers, but not as much as keeping with supported LTS since LTS support does overlap. On top of that all distros package their own releases so we shouldn't be bogged down by that anyway.

My recommendation: switch to newest LTS as soon as it's available in CI and drop old ones immediately. That'd free up CI and dev time.

czw., 2 lut 2023 o 18:06 parafin @.***> napisał(a):

Several months ago I've outlined the way forward as I see it: #11579 (comment) https://github.com/darktable-org/darktable/pull/11579#issuecomment-1140500390

Someone has to take responsibility and basically take over my duty as macOS maintainer.

— Reply to this email directly, view it on GitHub https://github.com/darktable-org/darktable/issues/13509#issuecomment-1414077340, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRRKFNY57LQAOEBOVPQRD3WVPSQ5ANCNFSM6AAAAAAUPIW7XQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Pozdrawiam, Hubert Kowalski

[victoryforce]

My recommendation: switch to newest LTS as soon as it's available in CI and drop old ones immediately. That'd free up CI and dev time.

Totally agree, I'd keep the oldest LTS for nightly only, I plan to add AppImage generation in nightly (and an AppImage is most useful when generated on as old a distro as possible).

[LebedevRI]

In terms of linux compiller support I'd be for dropping old Ubuntu LTS in favour of keeping int only the most recent one.

i think that should say: support debian stable and the most recent Ubuntu LTS. This will help with trimming the GCC versions, but not LLVM / XCode ones.

[johnny-bit]

I 100% agree with Roman. For XCode as parafin said peeps need to step up and be maintainer otherwise we're stuck. And Victor - having oldest LTS for nightly block as the same way as having it in CI anyway and is even worse because if CI won't fail but nightly will it'll create problems :) Let's assume that anybody who tries nightly dt also has enough tech savvyness to have latest LTS :)

czw., 2 lut 2023 o 19:56 Roman Lebedev @.***> napisał(a):

In terms of linux compiller support I'd be for dropping old Ubuntu LTS in favour of keeping int only the most recent one.

i think that should say: support debian stable and the most recent Ubuntu LTS. This will help with trimming the GCC versions, but not LLVM / XCode ones.

— Reply to this email directly, view it on GitHub https://github.com/darktable-org/darktable/issues/13509#issuecomment-1414215028, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRRKFLHIUJPXRG4VTCMXQLWVP7OTANCNFSM6AAAAAAUPIW7XQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Pozdrawiam, Hubert Kowalski

[LebedevRI]

For XCode as parafin said peeps need to step up and be maintainer otherwise we're stuck.

I mean, we may not have a choice here in the end. The situation is not dire yet, but unless this is resolved by, say, next major release or so...

There's of course the one more alternative, which is probably impossible in nowadays world: simply crowdfund a new hardware for @parafin.

[parafin]

That's a kind suggestion, but it's actually more about the lack of time and motivation than any other resources (I am able to find the necessary hardware at work to do the builds like I did with ARM64).

[LebedevRI]

I see, then the situation is much simpler.

[LebedevRI]

@patdavid @TurboGit @johnny-bit @MStraeten i think we need to make this situation more visible, I think it would be useful to come up with a message, post it on discuss and perhaps propagate it through social media.

I'm not aware of any way of having shared draft posts on disscuss, so would anyone (@patdavid hint hint) like to start and give a link?

[TurboGit]

@LebedevRI : Yes, so I propose to post this on pixls.us (I'm not on social media, so people can post a similar message):

Because of lack of resource on MacOS we will be obliged to discontinue the darktable port after 4.2.1. Fact is that on MacOS we are currently stuck with XCode 12.4 (LLVM10-based) and this is blocking progress on some part of the code base.

As outlined already we have a single MacOS maintainer at the moment (parafin). This is not a good situation and the lack of time to move forward the current MacOS build can no longer be responsible of blocking progress on the current code base.

If you are interested to help, please step forward. We really need a long term dedication to solve this. An Open Source software can only offer what the community can work on.

Is the message clear enough? Should some other things be outlined?

[patdavid]

I'm not aware of any way of having shared draft posts on disscuss,

I don't think it's possible at the moment. You can start a PM to multiple people and I think you can make the first message a wiki possibly?

[LebedevRI]

I've tried to edit that. @TurboGit How about this:

Dear users!

As everybody knows, darktable is a community-driven project.
One particular area that is always everyone's not-first choice
is maintenance. Especially, maintenance of the darktable as
a whole, on a particular platform.

As it happens, @parafin has been a person solely responsible
for maintaining, and packaging, darktable on OS X
for the last 12 years.  <--- FIXME
We would like to thank him for all of his efforts!  <--- FIXME

Everything has it's limit, and presently, it has been indicated
that @parafin wants to end that tenure, and pass on the mantle.

At the same time, there is a big roadblock on the OS X side:
currently, as requested by @parafin, the minimal required 
XCode version is XCode 12.4 (LLVM10-based),
and with LLVM16 about to be released in ~April,
that puts us to 7 (*sic*) LLVM versions to support,
in addition to currently supporting three GCC releases.

Not only is this support matrix unsustainable,
not having a path forward makes it impossible
to someday make use of the compiler (and library)
features introduced in later compiler versions.

In summary, unless someone steps forwards and commits
to the role of OSX maintainer, we will be forced to
fully and completely stop supporting OS X,
after the next minor release (4.2.1).

If you are interested to help, please step forward.
We really need a long term dedication to solve this.
An Open Source software can only offer what
the community can work on.

Roman, Pascal.

By stopping supporting it, i mean intentionally breaking it.

[TurboGit]

Thanks Roman, better than my proposal. If @parafin confirms that 12 years is correct I can post this on pixls.us.

@LebedevRI : I propose that both of us "sign" this message :

If you are interested to help, please step forward. We really need a long term dedication to solve this. An Open Source software can only offer what the community can work on.

Roman, Pascal.

Ok?

[LebedevRI]

Thanks Roman, better than my proposal. If @parafin confirms that 12 years is correct I can post this on pixls.us.

I guess it can be rounded to 10 years, since release-1.1rc1 or so.

@LebedevRI : I propose that both of us "sign" this message :

If you are interested to help, please step forward. We really need a long term dedication to solve this. An Open Source software can only offer what the community can work on. Roman, Pascal.

Ok?

Right.

[parafin]

Thanks Roman, better than my proposal. If @parafin confirms that 12 years is correct I can post this on pixls.us.

I guess it can be rounded to 10 years, since release-1.1rc1 or so.

Yes, 10 years sounds right: https://www.darktable.org/2012/08/bringing-current-darktable-to-os-x/

[zisoft]

Coming from Lightroom, On1, Capture One I finally found darktable. Now I'm completely shocked to read this.

Still being relatively new here (only contributed with a few small PRs so far) and I really want to see MacOS to be continued.

I can build darktable on my Mac using homebrew but this is of course by far not sufficient. So I think it can be helpful to describe what work needs to be done, what skills are required and so on.

[LebedevRI]

@TurboGit speaking of 4.2.1, i would like some headsup please (and an ETA). I'm hoping to soon update rs submodule in master, in a way that can be cherry-picked into 4.2.x

[TurboGit]

@LebedevRI : I sent a message to you, @kmilos was in copy haven't you received it?

Anyway the 4.2.1 is plan for mid-February and it would be nice to have OM-1 as we are ready on dt to fully support it. Of course others camera are welcomed.

[TurboGit]

@LebedevRI @parafin : I'm posting the message to pixls.us now.

[LebedevRI]

@LebedevRI : I sent a message to you, @kmilos was in copy haven't you received it?

I'm currently in stress-avoidance mode and i have not checked mail (or matrix/irc, or github notifications) in ~2 weeks.

[MStraeten]

at least it’s possible to build a x86_64 package with deployment target 10.14 in a macOS 13.2 virtual machine using recent Xcode 14.2 commandline tools and sdk 13.1 on an M1. I published a build on pixels.us - now waiting for feedback if it runs fine on older macOS versions (it at leat runs on intel macs with macOS 11.7 and 12.6)

[LebedevRI]

The point is that either someone actually commits to the role of maintainer of those instructions / infra / etc, or they are dead code and can/will be removed.

[MStraeten]

no need to remove build instructions that works fine …

[LebedevRI]

~Overriding~ Overruling such removal will be up to the maintainer for that distro.

[parafin]

I don't have an account on pixls.us (and don't want to create one just for this discussion), so I suggest to direct interested people to GitHub (e.g. to this ticket), especially since having a GitHub account would be a requirement for a maintainer anyway. And by "interested" I don't mean someone, who just wants to discuss the situation, but who is serious about doing this job. Some valid questions were raised on pixls.us, so I'm going to answer some of them here.

Current state of macOS port It's in pretty good state, macports build instructions are being maintained by me and are relatively simple. I've made an effort to remove any patching of dependencies, only exception is gtk-mac-bundler, which is required only for building the app bundle, not for compilation of darktable executable. Probably it would be enough to just update it (quite old version is being used ATM), but it needs to be carefully done to ensure that nothing breaks as a result. There are some outstanding macOS-specific bugs, all of them, I think, are GTK-related, e.g. main one is #12727, which was already fixed upstream, but macports yet to propagate that update. Another problem is mentioned in that same bug report at the end, color profile-related. Speaking of color profiles - currently darktable on macOS is limited to sRGB color profile for display, due to limitations in cairo (no way to set any other color profile for output), but it's color managed, as is everything on macOS, by OS itself. Another issue with macOS port is how fullscreen mode behaves - modal dialogs are often placed below the main window, so they are not visible. Maybe even some menus do that. This needs to be investigated and fixed somehow. Again it's GTK-related issue.

Future of macOS port (as seen by me) It would be nice to move away from manual building of macOS packages to CI-based one (#10360). darktable's CI is based on GitHub Actions infrastructure, and their macOS machines provide only homebrew as default package management software. So it makes sense to migrate from macports to homebrew, and some effort has already been done (e.g. there are packaging instructions in the repo and CI compiles code on macOS). But I'm not sure if homebrew build is feature-complete compared to macports, plus CI-built executables don't run (#11934), at least inside CI, so at the very least it's impossible to run tests, or they are completely unusable (which would be strange, since manually-built ones do work). As was mentioned on pixls.us, Apple has deprecated OpenCL support in favour of their vendor-locked Metal API. OpenCL drivers haven't been removed yet, but it's probably just a matter of time (which still could be years). Maybe someone will create compatibility layer on top of Metal to emulate OpenCL, like it was done with Vulkan. But I don't know if some work is being done in that direction, so it's unclear whether it will happen and be usable. I'm not sure if darktable without GPU acceleration is a "viable product", so that is something to keep in mind. Potential maintainer has to be someone, who still wants to work on macOS port despite these concerns (so someone optimistic;) ). BTW, there's very old Apple bug with OpenCL, where kernel compilation fails on second run (due to include path not working or smth like that), so when darktable updates, user needs to restart it several times before OpenCL starts working. Maybe it could be worked around in some way.

What is expected from macOS maintainer? It would really help if maintainer uses darktable on macOS (to notice when things are obviously broken) and does/have done some C/C++ (or better yet GTK3) development on macOS already (it's a special type of masochism to be honest, so I wouldn't suggest unprepared people to jump into that). Also it would be nice to have Apple Developer subscription (which is not free) to be able to sign and notarize macOS packages. But I don't see it as a requirement, I might help with that part if builds will be done by CI, or we could ship unsigned builds, since it's still possible to install them, at least for now. As for the work - main responsibility of the maintainer is to deal with breakage caused by dependencies and OS updates. As an example see already mentioned #12727. Also darktable sometimes adds new dependencies or updates minimal required versions, this also needs to be handled (e.g. build instructions and CI configs updated accordingly). Next one is to handle macOS-specific bugs, so monitor the GitHub tracker, try to reproduce the issues, fix them or help users to understand what's wrong if it's not a bug;) And of course maintainer needs to handle release process of macOS packages when it's that time of the year.

[TurboGit]

@parafin : I've linked this comment on pixls.us.

[parafin]

Thanks. One final thought from me about motivation. Alongside with two other required resources - access to Apple hardware and free time - motivation is important in the long run. For me, at least at first, it was the desire to use darktable on my MacBook by myself. It's probably the only viable motivation for open source project - to build smth for yourself, but of course one has to decide for him/herself what drives him/her.

[rgo]

@parafin thanks for the explanation. I'm going to study all the points that you exposed to confirm if I'll be capable of taking the baton.

And thanks for all your time invested all these years :)

[TiredOfGuessing]

Could someone spell out the GCC/LLVM problem? I get the basic idea about new tools with new capabilities not supported by older versions. Also that code may need to be tweaked to work with those new tools, and potential conflicts with the various tool versions. I am a darktable newbie ~1yr, so basically everything here is new to me, including macOS API and dev env.

So is the problem...

1. updating code for new tools?
2. updating build for new tools?

Also...

1. what is support goal for (multiple versions?) of tools?
2. what is support goal for multiple versions of macOS? I'm guessing this is something @MStraeten understands and is a function of Apple's API deprecation?

Basically everything here is new to me. Summary of my darktable background, concerns. https://discuss.pixls.us/t/darktable-for-macos-needs-you/35142/12

[parafin]

There are 2 variables that limits available compiler features:

1. XCode version (and so corresponding included LLVM version) - running new enough XCode requires new enough macOS, which in turn requires new enough Apple hardware. Also there is a question of supporting both Intel and ARM64 Apple hardware platforms, though it may be enough to have access to just ARM64.
2. Deployment target (minimal supported macOS version by resulting executables) - here it's a question of how many macOS versions do we want to support. Right now I build for macOS 10.14+. Moving package building to CI will most likely cause this number to bump. Which is IMHO fine, and should be defined by maintainer.

[parafin]

And there is no problem as such, it all works as far as I know, so it's only about how we do releases.

[LebedevRI]

@TiredOfGuessing the problem is the exact opposite. The issue isn't with supporting newer compilers, because if you don't do that, you might as well pack up shop. The issue is with having to support older compilers.

[zisoft]

There is no official statement from Apple, but inofficially only the last three versions of MacOS are supported:

MacOS end of life dates

So maybe darktable should as well commit to only these versions?

[MStraeten]

macOS 10.14 was the last version that supported 32bit apps, so there might be users around that didn’t step up to be able to run software that never was ported to 64Bit. As long as macports libraries can be built with deployment target 10.14 there’s no need to ditch that since recent Xcode toolchain supports builds with that deployment target.

[parafin]

It’s not this simple. For example std::filesystem support is available on macOS starting with 10.15. So building code utilising that C++ STL feature is only possible with deployment target 10.15 or higher. I’m not sure if this particular problem is actual for rawspeed, but there might be others.

[LebedevRI]

As long as macports libraries can be built with deployment target 10.14 there’s no need to ditch that

There's a subtle issue there. Just because you can build something for some other/older target, doesn't mean it will actually run on that target. See e.g.:

• https://github.com/darktable-org/darktable/issues/10087
• https://github.com/darktable-org/rawspeed/pull/317
• https://github.com/darktable-org/rawspeed/commit/52861607640f4eebd5a95e763dffed160883fced

I'm not sure if darktable builds already backdeply more recent libc++ than otherwise would be available on those older targets, but even if they do, i'm not sure that covers everything.

[zisoft]

Since we are discussing about stopping darktable for MacOS at all, I would prefer to only drop support for a few users which are running ancient MacOS...

[parafin]

My MacBook is quite old, so without hacks it’s only possible to run macOS 10.15, not higher, which is already unsupported by Apple. But I personally could live with it, if darktable dropped support for macOS 10.15, as long as I no longer have to do maintainer duties.

[MStraeten]

just drop support if it’s necessary. If the darktable codebase requires features no longer available with older macOS version then it’s time to drop that support. Macs can be run for longer than a decade …

[LebedevRI]

Again, the disscussion here is that unless someone maintains it, the support can be DeadCodeElemination'ed. But if someone does step forward as a new maintainer, then of course it's no longer dead code, and it will be up to them to decide what OSX versions / XCode versions are to be supported.

Next comment about that will be deleted as offtopic.

[zisoft]

There are already a few people who mentioned that they are interested. I think the main problem is that @parafin's skills and experiences have grown over 10 years and nobody can expect that a new maintainer can fully support from the beginning.

As @TurboGit mentioned on pixls.us:

Hopefully you’ll not be alone, I do really hope that we’ll find at least three people to team together to make the MacOS situation better.

Maybe someone can clarify how a maintainer change can proceed over time?

[kichererbse23]

Hopefully you’ll not be alone, I do really hope that we’ll find at least three people to team together to make the MacOS situation better.

As my current programming skills would not be sufficient to do the full job, I would definitely be in for supporting a new maintainer as much as I can.

[parafin]

As @TurboGit mentioned on pixls.us:

Hopefully you’ll not be alone, I do really hope that we’ll find at least three people to team together to make the MacOS situation better.

Maybe someone can clarify how a maintainer change can proceed over time?

There’s going to be 4.2.1 release soon, which I will take care of, and then in about half a year a new major release (4.4?). This half a year is a transition period, so that macOS package for new release should be built already by new maintainer. I’m here to answer questions and help otherwise during that time (and probably later), but the point is to shift responsibility away from me to the new maintainer or maintenance team.

As for splitting responsibilities, I see several potential roles:

1. release manager - takes care of build instructions or CI scripts and manages the release process (and optionally nightlies) for macOS package
2. tester - runs git master (and potentially some PR) darktable builds on macOS and reports any found bugs/issues
3. support - checks macOS-specific bug reports in GitHub tracker, tries to reproduce the issues and fix them (if possible), answers macOS-specific questions about how darktable behaves on that platform (e.g. how crash reports work)

Feel free to think of any other structure you want, since it’s not me who is supposed to manage it in the end. I think I’ve given enough mine thoughts here on how it all could work, I expect now more specific questions after some self-investigation of the work involved by interested parties.

[kichererbse23]

tester - runs git master (and potentially some PR) darktable builds on macOS and reports any found bugs/issues

this is definitely what I could do.

[kichererbse23]

Any progress?

If @parafin s proposal is fine with the new maintainer or the new team I would happily join for points 2 and partly 3 (except for fixing if this involves coding).

[trash-anger]

I guys, I just heard the news. I'm not a contributor but I use darktable on osx and I could help to build release automations.

I'm more coming from the world of web development but we could probably try to automate parts of the tests.

I'll fork the repo Sunday and see from where we can start on the point 1.

[leanderhutton]

I finally got a Github account! I can help with 2 and 3 and I have access to shiny new and old and busted Mac hardware to test on. Not sure I'd trust myself with number 1 yet but maybe in time as I learn more. Will track the Mac issues here and see if I can help!

Edit: now that it's (easier) to virtualize MacOS than in the past it should be easier to do compartmentalize this stuff.

[kichererbse23]

@trash-anger, @leanderhutton Great. I will also go for points 2 and 3. let us try to work in team. Perhaps we can manage to tackle point 1 together, too.

[leanderhutton]

@kichererbse23 cool deal! Getting macOS VMs set up on my new MBP to make a homebrew build environment for this, I'm mostly a Linux dude these days and old school so I've been using ports on my base machine. Starting to look into the build process and what not.

There are several Mac issues for older versions of darktable still floating around, do we need to tackle those or just look at stuff for 4.2+?

[LebedevRI]

There are several Mac issues for older versions of darktable still floating around, do we need to tackle those or just look at stuff for 4.2+?

It is my understanding that only the last stable release series is supported.