Anything in here that could be of use on Linux?

[RJVB]

AFAIU, Apple/Clang blocks are standard C structures behind the scenes, and support for them on Linux only relies on building with clang, not on ObjC (but possibly also on libBlocksRuntime provided by libdispatch).

It would be really nice to have some kind of interface to convert C++ closures and/or regular function pointers to blocks in order to make libdispatch more generally usable regardless of the compiler of choice. (Or even to use GCC on Mac now that it comes with libc++ support.)

[gershnik]

A really interesting question, thank you! I wasn't aware that libdispatch is available on Linux and will have to try it there to provide a definite answer.

Meanwhile theoretically the answer is 'probably yes'. 🙂

Even on Mac BlockUtil.h and CoDispatch.h facilities are also available for pure C++ code that doesn't rely on ObjC runtime. (See also this for some caveats to be aware of if mixing both types of code in the same project.)

You are correct that blocks are indeed just a bunch of C structs with compiler's special knowledge of them (you can see the details here). In absence of ObjC runtime there are some library calls like Block_copy that need to be available too.

I am going to check if things indeed work on Linux but it will take some time. If you already have a Linux project that uses libdispatch, please, give it a try - I'd be very curious to know the results!

Thank you!

[RJVB]

On Tuesday August 06 2024 05:53:11 Eugene Gershnik wrote:

If you already have a Linux project that uses libdispatch, please, give it a try - I'd be very curious to know the results!

libdispatch is in the DebUbuntu repos, and I see (only) forked-daapd as a dependency, but a locate libdispatch also gave me hits in GNUStep-related directories, as well as in my Swift installation. It seems safe to assume that all those depend on being built with clang. I have a forked & adapted MacPorts package manager running and in the past this also required blocks support.

FWIW I'd be fine with an interface library that itself needs to be built with clang. Or maybe a libdispatchmm aka GCD++ binding library ;)

[gershnik]

Well, after some minor portability fixes (see commit https://github.com/gershnik/objc-helpers/commit/08cfbd0180cee330e06c1c52f703e75022387046) it seems to work just fine on Linux. I used libdispatch built from sources at https://github.com/apple/swift-corelibs-libdispatch. The clang command line needs to include

--std=c++20 -fblocks -ldispatch -lBlocksRuntime

I'll add some real tests and documentation for the next release which will take some time but in the meanwhile code from master branch should work fine.

[RJVB]

Thanks (and you're welcome for giving you the idea ;) ) !

On Tuesday August 06 2024 13:33:25 Eugene Gershnik wrote:

The clang command line needs to include --std=c++20 -fblocks -ldispatch -lBlocksRuntime

Is the c++20 bit true on Mac too? I recently learned that this version of the standard is only supported reliably in clang since v17. Not just because of libc++ but also because of actual miscompilations by the compiler. Probably true on Linux too, btw.

Oh, and you're leaving the implementation of a glue library callable from non-clang compilers as an exercise for the reader? ;)

[gershnik]

and you're welcome for giving you the idea ;)

Thank you!

Is the c++20 bit true on Mac too?

Yes. Technically BlockUtil.h requires __cpp_concepts to be defined and CoDispatch.h __cpp_impl_coroutine and __cpp_lib_coroutine. These features require C++20 AFAIK. Not sure at this point since when Xcode started to support C++20 but it's been a while.

Making any of this work with GCC is a whole another story and a much bigger project (if it is feasible at all). GCC lacks support for blocks and I don't even know if libdispatch itself is usable from it. It certainly requires clang to compile itself. I'll look into it but this will probably take quite some time.

[RJVB]

On Tuesday August 06 2024 14:31:29 Eugene Gershnik wrote:

Making any of this work with GCC is a whole another story and a much bigger project (if it is feasible at all). GCC lacks support for blocks

Hence the idea for an interface library which indeed would be built with clang itself. If your code can turn a C++ callable expression into a block, but maybe it can only do that when it has some kind of introspective access to the code via templates and what not?

(I'm one of those "learnt it on the job" programmers who also stuck to plain C for a very long time; lots of the features added in C++11 and onwards are a higher order of black magic to me than the Obfuscated C jokes from 40 years ago ;) )

[gershnik]

I see what you are saying and it could almost work, but the problem with this approach is that the shim pre-built library would need to receive a concrete type to turn into a block inside - you cannot use compile-time template 'magic' for this. Currently, the way BlockHelper is written, makeBlock returns something that is templated on the callable it wraps. It might be possible to 'erase' the callable type, likely at a cost of a virtual function call or something equivalent but it gets unpleasant very quick.

Even if this were possible such a shim would belong to a different library. This one is header-only and meant to be lightweight and simple to drop-in. Adding a component that needs to be separately built potentially with a different compiler would totally defeat this.

[gershnik]

Apropos it seems that there is active work being done right now to add blocks support to GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78352 This issue has been open since 2016 but perhaps it will finally be fixed in 2024 😂

[RJVB]

On Tuesday August 06 2024 18:48:02 Eugene Gershnik wrote:

I see what you are saying and it could almost work, but the problem with this approach is that the shim pre-built library would need to receive a concrete type to turn into a block inside - you cannot use compile-time template 'magic' for this. Currently, the way BlockHelper is written, makeBlock returns something that is templated on the callable it wraps. It might be possible to 'erase' the callable type, likely at a cost

That's what I feared.

Maybe the underlying question should be whether (the most common functions in) libdispatch require the full potential of the blocks mechanism. I just implemented a simplified equivalent to Qt's QMetaCall::invokeMethod() variant that takes a lambda:

template <typename F>
struct FunctorEvent : public QEvent
{
public:
    using Fun = typename std::decay<F>::type;
    Fun fun;
    FunctorEvent(Fun && fun, QSemaphore *sem = nullptr)
        : QEvent(QFUNCTOR_EVENT)
        , fun(std::move(fun))
        , semaphore(sem)
    {}
    FunctorEvent(const Fun & fun, QSemaphore *sem = nullptr)
        : QEvent(QFUNCTOR_EVENT)
        , fun(fun)
        , semaphore(sem)
    {}
    ~FunctorEvent()
    {
        fun();
        if (semaphore) {
#ifdef NEEDS_INVOKE_METHOD_FUNCTOR
            semaphore->release();
#else
            qWarning() << Q_FUNC_INFO << name() << "called but support for using the semaphore wasn't compiled in!";
#endif
        }
#ifdef DEBUG
        qDebug() << Q_FUNC_INFO << this << name() << "done on the main thread";
#endif
    }
    QString name() const
    {
        const auto mangled = typeid(fun).name();
        const auto demangled = abi::__cxa_demangle(mangled, nullptr, nullptr, nullptr);
        if (demangled) {
            const QString ret = QString::fromUtf8(demangled);
            free(demangled);
            return ret;
        }
        return QString::fromUtf8(mangled);
    }
private:
    QSemaphore *semaphore;
};

class PVCApplication : public QApplication
{
public:
    template <typename F>
    void invokeMethod(F && fun)
    {
        auto name = typeid(fun).name();
        const auto demangled = abi::__cxa_demangle(name, nullptr, nullptr, nullptr);
        if (demangled) {
            name = demangled;
        }
        qDebug() << "\n" << Q_FUNC_INFO;
        if (QThread::currentThread() == mainThread) {
            qWarning() << name << "being called on the main thread";
            fun();
        } else {
            QSemaphore sem;
            qDebug() << "invoking" << name << "on the main thread";
            postEvent(this, new FunctorEvent<F>(std::forward<F>(fun), &sem));
            sem.acquire();
        }
    }
}

#ifdef NEEDS_INVOKE_METHOD_FUNCTOR
#define INVOKE_METHOD(app,fnc) app->invokeMethod(fnc)
#else
#define INVOKE_METHOD(app,fnc) QMetaObject::invokeMethod(app, fnc, PVCConnection)
#endif

#define PVCAPP_FUNCTION(ptr,fnc) { \
    PVCApplication *app = static_cast<PVCApplication*>(ptr); \
    INVOKE_METHOD(app, [=]() { app-> fnc ; }); \
}

Of course those class definitions also have to be included in the compilation units that use them, but it would seem like a missed opportunity if C++ doesn't have a way to pass the function object to a generic function from where it can still be evoked without. It can do that with virtual class functions, after all...

I know Iain is supposedly working on blocks support for GCC. An implementation of it existed for Apple's GCC 4.2 version so it's not impossible.

The easiest solution would of course be to just use clang, but it's not exactly the compiler that ends all other compilers either. The big advantage of much faster build times has been lost a long time ago (and GCC is actually the winner here nowadays).

[gershnik]

Maybe the underlying question should be whether (the most common functions in) libdispatch require the full potential of the blocks mechanism.

Well, there are the _f variants for all the calls that take plain old function pointers. What blocks let you do is to associate state with the callback and have it be automatically lifecycle-managed. If you have such state it is almost impossible to do correctly from the 'outside' because you do not know when the pointer is no longer used 'inside'.

struct FunctorEvent

Have you tried to instantiate it with a functor that is non-copyable or non-movable? Lots of complexity in BlockUtil.h is due to having to handle all these cases.

QSemaphore sem; postEvent(this, new FunctorEvent(std::forward(fun), &sem)); sem.acquire();

It seems that there is a race condition here between sem.acquire and event execution....

it would seem like a missed opportunity if C++ doesn't have a way to pass the function object to a generic function from where it can still be evoked without. It can do that with virtual class functions, after all...

Well generic function and not compiling its code is a contradiction in terms in C++. Templates are instantiated at compile time. You can certainly pass any callable to a non-generic, separately compiled function using std::function but it comes with a (big) runtime cost precisely because it erases the type. (Many languages have runtime generics as opposed to templates but these all come with runtime cost too).

You can certainly have separately compiled something like this:

void my_dispatch_async(dispatch_queue_t queue, std::function<void ()> func) {
    dispatch_async(queue, ^ {
         func();
    });

and pass any callable to it. This will result in a couple of memory allocations though. Whether this is something you care about depends on context.

What BlockUtil tries to do is to provide a zero-overhead lambda invocation but it requires compiling templates in.

[RJVB]

On Wednesday August 07 2024 11:25:11 Eugene Gershnik wrote:

struct FunctorEvent

Have you tried to instantiate it with a functor that is non-copyable or non-movable?

No, only with [=]() { ... } lambdas and regular (non-member) functions, because that's all I need in this case.

QSemaphore sem; postEvent(this, new FunctorEvent(std::forward(fun), &sem)); sem.acquire();

It seems that there is a race condition here between sem.acquire and event execution....

That's what I have been wondering about too, but apparently there's not. This comes from https://stackoverflow.com/questions/21646467/how-to-execute-a-functor-or-a-lambda-in-a-given-thread-in-qt-gcd-style/21653558#21653558 and Qt's own QMetaCode::invokeMethod() variant that takes a generic lambda/function does very much the same thing. I suppose the magic must be in the fact that the QEvent calls the function and then releases the semaphore in its dtor.

R.

[gershnik]

You can easily end up with a non-copyable or non-movable lambdas by virtue of having captures with these properties. E.g. something like this for non-copyable lambda

std::unique_ptr<foo> ptr = ...;
invoke_lambda([ptr=std::move(ptr)]() {
    ptr->bar();
});

and similarly for a non-movable captured objects.

That's what I have been wondering about too, but apparently there's not.

Well the examples show how to run something on another thread, which is fine. The problem is that if the FunctorEvent gets scheduled and executed (via destructor) before postEvent even returns then the semaphore will be released first and then acquired. No idea if this is safe with Qt semaphores or not but most semaphore implementations wouldn't like that.

[RJVB]

On Wednesday August 07 2024 14:02:22 Eugene Gershnik wrote:

and similarly for a non-movable captured objects.

Good to know. What happens when one is naughty like that? ;)

Well the examples show how to run something on another thread, which is fine.

Exactly my use case.

The problem is that if the FunctorEvent gets scheduled and executed (via destructor) before postEvent even returns

My hunch is that there is some sort of mechanism in place that prevents that from happening.

then the semaphore will be released first and then acquired. No idea if this is safe with Qt semaphores or not but most semaphore implementations wouldn't like that.

A priori the semaphore wouldn't have any available resources left, so acquire() would block.

Actually, that may exactly be the mechanism by which the execution on the background thread is done synchronously, blocking the calling thread until the background operation has terminated. Which is the use case here. FWIW, background actually means foreground: I use these invokeMethod calls to handle GUI business from callbacks evoked by a threaded libpulse mainloop. The application is meant to control the PulseAudio daemon, not to do any sound playback or processing itself so it's perfectly acceptable that the GUI event loop constrains the libpulse mainloop. Most Qt GUI calls are themselves asynchronous so those blocking calls should never take long.

[gershnik]

This is now available in Release 3.1. See also Linux notes in README.

[RJVB]

Thanks again!

[barracuda156]

AFAIU, Apple/Clang blocks are standard C structures behind the scenes, and support for them on Linux only relies on building with clang, not on ObjC (but possibly also on libBlocksRuntime provided by libdispatch).

libBlocksRuntime is a part of LLVM suite, not of libdispatch, AFAIK.

It would be really nice to have some kind of interface to convert C++ closures and/or regular function pointers to blocks in order to make libdispatch more generally usable regardless of the compiler of choice. (Or even to use GCC on Mac now that it comes with libc++ support.)

It would be very useful to have the reverse working, so that we could avoid blocks in favor of a standard C++.

P. S. Just in case, there is a dispatch implementation which is supposed to be pretty flexible, though I did not test in beyond running its own tests. https://ports.macports.org/port/xdispatch2 Upstream: https://codeberg.org/emzeat/xdispatch2 This even works on powerpc MacOS (without Qt5 part of course).

[gershnik]

libBlocksRuntime is a part of LLVM suite, not of libdispatch, AFAIK.

Correct, though the getting it by building libdispatch seems to be the easiest way. FWIW on Ubuntu 24.04 installing libblocksruntime via apt brings in some ancient version that lacks symbols this library needs to use. Building libdispatch from sources installs version that is compatible to what you get on Mac.

It would be very useful to have the reverse working, so that we could avoid blocks in favor of a standard C++.

This is one reason this library is written 🙂

Just in case, there is a dispatch implementation which is supposed to be pretty flexible

I looked into its implementation and it has allocations, allocations everywhere. E.g. see https://codeberg.org/emzeat/xdispatch2/src/commit/eaae52d20ec26e82952e73de28477f65b4a2052f/src/libdispatch/libdispatch_queue.cpp line 50.

Now, in many contexts you don't care about it of course. Personally, I prefer low level stuff to not add any overhead though. Using BlockUtil.h you should be able to achieve 0 overhead using C++ callables compared to direct block/function use. (Unless there are bugs of course, in which case let me know).

[barracuda156]

Making any of this work with GCC is a whole another story and a much bigger project (if it is feasible at all). GCC lacks support for blocks and I don't even know if libdispatch itself is usable from it.

Why not? libdispatch itself does not strictly require blocks. I have built [a legacy version of] libdispatch on powerpc and I seems to work fine with gcc. (Clangs are broken, so not an option there. Blocks not supported in result.)

Re gcc and blocks, there are plans of upstream re adding it. See: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78352

It certainly requires clang to compile itself. I'll look into it but this will probably take quite some time.

It would be great if this works with gcc, which had a much better cross-platform support.

[gershnik]

Why not? libdispatch itself does not strictly require blocks.

Correct but dealing with lifecycle of C++ callables with (func_ptr, void *) approach correctly is a major pain in the butt. You will end up allocating a wrapper (like xdispatch does) of course. But then the inevitable question arrives: who will delete it if the task ends up not being executed on the queue for one reason or another. And down the rabbit hole we go... 🙂 The advantage of blocks is that all these lifecycle issues have been already handled for you correctly from the inside of dispatch machinery.

Re gcc and blocks, there are plans of upstream re adding it. See: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78352

Right, this is mentioned in this thread and the README.

It would be great if this works with gcc, which had a much better cross-platform support.

Indeed but given that active work seems to be happening on supporting blocks in GCC right now it seems prudent to simply wait for it to be done.

[barracuda156]

Indeed but given that active work seems to be happening on supporting blocks in GCC right now it seems prudent to simply wait for it to be done.

Yeah, fair enough. Hopefully Iain handles that by gcc15 release. We have some software broken due to missing blocks support on powerpc.