JuliaInterop / ObjectiveC.jl

Objective-C embedded in Julia
Other
39 stars 10 forks source link

Modernize the package #11

Closed maleadt closed 1 year ago

maleadt commented 1 year ago

This PR updates and significantly reworks the ObjectiveC.jl package. Basic updates include adding a Project.toml, fixing deprecations, adding some tests, etc. However, I also decided to rework the core calling mechanism to not rely on introspection and dynamic code generation anymore, instead opting for a more static @ccall-esque approach that allows for far better performance. It should also combine nicely with e.g. a Clang.jl-based code generator, but that's future work.

Reworking the core mechanism probably results in breakage of other functionality in this package, but it's hard to avoid that since there were no tests. In addition, because there hasn't even been a release, and because the package was broken on any recent Julia version anyway, it is probably fine to do so and gradually improve/unbreak functionality along the way.

cc @habemus-papadum

habemus-papadum commented 1 year ago

@maleadt Very nice!

maleadt commented 1 year ago

Replacing the @eval ccall with a generated :foreigncall significantly improves the performance of message calling to the point that it is viable (although we can still improve it by caching the selector lookups, or performing them at parse time if that's possible).

Before:

julia> f() = @objc [[[NSHost currentHost] localizedName] UTF8String]
f (generic function with 1 method)

julia> @benchmark f()
BenchmarkTools.Trial: 1465 samples with 1 evaluation.
 Range (min … max):  3.287 ms …   9.281 ms  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     3.370 ms               ┊ GC (median):    0.00%
 Time  (mean ± σ):   3.409 ms ± 264.106 μs  ┊ GC (mean ± σ):  0.24% ± 2.13%

    ▅▆▇▇█▇▇▇▇▆▅▄▄▃▃▁                                          ▁
  ▇▇████████████████▇████▆▅▆▄▁▄▁▄▅▄▄▁▅▁▄▄▁▄▆▆▆▄▆▅▆▆▆▆▆▅▄▆▆▁▄▄ █
  3.29 ms      Histogram: log(frequency) by time      3.88 ms <

 Memory estimate: 162.00 KiB, allocs estimate: 3229.

After:

julia> @benchmark f()
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max):  30.583 μs … 152.041 μs  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     32.500 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   33.221 μs ±   3.030 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

     ▂▆▂▄▅█▄▂
  ▁▂▆████████▆▅▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂
  30.6 μs         Histogram: frequency by time         44.3 μs <

 Memory estimate: 2.06 KiB, allocs estimate: 56.
maleadt commented 1 year ago

https://gist.github.com/fjolnir/2211379/d77f93efd80690125b933667f79600e67da66c1e is also interesting, where there's caching of class and instance methods, and automatic retain/release based on the function name.

habemus-papadum commented 1 year ago

performing them at parse time if that's possible:

Selector lookups are usually done at load time. For instance this is what the metalcpp wrapper is doing: #define _MTL_PRIVATE_DEF_SEL(accessor, symbol) SEL s_k##accessor _MTL_PRIVATE_VISIBILITY = sel_registerName(symbol);

namespace MTK::Private::Selector
{

_MTK_PRIVATE_DEF_SEL( autoresizeDrawable,
                     "autoresizeDrawable" );
...

Something similar could be done in an __init__ function for a Julia package, and presumably this is what a wrapper tool would do in generated code.

However, being able to do the selector lookup on the fly with relatively low overhead is quite useful -- for situations where one wants to quickly wrap a portion of a framework by hand and also quick experimentation -- so I find this PR really great.

although we can still improve it by caching the selector lookups

Note: the process of invoking a method in Objective-C is:

So caching selector lookups and checking a hashtable in Julia at runtime is not likely to provide any improvement to just calling sel_registername on each method invocation (there will be a hahstable lookup either in Julia or C) the "convert object-c instance and SEL to a concrete method" is probably the largest portion of typical runtime overhead in objective-C, but it is also the source of its dynamism. The runtime does some caching of frequently used methods (https://developer.apple.com/documentation/objectivec/objective-c_runtime/objc_cache?language=objc) to mitigate this. i.e. it probably is neither correct nor necessary for us to do any other performance optimizations.

What the clang compiler also does is static analysis to make sure that objective-c method invocations will not generate runtime errors and that is what a Julia wrapper could do as well (i.e. not improve performance, but help ensure correct use).

Also, a good place to read about the low level details is: https://developer.apple.com/documentation/objectivec/objective-c_runtime?language=objc

maleadt commented 1 year ago

However, being able to do the selector lookup on the fly with relatively low overhead is quite useful -- for situations where one wants to quickly wrap a portion of a framework by hand and also quick experimentation -- so I find this PR really great.

I wouldn't remove that, but require type annotations so that we can statically generate code instead of having to introspect into the ObjC runtime in order to build a ccall expression. I'm exploring that right now, and it does look very promising.

This is how the package currently works:

julia> str = @objc [NSString string]
__NSCFConstantString Object

julia> @objc [str length]
0x0000000000000000

julia> f() = @objc [str length]
f (generic function with 1 method)

Current master branch:

julia> @benchmark f()
BenchmarkTools.Trial: 6509 samples with 1 evaluation.
 Range (min … max):  702.417 μs …   3.822 ms  ┊ GC (min … max): 0.00% … 50.89%
 Time  (median):     735.834 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   765.605 μs ± 129.942 μs  ┊ GC (mean ± σ):  0.29% ±  1.94%

  ▅███▇▆▅▅▄▃▃▂▂▂ ▁▁ ▁   ▁                                       ▂
  ███████████████████▇█▇███▇▇▇▇▇▇▇▇▇▆▇█▇▇▇▆▇▇▆▆▆▇▃▄▅▅▄▅▁▄▅▄▄▁▅▅ █
  702 μs        Histogram: log(frequency) by time       1.25 ms <

 Memory estimate: 59.15 KiB, allocs estimate: 1169.

This PR (removal of @eval, use of generated ccall):

julia> @benchmark f()
BenchmarkTools.Trial: 10000 samples with 10 evaluations.
 Range (min … max):  1.317 μs … 151.921 μs  ┊ GC (min … max): 0.00% … 97.31%
 Time  (median):     1.354 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   1.389 μs ±   1.508 μs  ┊ GC (mean ± σ):  1.06% ±  0.97%

   ▁▄▅▆▆█▆▆▅▄▃▃▃▂▁▁▁▁▂▂▂▂▂▂▂▃▁▁   ▁                           ▂
  ▆████████████████████████████████▇▇█▇▇▆█▆▆▇▆▆▇▆▆▆▄▅▅▆▅▅▄▅▅▆ █
  1.32 μs      Histogram: log(frequency) by time       1.6 μs <

 Memory estimate: 648 bytes, allocs estimate: 16.

Proposed static approach:

julia> str = @objc [NSString string]::id
Ptr{ObjectiveC.OpaqueObject} @0x00000001eb6b4d28

julia> @objc [str::id length]::NSUInteger
0x0000000000000000

julia> f() = @objc [str::Id length]::NSUInteger
f (generic function with 1 method)

julia> @benchmark f()
BenchmarkTools.Trial: 10000 samples with 946 evaluations.
 Range (min … max):  98.661 ns … 137.024 ns  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     99.014 ns               ┊ GC (median):    0.00%
 Time  (mean ± σ):   99.406 ns ±   2.098 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ▃█▇▆▄▂             ▁▁▁▁                                      ▂
  ██████▇▆▅▄▃▅▅▅▅▅▄▅▆█████▅▅▅▅▅▄▄▃▃▅▅▅▄▄▄▄▅▄▄▅▅▆▆▃▅▄▅▄▁▁▃▁▃▄▄▄ █
  98.7 ns       Histogram: log(frequency) by time       109 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

i.e. keeping most of the syntax, only adding some type assertions so that we have the information we need. Note the use of id, I haven't found a good way of encoding additional type information yet (well, it's obvious to switch ::id to ::NSString, but I haven't found a good Object hierarchy yet that makes this possible without too much boilerplate).

maleadt commented 1 year ago

Something that may be worth discussing: in https://github.com/JuliaInterop/ObjectiveC.jl/pull/11/commits/a8c859f3f704b460ddb7e287f5b5eda91f70f954 I decided to 'split' the Julia class definitions we generate with @objcwrapper (a helper macro to generate object classes) into two: an abstract class to implement the hierarchy, and a concrete one that contains the object pointer. The motivation here is that Objective-C has multilevel inheritance, e.g., MTLEvent <: NSObject is a perfectly valid object, but there's also MTLSharedEvent <: MTLEvent which augments MTLEvent. We can't express that in Julia, lacking concrete inheritance, and the traditional alternatives (composition instead of inheritance) break dispatch.

After https://github.com/JuliaInterop/ObjectiveC.jl/pull/11/commits/a8c859f3f704b460ddb7e287f5b5eda91f70f954, we have MTLEvent be an abstract class so that MTLSharedEvent can inherit from it, and objects will be put in the MTLEventInstance or MTLSharedEventInstance types, depending on the requested type. To make this all invisible, I generate a pseudo constructor for the abstract classes (so calling MTLEvent(ptr::id) will give an MTLEventInstance). Case in point, the change did not break the WIP wrappers I have over in https://github.com/JuliaGPU/Metal.jl/pull/117, yet it did allow me to remove the 'fake' MTLAbstractEvent class I had introduced there. So this looks like a potentially interesting solution for supporting Objective-C's object model (although I may still be missing things, as I'm not really familiar with the language yet and am just designing things in function of the Metal APIs).

I also considered making everything an Object{T}, but that would make methods ugly (e.g., foo(obj::Object{<:MTLEvent}) instead of now simply foo(obj::MTLEvent)).

habemus-papadum commented 1 year ago

I was thinking about "Type Modeling" -- @maleadt your approach is better than what I was thinking about, and feels correct. I'll spend time looking at the specifics.

In terms of interop completeness, the othervoutstanding issues I can think of are:

These are main things I can think of needed to create a modern Mac app (e.g. a game) in Julia -- not that that needs to be the actual goal but it is amazing how much progress has been made!

habemus-papadum commented 1 year ago

This might be a good reference for blocks: https://github.com/SSheldon/rust-block, with the goal being an @objcblock macro similar to @cfunction.

habemus-papadum commented 1 year ago

Also, assuming but will check @class macro still works, it will need to support syntax for Protocols, so that Delegates (e.g. AppDelegate) can be implemented in Julia

habemus-papadum commented 1 year ago

Writeup on the calling convention of objc_msgsend. https://www.mikeash.com/pyblog/objc_msgsends-new-prototype.html

habemus-papadum commented 1 year ago

Below is some code that investigates struct parameters and return values.
Some notes: according to https://www.mikeash.com/pyblog/objc_msgsends-new-prototype.html

Demo code for structs

##
using Libdl
using  ObjectiveC

##
function compile(code)
    mktempdir() do p
        f   = joinpath(p,"temp.m")
        lib = joinpath(p,"libtemp.$(Libdl.dlext)")
        open(f,"w") do io
            write(io, code)
        end
        run(`clang -shared -fPIC -framework Foundation $f -o $lib`)
        dlopen(lib)
    end
end

macro dylib_str(code)
    compile(code)
end

##
dl = dylib"""
#import <Foundation/Foundation.h>
#include <stdint.h>

typedef struct {int64_t a;int64_t b;int64_t c;} Bar;

@interface Foo : NSObject
- (int64_t) m0:(int64_t) x;
- (int64_t) m1:(Bar    ) x;
- (Bar    ) m2:(Bar    ) x;
@end
@implementation Foo {} 
- (int64_t) m0:(int64_t) x {return 2*x                       ;}
- (int64_t) m1:(Bar    ) x {return x.a*x.b*x.c               ;}
- (Bar    ) m2:(Bar    ) x {x.a*=2; x.b*=10; x.c*=20;return x;}
@end
"""

##
struct Bar
    a::Int64
    b::Int64
    c::Int64
end
##
Foo = Class("Foo")
foo = @objc [Foo new]::id

## 
b = Bar(3,4,5)
@show @objc [foo::id m0 :3::Int64]::Int64
@show @objc [foo::id m1 :b::Bar  ]::Int64
@show @objc [foo::id m2 :b::Bar  ]::Bar 
##⬆ should not work (but looks like it does...), needs to use objc_msgSend_stret

dlclose(dl)
vchuravy commented 1 year ago

How to define Objective-C blocks in Julia (This seems hard -- the details seemed buried in Clang -- but also important most modern Objective-C apis are moving to blocks instead of delegates. e.g. MtlCommandBuffer hooks)

@tgymnich was looking at blocks yesterday :)

maleadt commented 1 year ago

Okay, this is now at a point where it is sufficiently powerful to wrap all of Metal that Metal.jl needs... except for the two onCompleted APIs that requires blocks. @tgymnich did you find anything out? Clang has documented the ABI, https://clang.llvm.org/docs/Block-ABI-Apple.html, and there's existing FFI implementations like https://crates.io/crates/block, but it looks pretty complicated.

habemus-papadum commented 1 year ago

@maleadt @tgymnich the lua implementation of blocks might be a little easier to understad: https://github.com/rweichler/objc.lua/blob/cbe2a80462fdcc270c37482f648135de079040e7/src/init.lua#L1996-L2070

maleadt commented 1 year ago

This works in Julia:

using ObjectiveC, .Foundation
using CEnum

## NSBlock infrastructure

export NSBlock

@cenum Block_flags::Cint begin
    BLOCK_DEALLOCATING      = 0x0001
    BLOCK_REFCOUNT_MASK     = 0xfffe

    BLOCK_IS_NOESCAPE       = 1 << 23
    BLOCK_NEEDS_FREE        = 1 << 24
    BLOCK_HAS_COPY_DISPOSE  = 1 << 25
    BLOCK_HAS_CTOR          = 1 << 26
    BLOCK_IS_GLOBAL         = 1 << 28
    BLOCK_HAS_STRET         = 1 << 29
    BLOCK_HAS_SIGNATURE     = 1 << 30
end

const _NSConcreteGlobalBlock = cglobal(:_NSConcreteGlobalBlock)
const _NSConcreteStackBlock  = cglobal(:_NSConcreteStackBlock)

struct JuliaBlockDescriptor
    reserved::Culong
    size::Culong
    copy_helper::Ptr{Cvoid}
    dispose_helper::Ptr{Cvoid}
end

struct JuliaBlock
    isa::Ptr{Cvoid}
    flags::Cint
    reserved::Cint
    invoke::Ptr{Cvoid}
    descriptor::Ptr{JuliaBlockDescriptor}

    # custom fields
    lambda::Function
end

@objcwrapper NSBlock <: NSObject

# NSBlock is the opaque version of JuliaBlock, so make it possible to derive an id pointer
# from a boxed block (assuming the caller doesn't use the pointer beyond the box's lifetime)
Base.unsafe_convert(T::Type{id{Object}}, box::Base.RefValue{JuliaBlock}) =
    reinterpret(T, Base.unsafe_convert(Ptr{Cvoid}, box))

## high-level macro

# static descriptor for a simple Julia-based block
const julia_block_descriptor = Ref(JuliaBlockDescriptor(0, sizeof(JuliaBlock), 0, C_NULL))

function julia_block_trampoline(_block, _self, args...)
    block = unsafe_load(_block)
    nsblock = NSBlock(reinterpret(id{NSBlock}, _block))

    # call the user lambda
    block.lambda(args...)
end

macro objcblock(callable, rettyp, argtyps)
    quote
        cb = @cfunction(julia_block_trampoline, $rettyp, (Ptr{JuliaBlock}, id{Object}, $argtyps...))
        GC.@preserve cb begin
            # set-up the block data structures
            trampoline_ptr = Base.unsafe_convert(Ptr{Cvoid}, cb)
            desc_ptr = Base.unsafe_convert(Ptr{Cvoid}, julia_block_descriptor)
            block = JuliaBlock(_NSConcreteStackBlock, 0, 0, trampoline_ptr, desc_ptr, $callable)

            # copy the block to the heap and have Objective-C use that object.
            # our original block was a plain Julia struct so doesn't need to be released.
            # we also don't need to worry about the lifetime of the block object, as the
            # only references to Julia data it contains are the descriptor, which is
            # statically allocated, and the lambda (Julia code currently isn't GC'd).
            @objc [Ref(block)::id{NSBlock} copy]::id{NSBlock}
        end
    end
end

## demo

function hello(x)
    println("Hello, world $(x)!")
    return Cint(42)
end

block = @objcblock(hello, Cint, (Cint,))

# for validation, register our block with a class method
@objcwrapper BlockWrapper <: NSObject
wrapper_class = ObjectiveC.allocclass(:BlockWrapper, Class(:NSObject))
imp = ccall(:imp_implementationWithBlock, Ptr{Cvoid}, (id{NSBlock},), block)
types = (Cint, Object, Selector, Cint)
typestr = ObjectiveC.encodetype(types...)
@assert ccall(:class_addMethod, Bool,
              (Ptr{Cvoid}, Ptr{Cvoid}, Ptr{Cvoid}, Ptr{Cchar}),
              wrapper_class, sel"invoke:", imp, typestr)
ObjectiveC.register(wrapper_class)

# create a wrapper instance and call our block
wrapper = BlockWrapper(@objc [BlockWrapper alloc]::id{BlockWrapper})
ret = @objc [wrapper::id{BlockWrapper} invoke:41::Cint]::Cint
@show ret

Doesn't do any memory management; I'm not sure we need to if we won't capture things (by setting BLOCK_IS_GLOBAL; we can avoid doing so because of Julia's runtime supporting closures with @cfunction).

habemus-papadum commented 1 year ago

Doesn't do any memory management; I'm not sure we need to if we won't capture things (by setting BLOCK_IS_GLOBAL; we can avoid doing so because of Julia's runtime supporting closures with @cfunction).

This sounds right and probably greatly simplifies everything -- though I was wondering why lua did not make this decision (but don;t really know much about lua)

vtjnash commented 1 year ago

Emulated closures can be slow, and not widely supported, so if you can use native closures (eventually) that is preferable. But there is not an immediate need for it.

habemus-papadum commented 1 year ago

Some type modeling questions (probably for a later PR...)

vtjnash commented 1 year ago

Something that may be worth discussing: in a8c859f I decided to 'split' the Julia class definitions we generate with @objcwrapper (a helper macro to generate object classes) into two: an abstract class to implement the hierarchy, and a concrete one that contains the object pointer. The motivation here is that Objective-C has multilevel inheritance, e.g., MTLEvent <: NSObject is a perfectly valid object, but there's also MTLSharedEvent <: MTLEvent which augments MTLEvent. We can't express that in Julia, lacking concrete inheritance, and the traditional alternatives (composition instead of inheritance) break dispatch.

After a8c859f, we have MTLEvent be an abstract class so that MTLSharedEvent can inherit from it, and objects will be put in the MTLEventInstance or MTLSharedEventInstance types, depending on the requested type. To make this all invisible, I generate a pseudo constructor for the abstract classes (so calling MTLEvent(ptr::id) will give an MTLEventInstance). Case in point, the change did not break the WIP wrappers I have over in JuliaGPU/Metal.jl#117, yet it did allow me to remove the 'fake' MTLAbstractEvent class I had introduced there. So this looks like a potentially interesting solution for supporting Objective-C's object model (although I may still be missing things, as I'm not really familiar with the language yet and am just designing things in function of the Metal APIs).

I also considered making everything an Object{T}, but that would make methods ugly (e.g., foo(obj::Object{<:MTLEvent}) instead of now simply foo(obj::MTLEvent)).

I faced this challenge in Gtk.jl and ended up doing something similar to this, with GObject and GObjectLeaf (making the interface the easier type, similar to Ref vs. Base.RefValue). I was never quite satisfied with this though, as it felt awkward, led to excess specialization, was a bit hard to extend in disjoint packages, and never quite mapped properly between the type systems (since calling interface methods requires an explicit type cast in the Julia Gtk.jl wrapper). If I was to revisit that decision, I might have gone with a runtime-traits-based approach instead, perhaps with optional mixins.

mutable struct GObject{Mixin}
    classid::Int # or encoded in gptr
    gptr::Ptr{Cvoid}
    mixins::Mixin
end

Then at runtime, it would be sufficient to do the reflection calls needed to decide if a particular interface was applicable, and perhaps also to provide a custom dispatch table hook for places where it is required. This would have let it provide all of the features (multiple inheritance, concrete inheritance, interfaces with inheritance) that are not viable normally.

Secondly, with the current approach, I think I should have at least implemented many convert-trait methods, and coded them up to be visible to reflection:

abstract type AnyInterface; end
abstract type AnyObject; end
struct Interface{T} <: AnyInterface; obj::T; end
inherits_from(::Type, ::Type) = false
inherits_from(::Type{<:Interface}, ::Type{<:Abstract}) = true
convert(::Type{I}, obj::AnyObject) where {I<:AnyInterface} = I(obj)

then when I auto-generated interface calls such as label, I could also auto-generate dispatch methods that try to select the best matching ccall:

struct HasLabel{T} <: AnyInterface; obj::T; end
label(obj::HasLabel) = ccall
function label(obj)
    inherits_from(HasLabel, obj) && return label(HasLabel(obj))
    throw(InterfaceMissingError(label, obj))
end
maleadt commented 1 year ago

Thanks for the write-up, I should think about if an how to incorporate that here.

Regarding blocks, I pushed an initial version that simply looks like:

function hello(x)
    println("Hello, world $(x)!")
    return Cint(42)
end

block = @objcblock(hello, Cint, (Cint,))

This works when attaching the block to a class, now I'll see if it works for Metal. I'm not sure I'm correctly managing memory (i.e., I'm not doing much). As our block doesn't contain any GC-managed data (only a Julia function, and IIRC functions aren't garbage collected), I think I can get away by just copying it to the heap and doing nothing (well, not doing any refcounting). I'm not sure who's responsible for releasing the block object itself though.

vchuravy commented 1 year ago

; we can avoid doing so because of Julia's runtime supporting closures with @cfunction

No on AArch64 sadly

tgymnich commented 1 year ago

This might be useful for reference.

maleadt commented 1 year ago

; we can avoid doing so because of Julia's runtime supporting closures with @cfunction

No on AArch64 sadly

I have an alternative, storing the jl_value_t pointing to the callable (it be a singleton function or a closure struct), but that does require memory management. So I'll look into that.

vtjnash commented 1 year ago

only a Julia function, and IIRC functions aren't garbage collected

vtjnash commented 1 year ago

cfunction is garbage collected (that's why it is unsafe to convert it to a pointer)

maleadt commented 1 year ago

cfunction is garbage collected (that's why it is unsafe to convert it to a pointer)

All of cfunction, so also when not using closures? But yeah I'll work on adding proper memory management (probably just rooting the Block in a global dict and evicting it from the Block's dispose callback).

maleadt commented 1 year ago

OK, I got the basic functionality to support Metal.jl working, so I think I'll cut a 0.1 release so that I can more easily test this package in a realistic setting. FWIW, I removed the broken bits that I didn't (yet) need, including support for defining classes, so I'll create an issue to keep track of bringing those bits back. But since ObjectiveC.jl hadn't even been tagged, and was completely broken at this point, I don't feel bad removing it for now.