Bindgen: D language - Githubissues

kassane commented 6 months ago

Based on gen_zig.py.

Result: https://github.com/kassane/sokol-d (unofficial)

Note: I will archive my repository if there is interest in official support.

Reference

cc: @floooh

floooh commented 6 months ago

Wow interesting! I don't know yet whether I will merge before new year because I'm currently back in retro-land and I want to take time to test the bindings (which means getting at least a bit familiar with the D language and toolchain), but I'm definitely interested to make this "official".

PS: would be nice to have a handful samples in the bindings repository, like in the other bindings :)

floooh commented 6 months ago

PS: please ignore the CI pipeline errors for now, the Emscripten error is caused by a Clang update (they have a new warning), and the error in the gen-bindings job needs to be fixed in the github actions script (I think).

kassane commented 6 months ago

I'm working on the examples and the cross-compiling easy using zig as a reference.

WiP

$> zig build debugtext_print --verbose --summary all
# Zig commands
/home/kassane/zig/0.12.0-dev.1828+225fe6ddb/files/zig build-lib -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_log.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_app.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_gfx.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_glue.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_time.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_audio.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_gl.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_debugtext.c -cflags -DIMPL -DSOKOL_GLCORE33   -DSOKOL_DISABLE_WAYLAND -- /home/kassane/sokol-d/src/sokol/c/sokol_shape.c -lasound -lGL -lX11 -lXi -lXcursor -lc --cache-dir /home/kassane/sokol-d/zig-cache --global-cache-dir /home/kassane/.cache/zig --name sokol -static -fcompiler-rt -fPIE --listen=- 

# LDC (LLVM D Compiler commands) based on libsokol config
/usr/bin/ldc2 -d-debug --gc -g --O0 -w --wo --vcolumns -od=/home/kassane/sokol-d/zig-cache/o/4030eb13042ae0eaeb90332d5358ac3e --oq --Hkeep-all-bodies -I/home/kassane/sokol-d/src/sokol /home/kassane/sokol-d/src/sokol/app.d /home/kassane/sokol-d/src/sokol/audio.d /home/kassane/sokol-d/src/sokol/gl.d /home/kassane/sokol-d/src/sokol/gfx.d /home/kassane/sokol-d/src/sokol/glue.d /home/kassane/sokol-d/src/sokol/log.d /home/kassane/sokol-d/src/sokol/shape.d /home/kassane/sokol-d/src/sokol/time.d /home/kassane/sokol-d/src/sokol/debugtext.d /home/kassane/sokol-d/src/examples/debugtext_print.d -L-L/home/kassane/sokol-d/zig-out/lib -L-lsokol -L-lasound -L-lGL -L-lX11 -L-lXi -L-lXcursor --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DIMPL --Xcc=-DSOKOL_GLCORE33 --Xcc=-DSOKOL_DISABLE_WAYLAND --Xcc=-DSOKOL_GLCORE33 --mtriple=x86_64-linux-gnu --mcpu=znver3 --of=/home/kassane/sokol-d/zig-out/bin/debugtext_print 

Build Summary: 5/5 steps succeeded
debugtext_print success
└─ run /usr/bin/ldc2 success 819ms MaxRSS:291M
   └─ install success
      └─ install sokol success
         └─ zig build-lib sokol Debug native success 852ms MaxRSS:118M

kassane commented 6 months ago

Some examples are added.

clear works!!
sgl_context - fixme
debugtext_print - fixme (rewrite format solution, now write on terminal/console [works])
cube - fixme (missing glsl)

Having completed some initial tests with the added (functional) examples, bindgen will be enhanced according to the configurations of the ported examples.

Currently, the main issue facing me is mangling (D ABI by default) where some generated functions ( especially the logger) need extern(C). Also, the glsl files no have yet to be ported... [TODO]

kassane commented 6 months ago

Also, the glsl files no have yet to be ported...

Ported!! WiP cube example...

After porting some zig/rust examples to D, I still didn't get a satisfactory result.

I am evaluating the issue of structs and memory layout on the part of D, even in BetterC mode and using extern(C) (no mangling).

References

floooh commented 6 months ago

Is it a good time for me to start looking into the PR already, or should I wait a bit (and pick another topic in the meantime)?

kassane commented 6 months ago

Is it a good time for me to start looking into the PR already, or should I wait a bit (and pick another topic in the meantime)?

Feel free to review it!

kassane commented 6 months ago

The sokol-d project repository is not available in CI/CD! Want me to add my repo or give you the sokol-d repository? (No problem for me... I can continue to help you by contributing)

floooh commented 6 months ago

Hmm, testing on my M1 Mac with ldc2 I'm seeing a couple of weird issues:

Warnings like this:

'apple_a14' is not a recognized processor for this target (ignoring processor)
ld: warning: no platform load command found in... debugtext_print.o', assuming: macOS
ld: warning: object file ... was built for newer 'macOS' version (14.1.2) than being linked (12.0)

...it's interesting that debugtext_print.o stands out and this is the sample that fails with an assertion.

Each sample has strange behaviour:

the clear sample doesn't cycle through the colors, it's just a static orange
the cube sample only shows a bright blue background but no cube
the blend sample only has a black background
the debugtext sample has this weird assertion you're also seeing: Assertion failed: (res.canvas_width > 0.0f), function _sdtx_context_desc_defaults, file sokol_debugtext.h, line 3780.
the sgl_context sample also just shows a bright blue background

I'll try to come up with a way to debug via VSCode.

Should I use a different platform for testing for now?

In the meantime though I will fix the sokol-zig build files for the latest zig build system changes ;)

kassane commented 6 months ago

Hmm, testing on my M1 Mac with ldc2 I'm seeing a couple of weird issues:

Warnings like this:

What flags were used? I tested only on linux and windows (macOS, only via CI/CD).

Any errors by the target will include the flags renamed by zig to ldc2/ldmd2. https://github.com/kassane/sokol-d/blob/c8f10530339fcd16cee7e927f7177255877f9f06/build.zig#L485-L491

If possible, share in the sokol-d issue the output of this in verbose.

floooh commented 6 months ago

What flags were used?

Hmm, I was just running zig build clear etc... and then started zig-out/bin/clear.

If you haven't tested macOS so far then it's not all that surprising though.

I will check on my Windows and Linux machine next (probably on Monday when I'm back in Berlin, and I can also check on my old Intel Mac there).

We can figure out what the problem is on M1 Macs later :)

kassane commented 5 months ago

Hmm, I was just running zig build clear etc... and then started zig-out/bin/clear.

After some changes to build.zig, to build and run are similar to sokol-zig. https://github.com/kassane/sokol-d/commit/ad3ca49daba6fdbd410ca055237247ecb560839d

If you haven't tested macOS so far then it's not all that surprising though.

I will check on my Windows and Linux machine next (probably on Monday when I'm back in Berlin, and I can also check on my old Intel Mac there).

We can figure out what the problem is on M1 Macs later :)

Nice! I made some improvements regarding support for MacOS and Windows.

kassane commented 5 months ago

Remove helper code (by @ryuukk)

There will be no more auto-generation of helper functions. Providing these conversions will be the responsibility of the user. It may also be possible to add helper functions to the sokol.utils module without having to include them in all of the generated modules.

https://github.com/kassane/sokol-d/commit/06840586d76619d1c63af55869e82cb9ae3a5aaa

kassane commented 5 months ago

Now, solved NaN init issues! (debugtext_print and sgl_context fixed) But, need fix shaders in pipeline objects conflicts!

kassane commented 5 months ago

Finally CI (gen d binding added) :heavy_check_mark:

kassane commented 4 months ago

For initial support, this solution already works. Waiting for your review! cc: @floooh

-- Edit

Any corrections and improvements will be handled by sokol-d directly, since I haven't yet identified a reason from bindgen.

https://github.com/kassane/sokol-d/issues/13

floooh commented 4 months ago

Ok, I'll start looking into this again now.

floooh commented 4 months ago

Did you get debugging to work somewhow? I'm trying this extension on macOS: https://marketplace.visualstudio.com/items?itemName=webfreak.code-d, and while it starts the debuggee, it doesn't stop on breakpoints.

(I'm using the ldc debugger on macOS btw via brew install dlc, is there a better option?

floooh commented 4 months ago

On Linux I'm getting tons of errors like this:

/home/floh/projects/sokol/bindgen/sokol-d/src/sokol/gfx.d(1813,30): Error: function `sokol/home/floh/projects/sokol/bindgen/sokol-d/src/sokol/gfx.d(1813,30): Error: function `sokol.gfx.sg_d3d11_device` functions cannot be `scope`
.gfx.extern(C) scope const(void)* sg_d3d11_device() @system @nogc nothrow;

...with this ldc version:

sokol-d ➤ ldc2 --version
LDC - the LLVM D compiler (1.30.0):
  based on DMD v2.100.1 and LLVM 14.0.6
  built with LDC - the LLVM D compiler (1.30.0)

...on macOS the ldc2 version is slightly more recent:

bin ➤ ldc2 --version
LDC - the LLVM D compiler (1.37.0):
  based on DMD v2.107.1 and LLVM 17.0.6
  built with LDC - the LLVM D compiler (1.37.0)

floooh commented 4 months ago

...on Ubuntu with ldc2 version 1.30.0, if I remove the scope attribute from functions, the samples compile (with a couple of warnings about GC and thread-local variables), but there are problems:

All samples that render directly via D code are blank (which could mean that something about shaders or the shader-desc is off). The samples which render indirectly through other sokol headers (e.g. sokol_debugtext.h or sokol_gl.h) seem to render fine.

This is on Ubuntu though, on macOS all samples are either blank or crash outright.

kassane commented 4 months ago

Currently I use lldb extension to debug in vscode.

For scope issue, I also suggest trying to remove the -preview=all dflag (experimental features - including DIP1000 that makes @safe/scope more restrictive) https://github.com/kassane/sokol-d/blob/main/build.zig#L231-L235

kassane commented 4 months ago

...on Ubuntu with ldc2 version 1.30.0, if I remove the scope attribute from functions, the samples compile (with a couple of warnings about GC and thread-local variables), but there are problems:

The scope only becomes impactful for those who use the @safe attribute.

TLS and GC warnings are intentional in debug mode only. Flags: -vgc, -vtls. Making it easier to identify behavior for non D developers. https://github.com/kassane/sokol-d/blob/main/build.zig#L316-L326

Any allocation (dynArray or assocArray/HashMap) requires GC (stdlib phobos2 requires GC on almost everything, except core.stdc imports).

All samples that render directly via D code are blank (which could mean that something about shaders or the shader-desc is off).

This would justify the absence of rendering.

This is on Ubuntu though, on macOS all samples are either blank or crash outright.

Some examples for me, worked on archlinux, windows11, and wasm32-emscripten (for D is freestanding).

floooh commented 4 months ago

Quick update:

I can get debugging to work with the CodeLLDB extension in VSCode if I remove this option from the build.zig:

    // remove object files after success build, and put them in a unique temp directory
    if (options.kind != .obj)
        try cmds.append("-cleanup-obj");

...the warning 'apple_a14' is not a recognized processor for this target (ignoring processor) seems to be triggered by the code which sets the target triple and processor (I just commented all this stuff out to home in on the problems I'm seeing)

    // ldc2 doesn't support zig native (a.k.a: native-native or native)
    //    const mtriple = if (options.target.result.isDarwin())
    //        b.fmt("{s}-apple-{s}", .{ if (options.target.result.cpu.arch.isAARCH64()) "arm64" else @tagName(options.target.result.cpu.arch), @tagName(options.target.result.os.tag) })
    //    else if (options.target.result.isWasm())
    //        b.fmt("{s}-unknown-unknown-wasm", .{@tagName(options.target.result.cpu.arch)})
    //    else if (options.target.result.isWasm() and options.target.result.os.tag == .wasi)
    //        b.fmt("{s}-unknown-{s}", .{ @tagName(options.target.result.cpu.arch), @tagName(options.target.result.os.tag) })
    //    else
    //        b.fmt("{s}-{s}-{s}", .{ @tagName(options.target.result.cpu.arch), @tagName(options.target.result.os.tag), @tagName(options.target.result.abi) });
    //
    //    try cmds.append(b.fmt("-mtriple={s}", .{mtriple}));

    // cpu model (e.g. "baseline")
    //if (options.target.query.isNative())
    //    try cmds.append(b.fmt("-mcpu={s}", .{builtin.cpu.model.name}));

With trimming down the ldmd2 command line I'm now actually able to debug:

For reference an ldmd2 command line currently looks like this (with lots of commented out build.zig code):

.{ /opt/homebrew/Cellar/ldc/1.37.0/bin/ldmd2, -w, -preview=all, -gc, -od=/Users/floh/projects/sokol-d/zig-cache/o/5a68ee21cbd019e22687ed19bacef6e7, -cache=/Users/floh/projects/sokol-d/zig-cache/o/5a68ee21cbd019e22687ed19bacef6e7, -oq, -disable-verify, -Hkeep-all-bodies, -i, -I/Users/floh/projects/sokol-d/src, /Users/floh/projects/sokol-d/src/examples/clear.d, -L-w, -Xcc=-ObjC, -Xcc=-DIMPL, -Xcc=-DSOKOL_METAL, -L-framework, -LFoundation, -L-framework, -LAudioToolbox, -L-framework, -LMetalKit, -L-framework, -LMetal, -L-framework, -LCocoa, -L-framework, -LQuartzCore, -of=/Users/floh/projects/sokol-d/zig-out/bin/clear }

floooh commented 4 months ago

Ok, I'm working myself through the cube sample. There's quite a few issues:

The sg.Range helper seems to be broken, it returns a .size of 8 (the size of a pointer) instead of the size of the array data, I guess because the argument here is a pointer (e.g. sgutil.asRange(&vertices[0])}).

I replaced the sgutil.asRange calls with manual code like this (from the cube sample):

    sg.BufferDesc vbuffer = {
        data: {
            ptr: vertices.ptr,
            size: vertices.length * 4,
        }
    };
    state.bind.vertex_buffers[0] = sg.makeBuffer(vbuffer)

A fix for the sg.asRange() helper function would of course be better.

The indices array in the cube sample was of data type double, but it needs to be ushort:

 ushort[36] indices = [...];

The computeVsParams function returns a matrix made completely of NaNs:

The reason are the rx and ry values which are not initialized (and thus NaN and then poison all math operations). Unfortunate that D doesn't warn about this :/ Initializing the rx and ry variables with 0 fixes this specific problem. I didn't verify that the mvp matrix is valid now (only that it no longer consists on NaNs).

After changing the uniform range statement like this: sg.Range r = { ptr: &vsParams, size: vsParams.sizeof }; I had to add scope to the sg.applyUniforms data arg:

void applyUniforms(ShaderStage stage, uint ub_index, scope ref Range data) @trusted @nogc nothrow {...

...otherwise I would get a compile error:

Error: scope variable `r` assigned to non-scope parameter `data` calling `applyUniforms

...despite all those fixes I still get a blank screen in the cube sample. I'll try to investigate more over the next days.

All in all, having a proper step debugger helps a lot in finding those issues.

kassane commented 4 months ago

...the warning 'apple_a14' is not a recognized processor for this target (ignoring processor) seems to be triggered by the code which sets the target triple and processor (I just commented all this stuff out to home in on the problems I'm seeing)

Zig prefers native compilation on the host target, unlike other compilers. To solve this problem on MacOS, I think the removal mentioned below is sufficient. (I haven't seen anything like this on any other operating system).

    if (options.target.query.isNative())
        try cmds.append(b.fmt("-mcpu={s}", .{builtin.cpu.model.name}));

Or need rename, like mtriple targets.

$> zig build-obj -target aarch64-macos -mcpu= --show-builtin
info: available CPUs for architecture 'aarch64':
 [...]
 apple_a10
 apple_a11
 apple_a12
 apple_a13
 apple_a14 # HERE
 apple_a15
 apple_a16
[...]

# like clang output (replacing `-mtriple=arm64-apple` to `-target aarch64`)
$> ldc2 -mtriple=arm64-apple -mcpu=help
Targeting aarch64. Available CPUs for this target:
 [...]
  apple-a10       - Select the apple-a10 processor.
  apple-a11       - Select the apple-a11 processor.
  apple-a12       - Select the apple-a12 processor.
  apple-a13       - Select the apple-a13 processor.
  apple-a14       - Select the apple-a14 processor. # HERE
  apple-a15       - Select the apple-a15 processor.
  apple-a16       - Select the apple-a16 processor.
[...]
  apple-m1        - Select the apple-m1 processor.
[...]

kassane commented 4 months ago

The sg.Range helper seems to be broken, it returns a .size of 8 (the size of a pointer) instead of the size of the array data, I guess because the argument here is a pointer (e.g. sgutil.asRange(&vertices[0])}).

I replaced the sgutil.asRange calls with manual code like this [...]

Well, remembering that this implementation was still in wip, this is really a problem that is still an open issue.

https://github.com/kassane/sokol-d/issues/6

floooh commented 3 months ago

Ok, the last remaining problem is about the math library, somehow the mvp matrix in the cube.d sample comes out wrong.

When replacing the matrix computation with static values copied from the cube-sapp.c sample like this, it works:

shd.VsParams vsParams = {
    mvp: Mat4([
        [1.73099566, 0.0209092628, 0.0337720774, 0.0337046012 ],
        [0, 2.23033166, -0.259949386, -0.259429991 ],
        [0.0604476966, -0.598763049,  -0.967105925, -0.965173661 ],
        [0, -6.88255212E-8, 6.17702007, 6.18465853 ]
    ])
};

So the required changes are:

remove or fix the asRange helper function
fix the math.d module to behave like HandmadeMath.h
add a scope to the applyUniforms data arg

Also: for static arrays .sizeof can be used to populate the sg_range structs, however for dynamic arrays .sizeof always seems to be zero and .length * itemSize must be used instead.

E.g. for static arrays this works:

    sg.BufferDesc vbuffer = {
        data: {
            ptr: vertices.ptr,
            size: vertices.sizeof,
        }
    };

...but for dynamic arrays, this must be used:

    sg.BufferDesc vbuffer = {
        data: {
            ptr: vertices.ptr,
            size: vertices.length * float.sizeof,
        }
    };

...or alternatively this also works:

    sg.BufferDesc vbuffer = {
        data: {
            ptr: vertices.ptr,
            size: vertices.length * vertices[0].sizeof,
        }
    };

floooh commented 1 month ago

Note: I'm going to put a 'scope' in front of all pointer and ref params in my merge-thread, that way the init() function in the samples doesn't need a @trusted, and in the sokol headers there's generally no ownership transfer, so it should be safe (if I understand the meaning of scope right.

floooh commented 1 month ago

This is my local 'merge-branch' btw:

https://github.com/floooh/sokol/tree/kassane-dlang_bindgen

kassane commented 1 month ago

This is my local 'merge-branch' btw:

https://github.com/floooh/sokol/tree/kassane-dlang_bindgen

Merge your minor fixes...

floooh commented 1 month ago

Ok, so just to clarify :)

I'll merge my inflight-PR which is this PR plus the minor scope changes
after the merge each commit to the sokol repo will push the updated bindings to https://github.com/kassane/sokol-d via CI (which I think should work because you added me as collaborator)
this means https://github.com/kassane/sokol-d will become the "official" D bindings repo, and I'll delete my temp fork (which I created for the PR to fix the samples) to avoid confusion

Let me know if that makes sense and I'll do the merge :)

PS: We'll have to see how the above model works when there are breaking changes to the sokol headers and the examples in the bindings need to be updated, in this case I usually create a number of related PRs for the language bindings which then need to be merged right before the sokol main repository is merged. But for smaller things I guess I can handle those myself without requiring actions from your side (when in doubt I'll wait for your approval though).

floooh commented 1 month ago

Ok, brace for impact :D

floooh commented 1 month ago

...and merged. Next I'll check if the bindings CI pipeline works:

https://github.com/floooh/sokol/actions/runs/9064596060

...then I'll update the changelog and announce the new bindings.

Thanks for the hard work and apologies for the long delay.

floooh commented 1 month ago

...and failed :D https://github.com/floooh/sokol/actions/runs/9064596060/job/24903528985

...but I know why, I need to setup an access token in both repositories, one sec, let me check if I can do this on my own.

floooh commented 1 month ago

Ok, I need to figure out again how this works :D But I definitely need your help because I don't have access to the Github settings of https://github.com/kassane/sokol-d (we'll need to add a deploy key there).

This also involves creating an ssh public/private key pair, I just need to figure out again where the public and pivate key goes, and whether you or me need to create the key pair, one sec...

floooh commented 1 month ago

Hmm ok interesting, looks like the public key goes into the deploy key of the target repository...

@kassane can you do the following please:

at https://github.com/kassane/sokol-d go to Settings => Deploy Keys
click the Add deploy key button
as title use ghactions_push
check the Allow write access checkbox

into the Key field, copy-paste the following public key:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFcIhdAkeykg/0rKPRKlCf/JrlAVQOwlb00cIAXkCW2r floooh@gmail.com

click the Add Key button and that's it...

Meanwhile on my side in the sokol repository I will add the private key part as the missing GHACTIONS_D_PUSH CI secret, and then pushing from sokol GH Actions into sokol-d shoudl work.

kassane commented 1 month ago

@floooh done.

floooh commented 1 month ago

Ok thanks, let's see what happens when I restart the pipeline...

floooh commented 1 month ago

Ok it worked :) This is the commit: https://github.com/kassane/sokol-d/commit/0b2587fa619be83d7cde1b572926a4b0d4c685aa

(these are the WebGPU changes I merged earlier today)

Ok, then I'll add the bindings link and badge to the readme, add a changelog item and do an announcement.

floooh / sokol

Bindgen: D language #955

Reference

WiP

References