allyourcodebase / ffmpeg

FFmpeg Zig package
Other
181 stars 24 forks source link

WIP: Fix build on macOS #11

Closed dmorn closed 7 months ago

dmorn commented 1 year ago

This is an attempt on fixing building this library on macOS.

The build script was enabling the prctl.h functionality, which is not available on macOS. I've also enabled metal but might be out-of-scope.

dmorn commented 1 year ago

Right now the build fails here

dmorn@silver ~/p/zig-ffmpeg (main)> zig build
zig build-lib ffmpeg Debug native: error: thread 3291682 panic: access of union field 'success' while field 'failure' is active
/Users/dmorn/projects/zig/src/link.zig:1149:77: 0x1014453b3 in linkAsArchive (zig)
            object_files.appendAssumeCapacity(try arena.dupeZ(u8, key.status.success.object_path));
                                                                            ^
/Users/dmorn/projects/zig/src/link/MachO.zig:273:43: 0x1014461a7 in flush (zig)
            return self.base.linkAsArchive(comp, prog_node);
                                          ^
/Users/dmorn/projects/zig/src/link.zig:824:72: 0x10113420f in flush (zig)
            .macho => return @fieldParentPtr(MachO, "base", base).flush(comp, prog_node),
                                                                       ^
/Users/dmorn/projects/zig/src/Compilation.zig:2657:24: 0x1011339af in flush (zig)
    comp.bin_file.flush(comp, prog_node) catch |err| switch (err) {
                       ^
/Users/dmorn/projects/zig/src/Compilation.zig:2602:27: 0x10117290b in update (zig)
            try comp.flush(main_progress_node);
                          ^
/Users/dmorn/projects/zig/src/main.zig:3847:36: 0x10119d89f in serve (zig)
                    try comp.update(main_progress_node);
                                   ^
/Users/dmorn/projects/zig/src/main.zig:3641:22: 0x1011bf767 in buildOutputType (zig)
            try serve(
                     ^
/Users/dmorn/projects/zig/src/main.zig:280:31: 0x100ff5183 in mainArgs (zig)
        return buildOutputType(gpa, arena, args, .{ .build = .Lib });
                              ^
/Users/dmorn/projects/zig/src/main.zig:222:20: 0x100ff27bb in main (zig)
    return mainArgs(gpa, arena, args);
                   ^
/Users/dmorn/projects/zig/lib/std/start.zig:585:37: 0x100ff23bf in main (zig)
            const result = root.main() catch |err| {
                                    ^
???:?:?: 0x18d4bd0df in ??? (???)
???:?:?: 0x7801ffffffffffff in ??? (???)

zig build-lib ffmpeg Debug native: error: the following command terminated unexpectedly:
/Users/dmorn/projects/zig/build/stage3/bin/zig build-lib @/Users/dmorn/projects/zig-ffmpeg/zig-cache/args/1a877de3838a7ee5f71bfd503464abf522439331278a51bff173e09a95013d3a
Build Summary: 149/154 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
├─ install ffmpeg transitive failure
│  └─ zig build-lib ffmpeg Debug native failure
└─ install show_metadata transitive failure
   └─ zig build-exe show_metadata Debug native transitive failure
      └─ zig build-lib ffmpeg Debug native (+11 more reused dependencies)
error: the following build command failed with exit code 1:
/Users/dmorn/projects/zig-ffmpeg/zig-cache/o/523d4cde3388476c10170aa618ed62f1/build /Users/dmorn/projects/zig/build/stage3/bin/zig /Users/dmorn/projects/zig-ffmpeg /Users/dmorn/projects/zig-ffmpeg/zig-cache /Users/dmorn/.cache/zig --seed 0x45fa5b28

right now I don't really have a clue. Any hint would be helpful!

dmorn commented 1 year ago

I'm on zig @ 0.12.0-dev.1768+39a966b0a

andrewrk commented 1 year ago

Nice stack trace, I think you managed to crash the linker! cc @kubkon who I believe happens to be in the middle of a macho rewrite 😃

andrewrk commented 1 year ago

Actually that might be a frontend bug, it shouldn't be passing unsuccessful objects to the linker. Anyway I'll look into it soon.

kubkon commented 1 year ago

Nice stack trace, I think you managed to crash the linker! cc @kubkon who I believe happens to be in the middle of a macho rewrite 😃

If it's a MachO bug, please do let me know and I'll try to incorporate a fix into the rewrite - I am getting really close at something really usable for the more difficult arch out of the two (x86 being so much more hacky in MachO than arm64).

dmorn commented 1 year ago

Thanks for joining! 🫶 let me know if there is anything more you would like me to provide!

dmorn commented 1 year ago

I tried to debug it with lldb but I cannot backtrace as the process panics and exits, even though I set a breakpoint at link.zig:1149.

kubkon commented 12 months ago

Right now the build fails here

dmorn@silver ~/p/zig-ffmpeg (main)> zig build
zig build-lib ffmpeg Debug native: error: thread 3291682 panic: access of union field 'success' while field 'failure' is active
/Users/dmorn/projects/zig/src/link.zig:1149:77: 0x1014453b3 in linkAsArchive (zig)
            object_files.appendAssumeCapacity(try arena.dupeZ(u8, key.status.success.object_path));
                                                                            ^
/Users/dmorn/projects/zig/src/link/MachO.zig:273:43: 0x1014461a7 in flush (zig)
            return self.base.linkAsArchive(comp, prog_node);
                                          ^
/Users/dmorn/projects/zig/src/link.zig:824:72: 0x10113420f in flush (zig)
            .macho => return @fieldParentPtr(MachO, "base", base).flush(comp, prog_node),
                                                                       ^
/Users/dmorn/projects/zig/src/Compilation.zig:2657:24: 0x1011339af in flush (zig)
    comp.bin_file.flush(comp, prog_node) catch |err| switch (err) {
                       ^
/Users/dmorn/projects/zig/src/Compilation.zig:2602:27: 0x10117290b in update (zig)
            try comp.flush(main_progress_node);
                          ^
/Users/dmorn/projects/zig/src/main.zig:3847:36: 0x10119d89f in serve (zig)
                    try comp.update(main_progress_node);
                                   ^
/Users/dmorn/projects/zig/src/main.zig:3641:22: 0x1011bf767 in buildOutputType (zig)
            try serve(
                     ^
/Users/dmorn/projects/zig/src/main.zig:280:31: 0x100ff5183 in mainArgs (zig)
        return buildOutputType(gpa, arena, args, .{ .build = .Lib });
                              ^
/Users/dmorn/projects/zig/src/main.zig:222:20: 0x100ff27bb in main (zig)
    return mainArgs(gpa, arena, args);
                   ^
/Users/dmorn/projects/zig/lib/std/start.zig:585:37: 0x100ff23bf in main (zig)
            const result = root.main() catch |err| {
                                    ^
???:?:?: 0x18d4bd0df in ??? (???)
???:?:?: 0x7801ffffffffffff in ??? (???)

zig build-lib ffmpeg Debug native: error: the following command terminated unexpectedly:
/Users/dmorn/projects/zig/build/stage3/bin/zig build-lib @/Users/dmorn/projects/zig-ffmpeg/zig-cache/args/1a877de3838a7ee5f71bfd503464abf522439331278a51bff173e09a95013d3a
Build Summary: 149/154 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
├─ install ffmpeg transitive failure
│  └─ zig build-lib ffmpeg Debug native failure
└─ install show_metadata transitive failure
   └─ zig build-exe show_metadata Debug native transitive failure
      └─ zig build-lib ffmpeg Debug native (+11 more reused dependencies)
error: the following build command failed with exit code 1:
/Users/dmorn/projects/zig-ffmpeg/zig-cache/o/523d4cde3388476c10170aa618ed62f1/build /Users/dmorn/projects/zig/build/stage3/bin/zig /Users/dmorn/projects/zig-ffmpeg /Users/dmorn/projects/zig-ffmpeg/zig-cache /Users/dmorn/.cache/zig --seed 0x45fa5b28

right now I don't really have a clue. Any hint would be helpful!

I only now managed to get a good look at the stack trace and if it is valid, we crash when attempting to pass object(s) to llvm-ar or right before.

dmorn commented 11 months ago

Hi @andrewrk and @kubkon! Any update? I would love to help you out but I need some guidance, I'm not managing to debug the code properly as mentioned above

kubkon commented 11 months ago

Hi @andrewrk and @kubkon! Any update? I would love to help you out but I need some guidance, I'm not managing to debug the code properly as mentioned above

I haven't looked at it yet as I am still in the process of rewriting the MachO linker. The good news though is, that it's going rather well. To showcase my progress, here's the test matrix I currently have: https://github.com/kubkon/zld/blob/7e5826a07788186541959c5d7e06df7dd9d144fb/test/macho.zig#L22 Standalone zld MachO linker currently passes all tests, but it's restricted to x86_64 currently as I haven't started implementing aarch64 yet.

dmorn commented 11 months ago

@kubkon great! 🤩 I'll keen an eye on it. Side question: do you have any resource or hint for debugging such things? Why lldb exits instead of breaking when I try to debug the above error? Why is the breakpoint not triggered? I have some experience in debugging C code, this never happed. lldb finds and attaches the breakpoint successfully btw!

kubkon commented 11 months ago

@kubkon great! 🤩 I'll keen an eye on it. Side question: do you have any resource or hint for debugging such things? Why lldb exits instead of breaking when I try to debug the above error? Why is the breakpoint not triggered? I have some experience in debugging C code, this never happed. lldb finds and attaches the breakpoint successfully btw!

I cannot repro your issue with latest Zig master (I have checked out your branch). I actually get undefined symbols error:

$ zig build
zig build-exe show_metadata Debug native: error: the following command failed with 14 compilation errors:
/Users/kubkon/dev/zig/build/stage4/bin/zig build-exe /Users/kubkon/dev/ffmpeg.zig-dmorn/doc/examples/show_metadata.c /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/930ef54bd21a57e2889c476df53bf586/libz.a /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/f9b81991c3fa7ee164949117fe43bc2a/libmp3lame.a /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/1fe9c61c0949b6dcc8e1d73c4e4bc9c5/libvorbis.a /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/dee438830bbabc0742a290627bfcc241/libogg.a -lc --cache-dir /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache --global-cache-dir /Users/kubkon/.cache/zig --name show_metadata -I /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-out/include --listen=-
Build Summary: 151/154 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install show_metadata transitive failure
   └─ zig build-exe show_metadata Debug native 14 errors
error: undefined reference to symbol _ff_vp9_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_hevc_v4l2m2m_encoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_vp8_v4l2m2m_encoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_mpeg4_v4l2m2m_encoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_h264_v4l2m2m_encoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_h263_v4l2m2m_encoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_mpeg2_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_mpeg1_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_hevc_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_vp8_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_mpeg4_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_h264_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_h263_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
error: undefined reference to symbol _ff_vc1_v4l2m2m_decoder
    note: referenced in /Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/11f4937b6e4b5b5361577c2442d1d980/libffmpeg.a(/Users/kubkon/dev/ffmpeg.zig-dmorn/zig-cache/o/0a6caa78917ec0ada2ae7e5234ecf8ed/allcodecs.o)
dmorn commented 11 months ago

Hi @kubkon, I synced the fork but now the breaking changes hit (I'm on 0.12.0-dev.1832+f545fc28b)

/Users/dmorn/projects/ffmpeg.zig/build.zig:6:22: error: no field named 'result' in struct 'zig.CrossTarget'
    const t = target.result;
                     ^~~~~~
/Users/dmorn/projects/zig/build/stage3/lib/zig/std/zig/CrossTarget.zig:1:1: note: struct declared here
//! Contains all the same data as `Target`, additionally introducing the concept of "the native target".
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
referenced by:
    runBuild__anon_6390: /Users/dmorn/projects/zig/build/stage3/lib/zig/std/Build.zig:1882:27
    steps__anon_6201: /Users/dmorn/projects/zig/build/stage3/lib/zig/build_runner.zig:969:29
    remaining reference traces hidden; use '-freference-trace' to see all reference traces
andrewrk commented 7 months ago

Closing since there has been no activity for months and it is still WIP.