ziglang / zig

General-purpose programming language and toolchain for maintaining robust, optimal, and reusable software.
https://ziglang.org
MIT License
33.58k stars 2.45k forks source link

release fast doesn't work in complex project: `ld.lld: error: Invalid value reference from metadata` `Global is external, but doesn't have external or weak linkage!` #13349

Open IntegratedQuantum opened 1 year ago

IntegratedQuantum commented 1 year ago

Zig Version

0.10.0-dev.4640+20925b2f5

Steps to Reproduce and Observed Behavior

I tried to reduce this error, but I was unable to do that. Whenever I remove too many lines of code the error transforms into #13348. I think the error requires a certain function(+inlined calls) size. Maybe llvm tries to make the function external if it gets too big?

Anyways because I wasn't able to reduce it, here is a link to the exact commit that I tested it with: https://github.com/PixelGuys/Cubyz/commit/d4e9fa2c441bdb146388770d04fde6be74ff350f

This is the output that I get:

LLD Link... error(link): /home/mint/Downloads/zig/zig terminated with stderr:
ld.lld: error: Invalid value reference from metadata
Global is external, but doesn't have external or weak linkage!
ptr @renderer.render
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20404"
Global is external, but doesn't have external or weak linkage!
ptr @Thread.Mutex.FutexImpl.lockSlow
Global is external, but doesn't have external or weak linkage!
ptr @network.Connection.sendUnimportant
Global is external, but doesn't have external or weak linkage!
ptr @main.Window.toggleFullscreen
Global is external, but doesn't have external or weak linkage!
ptr @fs.file.File.write
Global is external, but doesn't have external or weak linkage!
ptr @fmt.formatBuf__anon_26731
Global is external, but doesn't have external or weak linkage!
ptr @network.Connection.disconnect
Global is external, but doesn't have external or weak linkage!
ptr @Thread.Condition.FutexImpl.wait
Global is external, but doesn't have external or weak linkage!
ptr @"heap.general_purpose_allocator.GeneralPurposeAllocator(.{.stack_trace_frames = 0, .enable_memory_limit = false, .safety = false, .thread_safe = true, .MutexType = null, .never_unmap = false, .retain_metadata = false, .verbose_log = false}).freeLarge"
Global is external, but doesn't have external or weak linkage!
ptr @graphics.TextureArray.generate
Global is external, but doesn't have external or weak linkage!
ptr @"mem.SplitIterator(u8).first"
Global is external, but doesn't have external or weak linkage!
ptr @"hash_map.HashMap([]const u8,json.JsonElement,hash_map.StringContext,80).put"
Global is external, but doesn't have external or weak linkage!
ptr @json.JsonElement.free
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.reallocAdvancedWithRetAddr__anon_22386
Global is external, but doesn't have external or weak linkage!
ptr @json.JsonElement.recurseToString__anon_19352
Global is external, but doesn't have external or weak linkage!
ptr @network.Connection.sendImportant
Global is external, but doesn't have external or weak linkage!
ptr @Thread.PosixThreadImpl.spawn__anon_20992.Instance.entryFn
Global is external, but doesn't have external or weak linkage!
ptr @crypto.xoodoo.State.permute
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18322.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20929"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18332.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20930"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18336.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20931"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18340.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20932"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18345.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20933"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18349.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20934"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18355.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20935"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18359.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20936"
Global is external, but doesn't have external or weak linkage!
ptr @network.Protocols.Protocols__struct_18363.receive
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20937"
Global is external, but doesn't have external or weak linkage!
ptr @renderer.buffers.updateBufferSize
Global is external, but doesn't have external or weak linkage!
ptr @"array_list.ArrayListAligned(u32,null).ensureTotalCapacityPrecise"
Global is external, but doesn't have external or weak linkage!
ptr @fs.Dir.openDirFlagsZ
Global is external, but doesn't have external or weak linkage!
ptr @fs.IterableDir.Iterator__struct_11199.next
Global is external, but doesn't have external or weak linkage!
ptr @"array_list.ArrayListAligned([]const u8,null).append"
Global is external, but doesn't have external or weak linkage!
ptr @assets.readAllJsonFilesInAddons
Global is external, but doesn't have external or weak linkage!
ptr @"array_list.ArrayListAligned([]const u8,null).deinit"
Global is external, but doesn't have external or weak linkage!
ptr @"array_list.ArrayListAligned(fs.Dir,null).deinit"
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init.gen.allocImpl
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init__anon_20694.gen.resizeImpl
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init.gen.freeImpl
Global is external, but doesn't have external or weak linkage!
ptr @heap.PageAllocator.alloc
Global is external, but doesn't have external or weak linkage!
ptr @heap.PageAllocator.resize
Global is external, but doesn't have external or weak linkage!
ptr @heap.PageAllocator.free
Global is external, but doesn't have external or weak linkage!
ptr @graphics.Shader.addShader
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_17538"
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init__anon_20485.gen.freeImpl
Global is external, but doesn't have external or weak linkage!
ptr @"log.scoped(.default).err__anon_20631"
Global is external, but doesn't have external or weak linkage!
ptr @debug.panicExtra__anon_20484
Global is external, but doesn't have external or weak linkage!
ptr @unicode.utf8CountCodepoints
Global is external, but doesn't have external or weak linkage!
ptr @"io.writer.Writer(*io.fixed_buffer_stream.FixedBufferStream([]u8),error{NoSpaceLeft},(function 'write')).writeByteNTimes"
Global is external, but doesn't have external or weak linkage!
ptr @"hash_map.HashMapUnmanaged(usize,heap.general_purpose_allocator.GeneralPurposeAllocator(.{.stack_trace_frames = 0, .enable_memory_limit = false, .safety = false, .thread_safe = true, .MutexType = null, .never_unmap = false, .retain_metadata = false, .verbose_log = false}).LargeAlloc,hash_map.AutoContext(usize),80).grow"
Global is external, but doesn't have external or weak linkage!
ptr @"heap.general_purpose_allocator.GeneralPurposeAllocator(.{.stack_trace_frames = 0, .enable_memory_limit = false, .safety = false, .thread_safe = true, .MutexType = null, .never_unmap = false, .retain_metadata = false, .verbose_log = false}).createBucket"
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init__anon_20485.gen.resizeImpl
Global is external, but doesn't have external or weak linkage!
ptr @os.openatZ
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init__anon_20470.gen.allocImpl
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init__anon_20470.gen.resizeImpl
Global is external, but doesn't have external or weak linkage!
ptr @mem.Allocator.init__anon_20470.gen.freeImpl
LLVM ERROR: Broken module found, compilation aborted!

error: LLDCrashed
error: Cubyzig...
error: The following command exited with error code 1:
/home/mint/Downloads/zig/zig build-exe /home/mint/Desktop/Cubyzig/src/main.zig -lc -cflags -g -std=c99 -D_GLFW_X11 -- /home/mint/Desktop/Cubyzig/lib/glfw/src/linux_joystick.c /home/mint/Desktop/Cubyzig/lib/glfw/src/x11_init.c /home/mint/Desktop/Cubyzig/lib/glfw/src/x11_monitor.c /home/mint/Desktop/Cubyzig/lib/glfw/src/x11_window.c /home/mint/Desktop/Cubyzig/lib/glfw/src/xkb_unicode.c /home/mint/Desktop/Cubyzig/lib/glfw/src/posix_time.c /home/mint/Desktop/Cubyzig/lib/glfw/src/posix_thread.c /home/mint/Desktop/Cubyzig/lib/glfw/src/glx_context.c /home/mint/Desktop/Cubyzig/lib/glfw/src/egl_context.c /home/mint/Desktop/Cubyzig/lib/glfw/src/osmesa_context.c /home/mint/Desktop/Cubyzig/lib/glfw/src/context.c /home/mint/Desktop/Cubyzig/lib/glfw/src/init.c /home/mint/Desktop/Cubyzig/lib/glfw/src/input.c /home/mint/Desktop/Cubyzig/lib/glfw/src/monitor.c /home/mint/Desktop/Cubyzig/lib/glfw/src/vulkan.c /home/mint/Desktop/Cubyzig/lib/glfw/src/window.c -lX11 -lGL -cflags -g -- /home/mint/Desktop/Cubyzig/lib/glad.c /home/mint/Desktop/Cubyzig/lib/stb_image.c /home/mint/Desktop/Cubyzig/lib/cross_platform_udp_socket.c -OReleaseFast --cache-dir /home/mint/Desktop/Cubyzig/zig-cache --global-cache-dir /home/mint/.cache/zig --name Cubyzig -I /home/mint/Desktop/Cubyzig/include --enable-cache 
error: the following build command failed with exit code 1:
/home/mint/Desktop/Cubyzig/zig-cache/o/8319b4f5701387a4e13ff51606d15bee/build /home/mint/Downloads/zig/zig /home/mint/Desktop/Cubyzig /home/mint/Desktop/Cubyzig/zig-cache /home/mint/.cache/zig run -Drelease-fast=true

Expected Behavior

It should compile without an error like in release-small and release-safe and debug.

andrewrk commented 1 year ago

As a workaround, you might try disabling LTO.

IntegratedQuantum commented 1 year ago

It compiles. However it doesn't run correctly. It's outputting wrong data to the network which in result leads to a crash.

I managed to find the relevant code. It appears to be the same piece of code as in #13348, so it's likely the same issue:

pub fn sendRequest(conn: *Connection, requests: []chunk.ChunkPosition) !void {
    if(requests.len == 0) return;
    var data = try main.threadAllocator.alloc(u8, 16*requests.len);
    defer main.threadAllocator.free(data);
    var remaining = data;
    for(requests) |req| {
        std.mem.writeIntBig(chunk.ChunkCoordinate, remaining[0..4], req.wx);
        std.mem.writeIntBig(chunk.ChunkCoordinate, remaining[4..8], req.wy);
        std.mem.writeIntBig(chunk.ChunkCoordinate, remaining[8..12], req.wz);
        std.mem.writeIntBig(chunk.ChunkCoordinate, remaining[12..16], req.voxelSize); // ← this line
        remaining = remaining[16..];
    }
    try conn.sendImportant(id, data);
}

req.voxelSize is a u31 and it only ever has the values {1, 2, 4, 8, 16, 32} chunk.ChunkCoordinate is i32 Here a sample of the data it actually produced:

      wx, wy, wz are equivalent to a debug build       all of these should be 0, 0, 0, 1
                            /\                                   /\
/¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯\ /¯¯¯¯¯¯¯¯¯\
255, 255, 255,  96, 255, 255, 255, 224, 255, 255, 255, 224,  0, 0, 0, 1,
255, 255, 255,  96, 255, 255, 255, 224,   0,   0,   0,   0,  0, 0, 0, 2,
255, 255, 255,  96,   0,   0,   0,   0, 255, 255, 255, 224,  0, 0, 0, 4,
255, 255, 255,  96,   0,   0,   0,   0,   0,   0,   0,   0,  0, 0, 0, 8,
255, 255, 255, 128, 255, 255, 255, 160, 255, 255, 255, 192,  0, 0, 0, 16,
255, 255, 255, 128, 255, 255, 255, 160, 255, 255, 255, 224,  0, 0, 0, 32,
255, 255, 255, 128, 255, 255, 255, 160,   0,   0,   0,   0,  0, 0, 0, 64,
255, 255, 255, 128, 255, 255, 255, 160,   0,   0,   0,  32,  0, 0, 0, 128,
255, 255, 255, 128, 255, 255, 255, 224, 255, 255, 255, 224,  0, 0, 0, 1

This pattern continues, cycling through {1, 2, 4, 8, 16, 32, 64, 128}

If I try to print the req.voxelSize directly, it starts working. So I'm guessing this is some issue in the optimizer.

IntegratedQuantum commented 1 year ago

I just updated to 0.11.0-dev.368+1829b6eab and the error seems to be resolved.

Vexu commented 1 year ago

I'm able to reproduce this (or something similar) with https://github.com/Vexu/bun/commit/94d54c3fd1743b8851aa1873aabacf0ce8064ff0 and running make headers.

Vexu commented 1 year ago

Reduction:

fn FnPtrStage2Wrap(comptime T: anytype) @TypeOf(&struct {
    const FnPtrType = T;
    pub const fn_ptr: FnPtrType = undefined;
}.fn_ptr) {
    return &struct {
        const FnPtrType = T;
        pub const fn_ptr: FnPtrType = undefined;
    }.fn_ptr;
}
fn foo() u32 {
    return 0;
}
test {
    const a: @TypeOf(FnPtrStage2Wrap(@TypeOf(foo))) = foo;
    _ = a();
}