terralang / terra

Terra is a low-level system programming language that is embedded in and meta-programmed by the Lua programming language.
terralang.org
Other
2.71k stars 197 forks source link

macOS arm64 test failure: vararg.t #605

Closed shoe42 closed 2 years ago

shoe42 commented 2 years ago

The vararg.t test fails on macOS arm64 (M1). macOS 12.5, using system clang and Homebrew LLVM 13. Trimmed output:

$ ../build/bin/terra vararg.t
Assertion failed: (Ty && "Invalid GetElementPtrInst indices for type!"), function checkGEPType, file Instructions.h, line 921.
zsh: abort      ../build/bin/terra vararg.t
elliottslaughter commented 2 years ago

For this one it would be helpful to get a full backtrace. Make sure to compile Terra in debug mode (-DCMAKE_BUILD_TYPE=Debug if you're using CMake, and also make sure to use a fresh build directory).

shoe42 commented 2 years ago

Apologies, that's all the output from Terra itself - from a debugger, there's:

* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #0: 0x00000001a314ad98 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001a317fee0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001a30ba340 libsystem_c.dylib`abort + 168
    frame #3: 0x00000001a30b9754 libsystem_c.dylib`__assert_rtn + 272
  * frame #4: 0x0000000100038cc4 terra`llvm::checkGEPType(Ty=0x0000000000000000) at Instructions.h:921:3
    frame #5: 0x0000000100038b10 terra`llvm::GetElementPtrInst::getGEPReturnType(ElTy=0x00000001098ec400, Ptr=0x0000600002908100, IdxList=ArrayRef<llvm::Value *> @ 0x000000016fdfac58) at Instructions.h:1096:26
    frame #6: 0x0000000100038970 terra`llvm::GetElementPtrInst::GetElementPtrInst(this=0x0000600003508110, PointeeType=0x00000001098ec400, Ptr=0x0000600002908100, IdxList=ArrayRef<llvm::Value *> @ 0x000000016fdfad38, Values=3, NameStr=0x000000016fdfae40, InsertBefore=0x0000000000000000) at Instructions.h:1169:19
    frame #7: 0x00000001000388e8 terra`llvm::GetElementPtrInst::GetElementPtrInst(this=0x0000600003508110, PointeeType=0x00000001098ec400, Ptr=0x0000600002908100, IdxList=ArrayRef<llvm::Value *> @ 0x000000016fdfada0, Values=3, NameStr=0x000000016fdfae40, InsertBefore=0x0000000000000000) at Instructions.h:1173:63
    frame #8: 0x00000001000387ac terra`llvm::GetElementPtrInst::Create(PointeeType=0x00000001098ec400, Ptr=0x0000600002908100, IdxList=ArrayRef<llvm::Value *> @ 0x000000016fdfae20, NameStr=0x000000016fdfae40, InsertBefore=0x0000000000000000) at Instructions.h:962:25
    frame #9: 0x000000010032a7dc terra`llvm::IRBuilderBase::CreateConstInBoundsGEP2_32(llvm::Type*, llvm::Value*, unsigned int, unsigned int, llvm::Twine const&) + 164
    frame #10: 0x00000001023b32e8 terra`clang::CodeGen::CGBuilderTy::CreateStructGEP(clang::CodeGen::Address, unsigned int, llvm::Twine const&) + 112
    frame #11: 0x0000000102615e40 terra`(anonymous namespace)::AArch64ABIInfo::EmitVAArg(clang::CodeGen::CodeGenFunction&, clang::CodeGen::Address, clang::QualType) const + 744
    frame #12: 0x00000001024bd224 terra`clang::StmtVisitorBase<std::__1::add_pointer, (anonymous namespace)::ScalarExprEmitter, llvm::Value*>::Visit(clang::Stmt*) + 8812
    frame #13: 0x00000001024b6034 terra`(anonymous namespace)::ScalarExprEmitter::Visit(clang::Expr*) + 52
    frame #14: 0x00000001024bfa3c terra`(anonymous namespace)::ScalarExprEmitter::EmitBinOps(clang::BinaryOperator const*) + 52
    frame #15: 0x00000001024bfe78 terra`(anonymous namespace)::ScalarExprEmitter::EmitCompare(clang::BinaryOperator const*, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, bool) + 896
    frame #16: 0x00000001024b6034 terra`(anonymous namespace)::ScalarExprEmitter::Visit(clang::Expr*) + 52
    frame #17: 0x00000001024b5ff4 terra`clang::CodeGen::CodeGenFunction::EmitScalarExpr(clang::Expr const*, bool) + 44
    frame #18: 0x000000010248a738 terra`clang::CodeGen::CodeGenFunction::EvaluateExprAsBool(clang::Expr const*) + 288
    frame #19: 0x0000000102598bb0 terra`clang::CodeGen::CodeGenFunction::EmitBranchOnBoolExpr(clang::Expr const*, llvm::BasicBlock*, llvm::BasicBlock*, unsigned long long, clang::Stmt::Likelihood) + 696
    frame #20: 0x000000010255c808 terra`clang::CodeGen::CodeGenFunction::EmitIfStmt(clang::IfStmt const&) + 748
    frame #21: 0x0000000102562bd0 terra`clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope(clang::CompoundStmt const&, bool, clang::CodeGen::AggValueSlot) + 316
    frame #22: 0x0000000102597920 terra`clang::CodeGen::CodeGenFunction::EmitFunctionBody(clang::Stmt const*) + 100
    frame #23: 0x0000000102597f30 terra`clang::CodeGen::CodeGenFunction::GenerateCode(clang::GlobalDecl, llvm::Function*, clang::CodeGen::CGFunctionInfo const&) + 704
    frame #24: 0x00000001025ab838 terra`clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(clang::GlobalDecl, llvm::GlobalValue*) + 308
    frame #25: 0x00000001025a723c terra`clang::CodeGen::CodeGenModule::EmitGlobalDefinition(clang::GlobalDecl, llvm::GlobalValue*) + 260
    frame #26: 0x00000001025ad8bc terra`clang::CodeGen::CodeGenModule::EmitTopLevelDecl(clang::Decl*) + 240
    frame #27: 0x00000001025ff8b0 terra`(anonymous namespace)::CodeGeneratorImpl::HandleTopLevelDecl(clang::DeclGroupRef) + 140
    frame #28: 0x00000001000743d8 terra`CodeGenProxy::HandleTopLevelDecl(this=0x0000600003900dd0, D=DeclGroupRef @ 0x000000016fdfe678) at tcwrapper.cpp:592:20
    frame #29: 0x000000010262f314 terra`clang::ParseAST(clang::Sema&, bool, bool) + 392
    frame #30: 0x0000000100068d88 terra`dofile(T=0x0000000109569d28, TT=0x0000600002c00080, code="#include <stdio.h>\n#include <string.h>\n#include <stdarg.h>\n\nint checkargs(const char* f, va_list vl)\n{\n  if(strcmp(f, \"%i%i%f%s\"))\n    return 1;\n    \n  if(va_arg(vl, int) != -10)\n    return 2;\n  if(va_arg(vl, unsigned long long) != 60000000000001ULL)\n    return 3;\n  if(va_arg(vl, double) != 60.01)\n    return 4;\n  if(strcmp(va_arg(vl, const char*), \"TesT\"))\n    return 5;\n  return 0;\n}\n\n", args=size=13, result=0x000000016fdfeba8) at tcwrapper.cpp:1045:5
    frame #31: 0x0000000100068914 terra`include_c(L=0x0000000109560380) at tcwrapper.cpp:1129:9
    frame #32: 0x0000000103070328 terra`lj_BC_FUNCC + 44
    frame #33: 0x0000000103017d74 terra`lua_pcall(L=<unavailable>, nargs=<unavailable>, nresults=<unavailable>, errfunc=<unavailable>) at lj_api.c:1145:12 [opt]
    frame #34: 0x000000010000517c terra`docall(L=0x0000000109560380, narg=0, clear=0) at main.cpp:339:14
    frame #35: 0x0000000100004d24 terra`main(argc=2, argv=0x000000016fdfef48) at main.cpp:119:13
    frame #36: 0x000000010936108c dyld`start + 520
elliottslaughter commented 2 years ago

Based on this backtrace, it sounds like the following Terra file should be sufficient to make this fail:

bug605.t:

C = terralib.includecstring [[
#include <stdio.h>
#include <string.h>
#include <stdarg.h>

int checkargs(const char* f, va_list vl)
{
  if(strcmp(f, "%i%i%f%s"))
    return 1;

  if(va_arg(vl, int) != -10)
    return 2;
  if(va_arg(vl, unsigned long long) != 60000000000001ULL)
    return 3;
  if(va_arg(vl, double) != 60.01)
    return 4;
  if(strcmp(va_arg(vl, const char*), "TesT"))
    return 5;
  return 0;
}

]]

Can you confirm?

shoe42 commented 2 years ago

Yup, looks like that triggers it too:

Assertion failed: (Ty && "Invalid GetElementPtrInst indices for type!"), function checkGEPType, file Instructions.h, line 921.
elliottslaughter commented 2 years ago

I suppose the next test is just to extract that C snippet out and see if we hit the same error in Clang:

bug605.c:

#include <stdio.h>
#include <string.h>
#include <stdarg.h>

int checkargs(const char* f, va_list vl)
{
  if(strcmp(f, "%i%i%f%s"))
    return 1;

  if(va_arg(vl, int) != -10)
    return 2;
  if(va_arg(vl, unsigned long long) != 60000000000001ULL)
    return 3;
  if(va_arg(vl, double) != 60.01)
    return 4;
  if(strcmp(va_arg(vl, const char*), "TesT"))
    return 5;
  return 0;
}
clang bug605.c -o bug605.o -c -Wall

And ideally run this with the same Clang you used for building Terra.

shoe42 commented 2 years ago

Clang seems to be able to build that fine. Will see if I can try get any info from inside the debugger at some point.

elliottslaughter commented 2 years ago

Since Clang compiles it fine, could you try:

clang bug605.c -o bug605.ll -S -emit-llvm -Wall

And attach that? It should provide us information on what Clang is doing.

After that we'd need to figure out the equivalent way to get information out of Terra, which will be more tricky because it's failing before it can even construct the IR.

But I suspect what's going on (and probably in #603 too) is that we're somehow configuring LLVM differently. Maybe the target machines are different or not entirely compatible, or something like that.

elliottslaughter commented 2 years ago

Here's Terra file you can run to at least give us the target and data layout for Terra:

terra f() return 123 end
print(terralib.saveobj(nil, "llvmir", {f=f}, nil, nil, false))
shoe42 commented 2 years ago

IR for bug605.c:

; ModuleID = 'bug605.c'
source_filename = "bug605.c"
target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
target triple = "arm64-apple-macosx12.0.0"

@.str = private unnamed_addr constant [9 x i8] c"%i%i%f%s\00", align 1
@.str.1 = private unnamed_addr constant [5 x i8] c"TesT\00", align 1

; Function Attrs: noinline nounwind optnone ssp uwtable
define i32 @checkargs(i8* %0, i8* %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i8*, align 8
  %5 = alloca i8*, align 8
  %6 = alloca i32, align 4
  %7 = alloca i64, align 8
  %8 = alloca double, align 8
  %9 = alloca i8*, align 8
  store i8* %0, i8** %4, align 8
  store i8* %1, i8** %5, align 8
  %10 = load i8*, i8** %4, align 8
  %11 = call i32 @strcmp(i8* %10, i8* getelementptr inbounds ([9 x i8], [9 x i8]* @.str, i64 0, i64 0))
  %12 = icmp ne i32 %11, 0
  br i1 %12, label %13, label %14

13:                                               ; preds = %2
  store i32 1, i32* %3, align 4
  br label %36

14:                                               ; preds = %2
  %15 = va_arg i8** %5, i32
  store i32 %15, i32* %6, align 4
  %16 = load i32, i32* %6, align 4
  %17 = icmp ne i32 %16, -10
  br i1 %17, label %18, label %19

18:                                               ; preds = %14
  store i32 2, i32* %3, align 4
  br label %36

19:                                               ; preds = %14
  %20 = va_arg i8** %5, i64
  store i64 %20, i64* %7, align 8
  %21 = load i64, i64* %7, align 8
  %22 = icmp ne i64 %21, 60000000000001
  br i1 %22, label %23, label %24

23:                                               ; preds = %19
  store i32 3, i32* %3, align 4
  br label %36

24:                                               ; preds = %19
  %25 = va_arg i8** %5, double
  store double %25, double* %8, align 8
  %26 = load double, double* %8, align 8
  %27 = fcmp une double %26, 0x404E0147AE147AE1
  br i1 %27, label %28, label %29

28:                                               ; preds = %24
  store i32 4, i32* %3, align 4
  br label %36

29:                                               ; preds = %24
  %30 = va_arg i8** %5, i8*
  store i8* %30, i8** %9, align 8
  %31 = load i8*, i8** %9, align 8
  %32 = call i32 @strcmp(i8* %31, i8* getelementptr inbounds ([5 x i8], [5 x i8]* @.str.1, i64 0, i64 0))
  %33 = icmp ne i32 %32, 0
  br i1 %33, label %34, label %35

34:                                               ; preds = %29
  store i32 5, i32* %3, align 4
  br label %36

35:                                               ; preds = %29
  store i32 0, i32* %3, align 4
  br label %36

36:                                               ; preds = %35, %34, %28, %23, %18, %13
  %37 = load i32, i32* %3, align 4
  ret i32 %37
}

declare i32 @strcmp(i8*, i8*) #1

attributes #0 = { noinline nounwind optnone ssp uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "probe-stack"="__chkstk_darwin" "stack-protector-buffer-size"="8" "target-cpu"="apple-m1" "target-features"="+aes,+crc,+crypto,+dotprod,+fp-armv8,+fp16fml,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+sha3,+sm4,+v8.5a,+zcm,+zcz" }
attributes #1 = { "frame-pointer"="non-leaf" "no-trapping-math"="true" "probe-stack"="__chkstk_darwin" "stack-protector-buffer-size"="8" "target-cpu"="apple-m1" "target-features"="+aes,+crc,+crypto,+dotprod,+fp-armv8,+fp16fml,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+sha3,+sm4,+v8.5a,+zcm,+zcz" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 2, !"SDK Version", [2 x i32] [i32 12, i32 3]}
!1 = !{i32 1, !"wchar_size", i32 4}
!2 = !{i32 1, !"branch-target-enforcement", i32 0}
!3 = !{i32 1, !"sign-return-address", i32 0}
!4 = !{i32 1, !"sign-return-address-all", i32 0}
!5 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!6 = !{i32 7, !"PIC Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Apple clang version 13.1.6 (clang-1316.0.21.2.5)"}

And for the latter:

; ModuleID = 'terra'
source_filename = "terra"
target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
target triple = "arm64-apple-darwin21.6.0"

define dso_local i32 @f() {
entry:
  ret i32 123

dead:                                             ; No predecessors!
  ret i32 undef
}

Looks like the data layouts are the same.

elliottslaughter commented 2 years ago

Here are the differences I see:

The data layouts do indeed look the same.

I'll make a patch to try to force the OS to be the same. If that doesn't work then unfortunately I think this is going to require digging deeper into Clang and differences in how we initialize it from Terra. (Which I can't do without access to an M1 machine, and probably will be hard unless you know Terra and Clang well.)

shoe42 commented 2 years ago

Running it with self-compiled Clang/LLVM (built in Debug mode) gives a slight different error:

Assertion failed: (isa<X>(Val) && "cast<Ty>() argument of incompatible type!"), function cast, file Casting.h, line 269.

With the backtrace being:

* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #0: 0x00000001a314ad98 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001a317fee0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001a30ba340 libsystem_c.dylib`abort + 168
    frame #3: 0x00000001a30b9754 libsystem_c.dylib`__assert_rtn + 272
  * frame #4: 0x0000000100015814 terra`llvm::cast_retty<llvm::StructType, llvm::Type*>::ret_type llvm::cast<llvm::StructType, llvm::Type>(Val=0x000000012d0c3400) at Casting.h:269:3
    frame #5: 0x0000000107c6ecf4 terra`clang::CodeGen::CGBuilderTy::CreateStructGEP(this=0x000000016fdfb2a8, Addr=Address @ 0x000000016fdf8a30, Index=3, Name=0x000000016fdf9608) at CGBuilder.h:190:30
    frame #6: 0x00000001084e7798 terra`(anonymous namespace)::AArch64ABIInfo::EmitAAPCSVAArg(this=0x00006000002073e0, VAListAddr=Address @ 0x000000016fdf97c0, Ty=QualType @ 0x000000016fdf97b8, CGF=0x000000016fdfb1a0) const at TargetInfo.cpp:5933:30
    frame #7: 0x00000001084e5764 terra`(anonymous namespace)::AArch64ABIInfo::EmitVAArg(this=0x00006000002073e0, CGF=0x000000016fdfb1a0, VAListAddr=Address @ 0x000000016fdf98a0, Ty=QualType @ 0x000000016fdf9898) const at TargetInfo.cpp:5458:44
    frame #8: 0x0000000107e3e384 terra`clang::CodeGen::CodeGenFunction::EmitVAArg(this=0x000000016fdfb1a0, VE=0x000000012c88e460, VAListAddr=0x000000016fdf9a78) at CGCall.cpp:5522:38
    frame #9: 0x0000000107fad28c terra`(anonymous namespace)::ScalarExprEmitter::VisitVAArgExpr(this=0x000000016fdfa3f0, VE=0x000000012c88e460) at CGExprScalar.cpp:4666:24
    frame #10: 0x0000000107fa4c28 terra`clang::StmtVisitorBase<std::__1::add_pointer, (anonymous namespace)::ScalarExprEmitter, llvm::Value*>::Visit(this=0x000000016fdfa3f0, S=0x000000012c88e460) at StmtNodes.inc:1435:1
    frame #11: 0x0000000107f9bf4c terra`(anonymous namespace)::ScalarExprEmitter::Visit(this=0x000000016fdfa3f0, E=0x000000012c88e460) at CGExprScalar.cpp:412:52
    frame #12: 0x0000000107fad8f0 terra`(anonymous namespace)::ScalarExprEmitter::EmitBinOps(this=0x000000016fdfa3f0, E=0x000000012c88e4c0) at CGExprScalar.cpp:2993:16
    frame #13: 0x0000000107fadc88 terra`(anonymous namespace)::ScalarExprEmitter::EmitCompare(this=0x000000016fdfa3f0, E=0x000000012c88e4c0, UICmpOpc=ICMP_NE, SICmpOpc=ICMP_NE, FCmpOpc=FCMP_UNE, IsSignaling=false) at CGExprScalar.cpp:4026:24
    frame #14: 0x0000000107fa543c terra`(anonymous namespace)::ScalarExprEmitter::VisitBinNE(this=0x000000016fdfa3f0, E=0x000000012c88e4c0) at CGExprScalar.cpp:833:3
    frame #15: 0x0000000107fa37fc terra`clang::StmtVisitorBase<std::__1::add_pointer, (anonymous namespace)::ScalarExprEmitter, llvm::Value*>::Visit(this=0x000000016fdfa3f0, S=0x000000012c88e4c0) at StmtVisitor.h:64:26
    frame #16: 0x0000000107f9bf4c terra`(anonymous namespace)::ScalarExprEmitter::Visit(this=0x000000016fdfa3f0, E=0x000000012c88e4c0) at CGExprScalar.cpp:412:52
    frame #17: 0x0000000107f9bebc terra`clang::CodeGen::CodeGenFunction::EmitScalarExpr(this=0x000000016fdfb1a0, E=0x000000012c88e4c0, IgnoreResultAssign=false) at CGExprScalar.cpp:4811:8
    frame #18: 0x0000000107f15cd0 terra`clang::CodeGen::CodeGenFunction::EvaluateExprAsBool(this=0x000000016fdfb1a0, E=0x000000012c88e4c0) at CGExpr.cpp:194:33
    frame #19: 0x0000000108229b3c terra`clang::CodeGen::CodeGenFunction::EmitBranchOnBoolExpr(this=0x000000016fdfb1a0, Cond=0x000000012c88e4c0, TrueBlock=0x0000600001724180, FalseBlock=0x0000600001724200, TrueCount=0, LH=LH_None) at CodeGenFunction.cpp:1800:13
    frame #20: 0x000000010815994c terra`clang::CodeGen::CodeGenFunction::EmitIfStmt(this=0x000000016fdfb1a0, S=0x000000012c88e510) at CGStmt.cpp:761:3
    frame #21: 0x00000001081586fc terra`clang::CodeGen::CodeGenFunction::EmitStmt(this=0x000000016fdfb1a0, S=0x000000012c88e510, Attrs=ArrayRef<const clang::Attr *> @ 0x000000016fdfabe0) at CGStmt.cpp:147:32
    frame #22: 0x00000001081625c4 terra`clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope(this=0x000000016fdfb1a0, S=0x000000012c88e8f8, GetLast=false, AggSlot=AggValueSlot @ 0x000000016fdfada0) at CGStmt.cpp:496:7
    frame #23: 0x00000001082279c8 terra`clang::CodeGen::CodeGenFunction::EmitFunctionBody(this=0x000000016fdfb1a0, Body=0x000000012c88e8f8) at CodeGenFunction.cpp:1199:5
    frame #24: 0x000000010822834c terra`clang::CodeGen::CodeGenFunction::GenerateCode(this=0x000000016fdfb1a0, GD=GlobalDecl @ 0x000000016fdfaff8, Fn=0x0000600002c00208, FnInfo=0x0000600003300140) at CodeGenFunction.cpp:1373:5
    frame #25: 0x00000001082520c4 terra`clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(this=0x000000012d0c1a00, GD=GlobalDecl @ 0x000000016fdfb190, GV=0x0000600002c00208) at CodeGenModule.cpp:4861:26
    frame #26: 0x00000001082496f4 terra`clang::CodeGen::CodeGenModule::EmitGlobalDefinition(this=0x000000012d0c1a00, GD=GlobalDecl @ 0x000000016fdfc8a0, GV=0x0000000000000000) at CodeGenModule.cpp:3230:12
    frame #27: 0x000000010824ddbc terra`clang::CodeGen::CodeGenModule::EmitGlobal(this=0x000000012d0c1a00, GD=GlobalDecl @ 0x000000016fdfcaa0) at CodeGenModule.cpp:2982:5
    frame #28: 0x000000010825607c terra`clang::CodeGen::CodeGenModule::EmitTopLevelDecl(this=0x000000012d0c1a00, D=0x000000012c88e1e0) at CodeGenModule.cpp:5698:5
    frame #29: 0x000000010843c64c terra`(anonymous namespace)::CodeGeneratorImpl::HandleTopLevelDecl(this=0x000000012d0c1200, DG=DeclGroupRef @ 0x000000016fdfe590) at ModuleBuilder.cpp:170:18
    frame #30: 0x00000001000711e0 terra`CodeGenProxy::HandleTopLevelDecl(this=0x0000600003900680, D=DeclGroupRef @ 0x000000016fdfe5e8) at tcwrapper.cpp:592:20
    frame #31: 0x00000001085238e8 terra`clang::ParseAST(S=0x000000012d0c4400, PrintStats=false, SkipFunctionBodies=false) at ParseAST.cpp:162:31
    frame #32: 0x0000000100065d78 terra`dofile(T=0x000000012c305d28, TT=0x0000600002c08000, code="#include <stdio.h>\n#include <string.h>\n#include <stdarg.h>\n\nint checkargs(const char* f, va_list vl)\n{\n  if(strcmp(f, \"%i%i%f%s\"))\n    return 1;\n    \n  if(va_arg(vl, int) != -10)\n    return 2;\n  if(va_arg(vl, unsigned long long) != 60000000000001ULL)\n    return 3;\n  if(va_arg(vl, double) != 60.01)\n    return 4;\n  if(strcmp(va_arg(vl, const char*), \"TesT\"))\n    return 5;\n  return 0;\n}\n\n", args=size=13, result=0x000000016fdfeba8) at tcwrapper.cpp:1045:5
    frame #33: 0x0000000100065904 terra`include_c(L=0x000000012c2fc380) at tcwrapper.cpp:1129:9
    frame #34: 0x000000010a896d38 terra`lj_BC_FUNCC + 44
    frame #35: 0x000000010a83e78c terra`lua_pcall(L=<unavailable>, nargs=<unavailable>, nresults=<unavailable>, errfunc=<unavailable>) at lj_api.c:1145:12 [opt]
    frame #36: 0x000000010000216c terra`docall(L=0x000000012c2fc380, narg=0, clear=0) at main.cpp:339:14
    frame #37: 0x0000000100001d14 terra`main(argc=2, argv=0x000000016fdfef48) at main.cpp:119:13
    frame #38: 0x000000012c0fd08c dyld`start + 520
elliottslaughter commented 2 years ago

Here is a reproducer that can be used on an x86 Mac:

local target = terralib.newtarget {
  Triple = 'arm64-apple-darwin21.6.0',
}

C = terralib.includecstring([[
#include <stdio.h>
#include <string.h>
#include <stdarg.h>

int checkargs(const char* f, va_list vl)
{
  if(strcmp(f, "%i%i%f%s"))
    return 1;

  if(va_arg(vl, int) != -10)
    return 2;
  if(va_arg(vl, unsigned long long) != 60000000000001ULL)
    return 3;
  if(va_arg(vl, double) != 60.01)
    return 4;
  if(strcmp(va_arg(vl, const char*), "TesT"))
    return 5;
  return 0;
}

]], nil, target)

Result with LLVM 14:

Assertion failed: (Ty && "Invalid GetElementPtrInst indices for type!"), function checkGEPType, file Instructions.h, line 923.
Backtrace ``` * thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert frame #4: 0x000000010003acb7 terra`llvm::checkGEPType(llvm::Type*) + 87 terra`llvm::checkGEPType: -> 0x10003acb7 <+87>: jmp 0x10003acbc ; <+92> 0x10003acbc <+92>: movq -0x8(%rbp), %rax 0x10003acc0 <+96>: addq $0x10, %rsp 0x10003acc4 <+100>: popq %rbp Target 0: (terra) stopped. (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert frame #0: 0x00007ff80760000e libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007ff8076361ff libsystem_pthread.dylib`pthread_kill + 263 frame #2: 0x00007ff807581d24 libsystem_c.dylib`abort + 123 frame #3: 0x00007ff8075810cb libsystem_c.dylib`__assert_rtn + 314 * frame #4: 0x000000010003acb7 terra`llvm::checkGEPType(llvm::Type*) + 87 frame #5: 0x000000010003aacd terra`llvm::GetElementPtrInst::getGEPReturnType(llvm::Type*, llvm::Value*, llvm::ArrayRef) + 109 frame #6: 0x00000001003dca91 terra`llvm::IRBuilderBase::CreateConstInBoundsGEP2_32(llvm::Type*, llvm::Value*, unsigned int, unsigned int, llvm::Twine const&) + 193 frame #7: 0x0000000103ad39a5 terra`(anonymous namespace)::AArch64ABIInfo::EmitVAArg(clang::CodeGen::CodeGenFunction&, clang::CodeGen::Address, clang::QualType) const + 1221 frame #8: 0x00000001038b628b terra`clang::StmtVisitorBase::Visit(clang::Stmt*) + 9211 frame #9: 0x00000001038bccbf terra`(anonymous namespace)::ScalarExprEmitter::EmitBinOps(clang::BinaryOperator const*) + 63 frame #10: 0x00000001038bd414 terra`(anonymous namespace)::ScalarExprEmitter::EmitCompare(clang::BinaryOperator const*, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, bool) + 1540 frame #11: 0x00000001038ac330 terra`clang::CodeGen::CodeGenFunction::EmitScalarExpr(clang::Expr const*, bool) + 80 frame #12: 0x0000000103864da4 terra`clang::CodeGen::CodeGenFunction::EvaluateExprAsBool(clang::Expr const*) + 404 frame #13: 0x0000000103a110ee terra`clang::CodeGen::CodeGenFunction::EmitBranchOnBoolExpr(clang::Expr const*, llvm::BasicBlock*, llvm::BasicBlock*, unsigned long long, clang::Stmt::Likelihood) + 766 frame #14: 0x00000001039b0a30 terra`clang::CodeGen::CodeGenFunction::EmitIfStmt(clang::IfStmt const&) + 1312 frame #15: 0x00000001039af758 terra`clang::CodeGen::CodeGenFunction::EmitStmt(clang::Stmt const*, llvm::ArrayRef) + 744 frame #16: 0x00000001039bbc90 terra`clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope(clang::CompoundStmt const&, bool, clang::CodeGen::AggValueSlot) + 512 frame #17: 0x0000000103a0f4b5 terra`clang::CodeGen::CodeGenFunction::EmitFunctionBody(clang::Stmt const*) + 293 frame #18: 0x0000000103a1008f terra`clang::CodeGen::CodeGenFunction::GenerateCode(clang::GlobalDecl, llvm::Function*, clang::CodeGen::CGFunctionInfo const&) + 1487 frame #19: 0x0000000103a2c4cb terra`clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(clang::GlobalDecl, llvm::GlobalValue*) + 331 frame #20: 0x0000000103a257a6 terra`clang::CodeGen::CodeGenModule::EmitGlobalDefinition(clang::GlobalDecl, llvm::GlobalValue*) + 390 frame #21: 0x0000000103a2fffd terra`clang::CodeGen::CodeGenModule::EmitTopLevelDecl(clang::Decl*) + 2349 frame #22: 0x0000000103ab0e7f terra`(anonymous namespace)::CodeGeneratorImpl::HandleTopLevelDecl(clang::DeclGroupRef) + 143 frame #23: 0x00000001000740d2 terra`CodeGenProxy::HandleTopLevelDecl(clang::DeclGroupRef) + 130 frame #24: 0x0000000103afbf07 terra`clang::ParseAST(clang::Sema&, bool, bool) + 775 frame #25: 0x000000010006900f terra`dofile(terra_State*, TerraTarget*, char const*, std::__1::vector > const&, Obj*) + 895 frame #26: 0x0000000100068ab3 terra`include_c(lua_State*) + 675 frame #27: 0x0000000104abafb6 terra`lj_BC_FUNCC + 68 frame #28: 0x0000000104a56c8b terra`lua_pcall + 155 frame #29: 0x00000001000038ca terra`docall(lua_State*, int, int) + 122 frame #30: 0x00000001000034d1 terra`main + 401 frame #31: 0x0000000109ca152e dyld`start + 462 ```

I think this is basically the same symptom as on M1, which hopefully means I can debug this the rest of the way on my side.

elliottslaughter commented 2 years ago

A backtrace with line numbers:

Backtrace ``` * thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert frame #0: 0x00007ff80760000e libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007ff8076361ff libsystem_pthread.dylib`pthread_kill + 263 frame #2: 0x00007ff807581d24 libsystem_c.dylib`abort + 123 frame #3: 0x00007ff8075810cb libsystem_c.dylib`__assert_rtn + 314 * frame #4: 0x000000010003b977 terra`llvm::checkGEPType(Ty=0x0000000000000000) at Instructions.h:923:3 frame #5: 0x000000010003b78d terra`llvm::GetElementPtrInst::getGEPReturnType(ElTy=0x000000010d0ebe00, Ptr=0x0000600002904170, IdxList=ArrayRef @ 0x00007ff7bfefb708) at Instructions.h:1080:26 frame #6: 0x00000001003c3201 terra`llvm::IRBuilderBase::CreateConstInBoundsGEP2_32(llvm::Type*, llvm::Value*, unsigned int, unsigned int, llvm::Twine const&) [inlined] llvm::GetElementPtrInst::GetElementPtrInst(this=0x0000600003508110, PointeeType=0x000000010d0ebe00, Ptr=0x0000600002904170, IdxList=ArrayRef @ 0x0000600000d12530, Values=3, NameStr=0x00007ff7bfefb730, InsertBefore=0x0000000000000000) at Instructions.h:1153:19 [opt] frame #7: 0x00000001003c31ee terra`llvm::IRBuilderBase::CreateConstInBoundsGEP2_32(llvm::Type*, llvm::Value*, unsigned int, unsigned int, llvm::Twine const&) [inlined] llvm::GetElementPtrInst::GetElementPtrInst(this=0x0000600003508110, PointeeType=0x000000010d0ebe00, Ptr=0x0000600002904170, IdxList=ArrayRef @ 0x0000600000d12530, Values=3, NameStr=0x00007ff7bfefb730, InsertBefore=0x0000000000000000) at Instructions.h:1157:63 [opt] frame #8: 0x00000001003c31ee terra`llvm::IRBuilderBase::CreateConstInBoundsGEP2_32(llvm::Type*, llvm::Value*, unsigned int, unsigned int, llvm::Twine const&) [inlined] llvm::GetElementPtrInst::Create(PointeeType=0x000000010d0ebe00, Ptr=0x0000600002904170, IdxList=ArrayRef @ 0x0000600000d124c0, NameStr=0x00007ff7bfefb730, InsertBefore=0x0000000000000000) at Instructions.h:964:25 [opt] frame #9: 0x00000001003c31d8 terra`llvm::IRBuilderBase::CreateConstInBoundsGEP2_32(llvm::Type*, llvm::Value*, unsigned int, unsigned int, llvm::Twine const&) [inlined] llvm::GetElementPtrInst::CreateInBounds(PointeeType=0x000000010d0ebe00, Ptr=0x0000600002904170, IdxList=ArrayRef @ 0x0000600000d124c0, NameStr=0x00007ff7bfefb730, InsertBefore=0x0000000000000000) at Instructions.h:987:9 [opt] frame #10: 0x00000001003c31d8 terra`llvm::IRBuilderBase::CreateConstInBoundsGEP2_32(this=0x00007ff7bfefc318, Ty=0x000000010d0ebe00, Ptr=0x0000600002904170, Idx0=0, Idx1=3, Name=0x00007ff7bfefb8c0) at IRBuilder.h:1805:19 [opt] frame #11: 0x00000001038703cc terra`(anonymous namespace)::AArch64ABIInfo::EmitVAArg(clang::CodeGen::CodeGenFunction&, clang::CodeGen::Address, clang::QualType) const [inlined] llvm::IRBuilderBase::CreateStructGEP(this=0x00007ff7bfefc318, Ty=0x000000010d0ebe00, Ptr=, Idx=3, Name=0x00007ff7bfefb8c0) at IRBuilder.h:1856:12 [opt] frame #12: 0x00000001038703b9 terra`(anonymous namespace)::AArch64ABIInfo::EmitVAArg(clang::CodeGen::CodeGenFunction&, clang::CodeGen::Address, clang::QualType) const [inlined] clang::CodeGen::CGBuilderTy::CreateStructGEP(this=0x00007ff7bfefc318, Addr=Address @ 0x0000600000d42320, Index=3, Name=0x00007ff7bfefb8c0) at CGBuilder.h:199:20 [opt] frame #13: 0x0000000103870374 terra`(anonymous namespace)::AArch64ABIInfo::EmitVAArg(clang::CodeGen::CodeGenFunction&, clang::CodeGen::Address, clang::QualType) const [inlined] (anonymous namespace)::AArch64ABIInfo::EmitAAPCSVAArg(this=, VAListAddr=Address @ 0x0000600000d42320, Ty=, CGF=0x00007ff7bfefc208) const at TargetInfo.cpp:5988:30 [opt] frame #14: 0x0000000103870059 terra`(anonymous namespace)::AArch64ABIInfo::EmitVAArg(this=, CGF=, VAListAddr=Address @ 0x0000600000d7e560, Ty=QualType @ 0x00007ff7bfefb878) const at TargetInfo.cpp:5513:44 [opt] frame #15: 0x000000010366833e terra`clang::StmtVisitorBase::Visit(clang::Stmt*) [inlined] (anonymous namespace)::ScalarExprEmitter::VisitVAArgExpr(this=0x00007ff7bfefbce0, VE=0x000000010b86da70) at CGExprScalar.cpp:4726:24 [opt] frame #16: 0x00000001036682f3 terra`clang::StmtVisitorBase::Visit(this=0x00007ff7bfefbce0, S=0x000000010b86da70) at StmtNodes.inc:1455:1 [opt] frame #17: 0x000000010366e3ef terra`(anonymous namespace)::ScalarExprEmitter::EmitCompare(clang::BinaryOperator const*, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, bool) [inlined] (anonymous namespace)::ScalarExprEmitter::Visit(this=0x00007ff7bfefbce0, E=0x000000010b86da70) at CGExprScalar.cpp:412:52 [opt] frame #18: 0x000000010366e3cf terra`(anonymous namespace)::ScalarExprEmitter::EmitCompare(clang::BinaryOperator const*, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, llvm::CmpInst::Predicate, bool) [inlined] (anonymous namespace)::ScalarExprEmitter::EmitBinOps(this=0x00007ff7bfefbce0, E=0x000000010b86dad0) at CGExprScalar.cpp:3053:16 [opt] frame #19: 0x000000010366e3c2 terra`(anonymous namespace)::ScalarExprEmitter::EmitCompare(this=0x00007ff7bfefbce0, E=0x000000010b86dad0, UICmpOpc=ICMP_NE, SICmpOpc=ICMP_NE, FCmpOpc=, IsSignaling=false) at CGExprScalar.cpp:4086:24 [opt] frame #20: 0x00000001036661fd terra`clang::StmtVisitorBase::Visit(this=, S=) at StmtVisitor.h:0 [opt] [artificial] frame #21: 0x000000010365da80 terra`clang::CodeGen::CodeGenFunction::EmitScalarExpr(clang::Expr const*, bool) [inlined] (anonymous namespace)::ScalarExprEmitter::Visit(this=0x00007ff7bfefbce0, E=0x000000010b86dad0) at CGExprScalar.cpp:412:52 [opt] frame #22: 0x000000010365da69 terra`clang::CodeGen::CodeGenFunction::EmitScalarExpr(this=, E=0x000000010b86dad0, IgnoreResultAssign=) at CGExprScalar.cpp:4866:8 [opt] frame #23: 0x0000000103618c34 terra`clang::CodeGen::CodeGenFunction::EvaluateExprAsBool(this=0x00007ff7bfefc208, E=0x000000010b86dad0) at CGExpr.cpp:182:33 [opt] frame #24: 0x00000001037b9351 terra`clang::CodeGen::CodeGenFunction::EmitBranchOnBoolExpr(this=, Cond=0x000000010b86dad0, TrueBlock=0x0000600001700380, FalseBlock=0x00006000017004c0, TrueCount=0, LH=LH_None) at CodeGenFunction.cpp:1853:13 [opt] frame #25: 0x000000010375bc36 terra`clang::CodeGen::CodeGenFunction::EmitIfStmt(this=0x00007ff7bfefc208, S=0x000000010b86db20) at CGStmt.cpp:779:3 [opt] frame #26: 0x000000010375a958 terra`clang::CodeGen::CodeGenFunction::EmitStmt(this=0x00007ff7bfefc208, S=0x000000010b86db20, Attrs=ArrayRef @ 0x0000600000d50a80) at CGStmt.cpp:148:32 [opt] frame #27: 0x000000010376590c terra`clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope(this=0x00007ff7bfefc208, S=, GetLast=, AggSlot=AggValueSlot @ 0x00007ff7bfefc030) at CGStmt.cpp:503:7 [opt] frame #28: 0x00000001037b77ff terra`clang::CodeGen::CodeGenFunction::EmitFunctionBody(this=0x00007ff7bfefc208, Body=0x000000010b86df08) at CodeGenFunction.cpp:1217:5 [opt] frame #29: 0x00000001037b83f1 terra`clang::CodeGen::CodeGenFunction::GenerateCode(this=0x00007ff7bfefc208, GD=, Fn=, FnInfo=0x0000600003300000) at CodeGenFunction.cpp:1426:5 [opt] frame #30: 0x00000001037d3bab terra`clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(this=0x000000010d0ea400, GD=, GV=0x0000600002c00088) at CodeGenModule.cpp:5051:26 [opt] frame #31: 0x00000001037ccca6 terra`clang::CodeGen::CodeGenModule::EmitGlobalDefinition(this=0x000000010d0ea400, GD=, GV=0x0000000000000000) at CodeGenModule.cpp:3323:12 [opt] frame #32: 0x00000001037d0d36 terra`clang::CodeGen::CodeGenModule::EmitGlobal(this=, GD=) at CodeGenModule.cpp:3064:5 [opt] [artificial] frame #33: 0x00000001037d789c terra`clang::CodeGen::CodeGenModule::EmitTopLevelDecl(this=0x000000010d0ea400, D=0x000000010b86d7f0) at CodeGenModule.cpp:5892:5 [opt] frame #34: 0x000000010384f9cf terra`(anonymous namespace)::CodeGeneratorImpl::HandleTopLevelDecl(this=0x000000010d0e9c00, DG=DeclGroupRef @ 0x00007ff7bfefefb0) at ModuleBuilder.cpp:174:18 [opt] frame #35: 0x0000000100074d92 terra`CodeGenProxy::HandleTopLevelDecl(this=0x0000600003905110, D=DeclGroupRef @ 0x00007ff7bfeff008) at tcwrapper.cpp:592:20 frame #36: 0x0000000103897067 terra`clang::ParseAST(S=0x000000010d0ece00, PrintStats=, SkipFunctionBodies=) at ParseAST.cpp:162:31 [opt] frame #37: 0x0000000100069ccf terra`dofile(T=0x000000010b57bd80, TT=0x0000600002c08500, code="#include \n#include \n#include \n\nint checkargs(const char* f, va_list vl)\n{\n if(strcmp(f, \"%i%i%f%s\"))\n return 1;\n \n if(va_arg(vl, int) != -10)\n return 2;\n if(va_arg(vl, unsigned long long) != 60000000000001ULL)\n return 3;\n if(va_arg(vl, double) != 60.01)\n return 4;\n if(strcmp(va_arg(vl, const char*), \"TesT\"))\n return 5;\n return 0;\n}\n\n", args=size=13, result=0x00007ff7bfeff548) at tcwrapper.cpp:1045:5 frame #38: 0x0000000100069773 terra`include_c(L=0x000000010b572380) at tcwrapper.cpp:1129:9 frame #39: 0x000000010475d256 terra`lj_BC_FUNCC + 68 frame #40: 0x00000001046f8e9b terra`lua_pcall(L=, nargs=, nresults=, errfunc=) at lj_api.c:1145:12 [opt] frame #41: 0x00000001000041fa terra`docall(L=0x000000010b572380, narg=0, clear=0) at main.cpp:339:14 frame #42: 0x0000000100003e01 terra`main(argc=2, argv=0x00007ff7bfeff828) at main.cpp:119:13 frame #43: 0x000000010b39552e dyld`start + 462 ```

(Again in LLVM 14.)

elliottslaughter commented 2 years ago

I made some progress debugging this in Terra vs Clang. The key difference seems to be this branch:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.6/clang/lib/CodeGen/TargetInfo.cpp#L5513

isDarwinPCS() returns true in Clang and false in Terra.

elliottslaughter commented 2 years ago

Kind gets initialized at:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/CodeGen/TargetInfo.cpp#L11280

Seems to be getting the ABI from the Target.

In Clang, this seems to come from the TargetOptions:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Basic/Targets.cpp#L702

This call does not appear to occur in Terra; probably the option is not set.

TargetOptions comes from the Invocation here:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Frontend/CompilerInstance.cpp#L105

I don't see CompilerInstance::setInvocation() used in Clang, so I suppose the value is coming from the constructor.

The CompilerInvocation seems to come from CompilerInvocation::CreateFromArgsImpl:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Frontend/CompilerInvocation.cpp#L4434

Which calls ParseTargetArgs:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Frontend/CompilerInvocation.cpp#L4471

Which goes into this heavily-macroized code:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Frontend/CompilerInvocation.cpp#L4405

clang/Driver/Options.inc seems to be a generated file that lists every possible Clang option.

This may be a rabbit hole, I'm not sure it's useful to continue.

elliottslaughter commented 2 years ago

Running my vanilla Clang with -v:

$ clang bug605.c -S -emit-llvm -o bug605.ll -target arm64-apple-darwin21.6.0 -v
clang version 14.0.0
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Users/elliott/Programming/Terra/llvm/llvm-14-src-reldeb/install/bin
 (in-process)
 "/Users/elliott/Programming/Terra/llvm/llvm-14-src-reldeb/install/bin/clang-14" -cc1 -triple arm64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-llvm -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name bug605.c -mrelocation-model pic -pic-level 2 -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -funwind-tables=2 -target-sdk-version=12.3 -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -target-cpu apple-m1 -target-feature +v8.5a -target-feature +fp-armv8 -target-feature +neon -target-feature +crc -target-feature +crypto -target-feature +dotprod -target-feature +fp16fml -target-feature +ras -target-feature +lse -target-feature +rdm -target-feature +rcpc -target-feature +zcm -target-feature +zcz -target-feature +fullfp16 -target-feature +sha2 -target-feature +aes -target-abi darwinpcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=lldb -target-linker-version 764 -v -fcoverage-compilation-dir=/Users/elliott/Programming/Terra/terra -resource-dir /Users/elliott/Programming/Terra/llvm/llvm-14-src-reldeb/install/lib/clang/14.0.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Users/elliott/Programming/Terra/llvm/llvm-14-src-reldeb/install/lib/clang/14.0.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -fdebug-compilation-dir=/Users/elliott/Programming/Terra/terra -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -fcolor-diagnostics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o bug605.ll -x c bug605.c
clang -cc1 version 14.0.0 based upon LLVM 14.0.0 default target x86_64-apple-darwin21.6.0
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /Users/elliott/Programming/Terra/llvm/llvm-14-src-reldeb/install/lib/clang/14.0.0/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.

I noticed this includes the flag -target-abi darwinpcs.

Adding that back into the Terra reproducer:

local target = terralib.newtarget {
  Triple = 'arm64-apple-darwin21.6.0',
}

C = terralib.includecstring([[
#include <stdio.h>
#include <string.h>
#include <stdarg.h>

int checkargs(const char* f, va_list vl)
{
  if(strcmp(f, "%i%i%f%s"))
    return 1;

  if(va_arg(vl, int) != -10)
    return 2;
  if(va_arg(vl, unsigned long long) != 60000000000001ULL)
    return 3;
  if(va_arg(vl, double) != 60.01)
    return 4;
  if(strcmp(va_arg(vl, const char*), "TesT"))
    return 5;
  return 0;
}

]], {"-target-abi", "darwinpcs"}, target)

And this passes!

$ ./build/bin/terra bug605.t

I'm not sure what process by which Clang chooses its default arguments, but it seems pretty clear here that Clang is selecting these differently from Terra, and a few of the default values are critical to correctly compiling on macOS M1.

elliottslaughter commented 2 years ago

Breaking at: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Frontend/CompilerInvocation.cpp#L4443

I have confirmed that CommandLineArgs already contains all of the extra flags I saw above:

$ lldb -- .../llvm-14-src-reldeb/install/bin/clang bug605.c -S -emit-llvm -o bug605.ll -target arm64-apple-darwin21.6.0-src-reldeb/install/bin/clang bug605.c -S -emit-llvm -o bug
(lldb) target create ".../llvm-14-src-reldeb/install/bin/clang"
Current executable set to '.../llvm-14-src-reldeb/install/bin/clang' (x86_64).
(lldb) settings set -- target.run-args  "bug605.c" "-S" "-emit-llvm" "-o" "bug605.ll" "-target" "arm64-apple-darwin21.6.0"
(lldb) b CompilerInvocation.cpp:4443
Breakpoint 1: where = clang`clang::CompilerInvocation::CreateFromArgsImpl(clang::CompilerInvocation&, llvm::ArrayRef<char const*>, clang::DiagnosticsEngine&, char const*) + 77 at CompilerInvocation.cpp:4443:28, address = 0x000000010288303d
(lldb) r
Process 45217 launched: '.../llvm-14-src-reldeb/install/bin/clang' (x86_64)
clang was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 45217 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x000000010288303d clang`clang::CompilerInvocation::CreateFromArgsImpl(Res=0x000000010b012c00, CommandLineArgs=ArrayRef<const char *> @ 0x0000600001202520, Diags=0x00007ff7bfefd660, Argv0=".../llvm-14-src-reldeb/install/bin/clang-14") at CompilerInvocation.cpp:4443:28 [opt]
   4440   const OptTable &Opts = getDriverOptTable();
   4441   const unsigned IncludedFlagsBitmask = options::CC1Option;
   4442   unsigned MissingArgIndex, MissingArgCount;
-> 4443   InputArgList Args = Opts.ParseArgs(CommandLineArgs, MissingArgIndex,
   4444                                      MissingArgCount, IncludedFlagsBitmask);
   4445   LangOptions &LangOpts = *Res.getLangOpts();
   4446 
Target 0: (clang) stopped.
(lldb) p CommandLineArgs
(llvm::ArrayRef<const char *>) $0 = {
  Data = 0x00007ff7bfefdda8
  Length = 95
}
(lldb) p CommandLineArgs.Data
(const char *const *) $1 = 0x00007ff7bfefdda8
(lldb) p CommandLineArgs.Data[0]
(const char *const) $2 = 0x0000000106166dd8 "-cc1"
(lldb) p CommandLineArgs.Data[1]
(const char *const) $3 = 0x0000000106173634 "-triple"
(lldb) p CommandLineArgs.Data[2]
(const char *const) $4 = 0x000060000020dbc0 "arm64-apple-macosx12.0.0"
(lldb) p CommandLineArgs.Data[3]
(const char *const) $5 = 0x000000010616a033 "-Wundef-prefix=TARGET_OS_"
(lldb) p CommandLineArgs.Data[4]
(const char *const) $6 = 0x000000010616a04d "-Werror=undef-prefix"
(lldb) p CommandLineArgs.Data[5]
(const char *const) $7 = 0x000000010616a062 "-Wdeprecated-objc-isa-usage"
(lldb) p CommandLineArgs.Data[6]
(const char *const) $8 = 0x000000010616a07e "-Werror=deprecated-objc-isa-usage"
(lldb) p CommandLineArgs.Data[7]
(const char *const) $9 = 0x0000000106166e5c "-emit-llvm"
(lldb) p CommandLineArgs.Data[8]
(const char *const) $10 = 0x000000010617346e "-disable-free"
(lldb) p CommandLineArgs.Data[9]
(const char *const) $11 = 0x000000010617044b "-clear-ast-before-backend"
(lldb) p CommandLineArgs.Data[10]
(const char *const) $12 = 0x0000000106170023 "-disable-llvm-verifier"
(lldb) p CommandLineArgs.Data[11]
(const char *const) $13 = 0x0000000106166f32 "-discard-value-names"
(lldb) p CommandLineArgs.Data[12]
(const char *const) $14 = 0x0000000106170493 "-main-file-name"
(lldb) p CommandLineArgs.Data[13]
(const char *const) $15 = 0x0000600000c20d31 "bug605.c"
(lldb) p CommandLineArgs.Data[14]
(const char *const) $16 = 0x0000000106166fa4 "-mrelocation-model"
(lldb) p CommandLineArgs.Data[15]
(const char *const) $17 = 0x000000010616898c "pic"
(lldb) p CommandLineArgs.Data[16]
(const char *const) $18 = 0x00000001061711c2 "-pic-level"
(lldb) p CommandLineArgs.Data[17]
(const char *const) $19 = 0x0000000106166fc4 "2"
(lldb) p CommandLineArgs.Data[18]
...
(const char *const) $61 = 0x000000010617363c "-target-abi"
(lldb) p CommandLineArgs.Data[60]
(const char *const) $62 = 0x0000000106168132 "darwinpcs"

This in turn seems to come straight from cc1_main:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/tools/driver/cc1_main.cpp#L212

Which goes through driver.cpp:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/tools/driver/driver.cpp#L317

Some sort of job management infrastructure (not changing the args as far as I can tell):

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Job.cpp#L407

Through Compilation::ExecuteCommand:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Compilation.cpp#L196

(C.Arguments here has size 95, so it seems the command has already been filled at this point.)

Compilation::ExecuteJobs:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Compilation.cpp#L249

Driver::ExecuteCompilation:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L1614

And finally back full circle to main in driver.cpp:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/tools/driver/driver.cpp#L489

I note that Args here has size 8 (maybe some arguments have been removed?).

The interesting part of this file seems to be the call to Driver::BuildCompilation:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L1063

ArgList on entry:

(lldb) p ArgList
(llvm::ArrayRef<const char *>) $10 = {
  Data = 0x00007ff7bfefed08
  Length = 8
}
(lldb) p ArgList.Data[0]
(const char *const) $11 = 0x00007ff7bfeff998 ".../llvm-14-src-reldeb/install/bin/clang"
(lldb) p ArgList.Data[1]
(const char *const) $12 = 0x00007ff7bfeff9e3 "bug605.c"
(lldb) p ArgList.Data[2]
(const char *const) $13 = 0x00007ff7bfeff9ec "-S"
(lldb) p ArgList.Data[3]
(const char *const) $14 = 0x00007ff7bfeff9ef "-emit-llvm"
(lldb) p ArgList.Data[4]
(const char *const) $15 = 0x00007ff7bfeff9fa "-o"
(lldb) p ArgList.Data[5]
(const char *const) $16 = 0x00007ff7bfeff9fd "bug605.ll"
(lldb) p ArgList.Data[6]
(const char *const) $17 = 0x00007ff7bfeffa07 "-target"
(lldb) p ArgList.Data[7]
(const char *const) $18 = 0x00007ff7bfeffa0f "arm64-apple-darwin21.6.0"

I see that there is a TargetTriple being set:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L1194

At the call to getToolChain:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L1251

I see that Triple has this value:

(const llvm::Triple) $29 = (Data = "arm64-apple-darwin21.6.0", Arch = aarch64, SubArch = ARMSubArch_v8, Vendor = Apple, OS = Darwin, Environment = UnknownEnvironment, ObjectFormat = MachO)

We pass through the DarwinClang toolchain:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L5522

After we come back we initialize the Compilation:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Compilation.cpp#L36

After we come back we call Driver::HandleImmediateArgs:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L1812

This is where -v seems to get handled. (Running with -v in a debugger though, I do not see the verbose arguments get printed here. Just some basic version information and installation/resource dirs.)

After we come back we call Driver::BuildInputs:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L2276

This seems just to classify each input argument (e.g., bug605.c) and its associated language (if not otherwise specified).

After we return we call into Driver::BuildUniversalActions:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L2115

After we return we call into Driver::BuildJobs:

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/Driver.cpp#L4232

... Skipping ahead, there are many layers of calls and they're mostly not interesting...

Darwin::ComputeEffectiveClangTriple: This is where the OS seems to get set to something specific.

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Darwin.cpp#L986

Before this call, the Triple is arm64-apple-darwin21.6.0; afterwards it is arm64-apple-macosx12.0.0.

Clang::ConstructJob: This seems to be setting up a list of command line arguments (CmdArgs).

https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Darwin.cpp#L986

Note: this is where the -cc1 command line argument comes from: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L4417

Note also, this is the modified triple that gets inserted here (i.e., macos12.0.0): https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L4417

Warning options: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L4500

The output type: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L4578

The input filename: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L4810

PIC level: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L4874

(... Many, many options ...)

Target options: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L5281

Target CPU: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L5314

RenderTargetOptions: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Driver/ToolChains/Clang.cpp#L5320

(And then it comes back and finishes with the rest of the job construction logic.)

So that explains how all that happens, but it's not obvious to me if any piece of this can be reused; it's coupled pretty tightly to Clang's argument parsing and job construction.

shoe42 commented 2 years ago

Thanks for the thorough bisecting - it's interesting to see the steps that Clang goes through, and a learning experience on my end. So, to clarify, the issue is that Terra is selecting the default (wrong) aapcs ABI? RenderAArch64ABI seems to add darwinpcs for both xxx-xxx-darwin and xxx-xxx-macosx targets, which should check for either Darwin or MacOSX in the triple, but I presume Clang::AddAArch64TargetArgs is part of Clang's frontend, which Terra doesn't use? Please do correct me if I'm wrong. Would it be appropriate to handle adding the -target-api darwinpcs flag in terra_inittarget, if macOS + AArch64 is detected at compile-time?

elliottslaughter commented 2 years ago

The boundary between Clang and LLVM is messy and often confusing.

In this case, the ABI seems to be a feature of Clang, not LLVM. So it's not stored or tracked in the LLVM Target at all. It seems to be used by Clang to decide what code to generate; but afterwards everything flows through LLVM in a way that is completely agnostic to this.

But not everything is like this: the target OS (which Clang silently changes from darwin to macosx), and the various feature flags (that Clang adds by default) are part of the LLVM Target and are passed to LLVM. So far none of these seem to be a major issue, but they could be. In theory, Clang and Terra could generate incompatible code if they don't both use the same configuration.

So I think there is a design question here about what level to solve this at. Right now Terra only uses Clang in a very limited capacity to parse C header files. Because Clang doesn't expose much of the internals here, we do go through the CompilerInvocation::CreateFromArgs code path (see below). But this is not apparently sufficient to get the higher-level Clang default argument code path.

https://github.com/terralang/terra/blob/f46c5cabd13632417d382e27d35b91f8eac923a4/src/tcwrapper.cpp#L903-L909

Perhaps there is a better place to start the initialization of Clang. I'll try to do a bit more monkeying around to see what I can find.

If that doesn't work out, then we could try to just add the minimal set of flags to fix this specific case, probably in tcwrapper.cpp near where I linked above.

shoe42 commented 2 years ago

It looks like the constructor of clang::targets::AArch64TargetInfo is failing to set the correct ABI, as it defaults to aapcs and doesn't take into account darwinpcs on AArch64 Darwin: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Basic/Targets/AArch64.cpp#L66 - not sure if this is by design, or an edge case specific to Darwin AArch64 when not using clang::driver::tools::Clang::ConstructJob, and therefore not calling RenderAArch64ABI. The clang::targets::ARMTargetInfo constructor, for comparison, seems to handle the ABI manually: https://github.com/llvm/llvm-project/blob/llvmorg-14.0.0/clang/lib/Basic/Targets/ARM.cpp#L282-L326.

elliottslaughter commented 2 years ago

I just noticed that the entire infrastructure to call Driver::BuildCompilation is already here and has been since 2019. Ouch.

https://github.com/terralang/terra/blob/f46c5cabd13632417d382e27d35b91f8eac923a4/src/tcwrapper.cpp#L867-L870

I'll dig a bit more to figure out if we can extract what we need.

elliottslaughter commented 2 years ago

Re: the TargetInfo subclasses: I don't think that route is going to work because this all falls under Clang's initialization via CompilerInvocation::CreateFromArgs. Clang could change to make it work, and arguably it would be more consistent, but it doesn't work that way right now.

I was able to get a Driver::BuildCompilation to spit out the -cc1 job that I need to get the -target-abi flag. It's ugly, but it's what I've got right now. Maybe having built it, there is a way to use the constructed ToolChain to get that argument. I'll need to do more digging.

shoe42 commented 2 years ago

Do you think it's worth filing an upstream bug report regarding clang::targets::AArch64TargetInfo not initializing the ABI to darwinpcs when on AArch64 Darwin?

elliottslaughter commented 2 years ago

After a few quick searches on the LLVM bug tracker, the only related issue I can find is https://github.com/llvm/llvm-project/issues/45989#issuecomment-981030476, which makes a passing reference to RenderAArch64ABI and the toolchain.

I don't know how open the Clang project would be to considering this, but it seems worth trying. You might point out that (as best we can tell) LLDB uses the same CompilerInvocation::CreateFromArgs code path to instantiate Clang and is probably susceptible to the same bug we are here. So it's not just an issue in one client.

elliottslaughter commented 2 years ago

Tracing up the call chain from RenderAArch64ABI, where clang/lib/Driver/ToolChains/Clang.cpp adds the -target-abi flag for AArch64:

RenderAArch64ABI: anonymous namespace:

Clang::AddAArch64TargetArgs: private:

Clang::RenderTargetOptions: private:

Clang::ConstructJob: public:

It seems like there is no point in attempting to cut down to a level where we can access, because the first public method already adds basically all of the options and will be more cumbersome to use than Driver::BuildCompilation. RenderAArch64ABI does not store the ABI value in any sort of useful way and just sticks it in the output command line arguments.

So I guess we really do have to parse these command line arguments, there is no other way that is exposed by Clang right now.

elliottslaughter commented 2 years ago

It's not pretty, but you can try https://github.com/elliottslaughter/terra/tree/fix-macos-m1-target-abi and let me know if it works for you to fix this bug.

shoe42 commented 2 years ago

Looks like that fixes the test for vararg.t (tested against LLVM 13 built from source) - nice!

For what it's worth, the following tests still fail on that branch:

The following tests FAILED:
     27 - atomicrmw.t (Subprocess aborted)
     53 - bug603.t (Subprocess aborted)
     55 - bug604.t (Failed)
     66 - cconv_array.t (Failed)
elliottslaughter commented 2 years ago

I submitted an upstream issue at https://github.com/llvm/llvm-project/issues/57529. We'll see what the Clang developers have to say about this.

shoe42 commented 2 years ago

Closing as fixed for now.