Linker issue when indirect branch obfuscation is active

[NewDwarf]

I faced with the interesting issue specific to the indirect branch obfuscation in c++ code. The linker reports

ld: error: relocation refers to a discarded section: .text._ZNSt6__ndk19allocatorIcE10deallocateEPcm
>>> defined in second.o
>>> section group signature: _ZNSt6__ndk19allocatorIcE10deallocateEPcm
>>> prevailing definition is in first.o
>>> referenced by second.cpp
>>>               second.o:(.data+0xD28)

ld: error: relocation refers to a discarded section: .text.__clang_call_terminate
>>> defined in second.o
>>> section group signature: __clang_call_terminate
>>> prevailing definition is in first.o
>>> referenced by second.cpp
>>>               second.o:(.data+0xD38)
>>> referenced by second.cpp
>>>               second.o:(.data+0xD40)
>>> referenced by second.cpp
>>>               second.o:(.data+0xD48)
>>> referenced 13 more times

The data references here are the offset to the IndirectBranchingTargetAddress table. The issue can be reproduced using only the -mllvm -enable-indibran parameter.

Here is the sample files first.cpp and second.cpp used to create the shared library.

first.cpp

#include <iostream>
#include <string>

int first()
{
  char buffer[20];
  std::string str ("Test string...");
  std::size_t length = str.copy(buffer, 6, 5);
  buffer[length] = '\0';
  std::cout << "buffer contains: " << buffer << '\n';
  return 0;
}

second.cpp

#include <iostream>
#include <string>

int second ()
{
  std::string base="this is a test string.";
  std::string str2="n example";
  std::string str3="sample phrase";
  std::string str4="useful.";

  // replace signatures used in the same order as described above:

  // Using positions:                 0123456789*123456789*12345
  std::string str=base;           // "this is a test string."
  str.replace(9,5,str2);          // "this is an example string." (1)
  str.replace(19,6,str3,7,6);     // "this is an example phrase." (2)
  str.replace(8,10,"just a");     // "this is just a phrase."     (3)
  str.replace(8,6,"a shorty",7);  // "this is a short phrase."    (4)
  str.replace(22,1,3,'!');        // "this is a short phrase!!!"  (5)

  // Using iterators:                                               0123456789*123456789*
  str.replace(str.begin(),str.end()-3,str3);                    // "sample phrase!!!"      (1)
  str.replace(str.begin(),str.begin()+6,"replace");             // "replace phrase!!!"     (3)
  str.replace(str.begin()+8,str.begin()+14,"is coolness",7);    // "replace is cool!!!"    (4)
  str.replace(str.begin()+12,str.end()-4,4,'o');                // "replace is cooool!!!"  (5)
  str.replace(str.begin()+11,str.end(),str4.begin(),str4.end());// "replace is useful."    (6)
  std::cout << str << '\n';
  return 0;
}

Compile them

clang++ -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 first.cpp -c -mllvm -enable-indibran
and
clang++ -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 second.cpp -c -mllvm -enable-indibran

-mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 are used to emit more code with the branches so that to give the chance to the indirect branch obfuscator to construct the IndirectBranchingTargetAddress table.

Finally, link the object files clang++ -target aarch64-none-linux-android21 -fPIC -shared -Wl,-soname,libsample.so -o libsample.so first.o second.o

to reproduce the issue.

[61bcdefg]

Can you reproduce this problem when building for iOS/MacOS? I don't have Android Studio installed. Linking for macOS looks clear but I am not sure I did everything clearly.

[Naville]

Probably COMDAT issue, just remove all comdats

[NewDwarf]

Can you reproduce this problem when building for iOS/MacOS? I don't have Android Studio installed. Linking for macOS looks clear but I am not sure I did everything clearly.

You don't need Android Studio. Just install NDK for you host machine from https://developer.android.com/ndk/downloads I guess any clang compiler will understand the -target aarch64-none-linux-android21
At least it works for me.

clang -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 second.cpp -c -mllvm -enable-indibran --sysroot=/path/to/NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/sysroot

clang -target aarch64-none-linux-android21 -mllvm -enable-bcfobf -mllvm -enable-splitobf -mllvm -split_num=9 -O2 first.cpp -c -mllvm -enable-indibran --sysroot=/path/to/NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/sysroot

After that invoke the linker via NDK's clang

NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -fPIC -shared -Wl,-soname,libsample.so -o libsample.so first.o second.o

This can help to reproduce the issue quickly.

[Naville]

Can you reproduce this problem when building for iOS/MacOS? I don't have Android Studio installed. Linking for macOS looks clear but I am not sure I did everything clearly.

MachO has no such concept as COMDAT, so this bug will never trigger on Darwin

[61bcdefg]

MachO has no such concept as COMDAT, so this bug will never trigger on Darwin

Thank you for your reply, I don't know anything about ELF files, I need some time to understand it and try to fix it.

[Naville]

MachO has no such concept as COMDAT, so this bug will never trigger on Darwin

Thank you for your reply, I don't know anything about ELF files, I need some time to understand it and try to fix it.

As mentioned, just remove all the COMDATs should fix the issue

[NewDwarf]

@NeHyci I did some tests to give you details to understand how to fix it.

Let's try to reproduce it as simple as possible. Create three source files:

header.hpp

template <typename T> T myMax(T x, T y)
{
    return (x > y) ? x : y;
}

foo.cpp int foo(int a, int b) { return myMax(a, b); }

bar.cpp int bar(int a, int b) { return myMax(a, b); }

The linker will give such output:

ld: error: relocation refers to a discarded section: .text._Z5myMaxIiET_S0_S0_
>>> defined in bar.o
>>> section group signature: _Z5myMaxIiET_S0_S0_
>>> prevailing definition is in foo.o
>>> referenced by bar.cpp
>>>               bar.o:(.data+0x0)
>>> referenced by bar.cpp
>>>               bar.o:(.data+0x8)
>>> referenced by bar.cpp
>>>               bar.o:(.data+0x10)
>>> referenced 2 more times
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Let's generate human readable bitcode for both cpp files for normal situation (without indirect branch obfuscation). foo.ll.ok:

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

; Function Attrs: mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn
define dso_local i32 @_Z3fooii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = icmp sgt i32 %0, %1
  %4 = select i1 %3, i32 %0, i32 %1
  ret i32 %4
}

attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

bar.ll.ok:

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

; Function Attrs: mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn
define dso_local i32 @_Z3barii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = icmp sgt i32 %0, %1
  %4 = select i1 %3, i32 %0, i32 %1
  ret i32 %4
}

attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone uwtable willreturn "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

As we can see, there are no COMDAT

But the obfuscated by the indirect branch bitcode has COMDAT!!! bar.ll.bad

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*), i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress nounwind uwtable
define dso_local i32 @_Z3barii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = tail call i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1)
  ret i32 %3
}

; Function Attrs: mustprogress nounwind uwtable
define linkonce_odr dso_local i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1) #0 comdat {
  %3 = icmp sgt i32 %0, %1
  %4 = zext i1 %3 to i64
  %5 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i64 0, i64 %4
  %6 = load i8**, i8** %5, align 8
  indirectbr i8** %6, [label %7, label %8]

7:                                                ; preds = %2
  br label %8

8:                                                ; preds = %2, %7
  %9 = phi i32 [ %1, %7 ], [ %0, %2 ]
  ret i32 %9
}

attributes #0 = { mustprogress nounwind uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

foo.ll.bad

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %7), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %8)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*), i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress nounwind uwtable
define dso_local i32 @_Z3fooii(i32 %0, i32 %1) local_unnamed_addr #0 {
  %3 = tail call i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1)
  ret i32 %3
}

; Function Attrs: mustprogress nounwind uwtable
define linkonce_odr dso_local i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1) #0 comdat {
  %3 = icmp sgt i32 %0, %1
  %4 = zext i1 %3 to i64
  %5 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i64 0, i64 %4
  %6 = load i8**, i8** %5, align 8
  indirectbr i8** %6, [label %7, label %8]

7:                                                ; preds = %2
  br label %8

8:                                                ; preds = %2, %7
  %9 = phi i32 [ %1, %7 ], [ %0, %2 ]
  ret i32 %9
}

attributes #0 = { mustprogress nounwind uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+neon,+outline-atomics" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 1, !"branch-target-enforcement", i32 0}
!2 = !{i32 1, !"sign-return-address", i32 0}
!3 = !{i32 1, !"sign-return-address-all", i32 0}
!4 = !{i32 1, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"clang version 14.0.0 (https://github.com/apple/llvm-project.git c33f34f65cd5eda385964c4176906d55d7236206)"}

@Naville How to deal with COMDAT here? When we don't use the indirect branch obfuscation, foo.ll.ok and bar.ll.ok inline the template function from the header.hpp into the body of the caller functions. Everything clear in this scenario.

When we use indirect branch obfuscation, both foo.ll.bad and bar.ll.bad have definition of the template function from the header.hpp inside and it is marked as comdat. If we will remove the COMDAT mark from the template function definition, we will get duplicated definitions and linking also fails.

If we completely remove define linkonce_odr dso_local i32 @_Z5myMaxIiET_S0_S0_(i32 %0, i32 %1) #0 comdat The linker can also fail in the case if we remove all function definitions.

[NewDwarf]

@Naville In my understanding, the root of the problem are the header files from STL which has a lot of definitions of small (2-3 lines) template functions which are normally always inlined. When we run the indirect branch obfuscator, we make small template function defined in the header file too large to incorporate it in the caller and we have copies of these functions in each compilation unit. These functions are marked as COMDAT and we get ld: error: relocation refers to a discarded section Otherwise, we would get ld: error: duplicate symbol error

Probably, I didn't get your idea how to manage this issue...

[NewDwarf]

@Naville I guess, you meant that we have to do each function as local to the specific compilation unit. So it will be kind of the static function (from the C perspective) So, if we change define linkonce_odr dso_local i32 @_Z5myMaxIiET_S0S0(i32 %0, i32 %1) #0 comdat on define dso_local i32 @_Z5myMaxIiET_S0S0(i32 %0, i32 %1) #0 it will get invisible for all other object modules and linking will not fail.

Could you, please, confirm that?

And another one question. $_Z5myMaxIiET_S0S0 = comdat any To get rid of any COMDAT section in the object file, we have to remove it too?

[NewDwarf]

@NeHyci I edited manually both *bad.ll files with removing $Z5myMaxIiET_S0_S0 = comdat any and changing function declaration from define linkonce_odr dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0 comdat on define dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0

After that, I was able to link the bitcode modules without issues! So, you need to find a way to remove it programmatically.

[Naville]

define linkonce_odr dso_local i32 @Z5myMaxIiET_S0_S0(i32 %0, i32 %1) #0 comdat

I doubt changing linkage is required

[NewDwarf]

The main question for me is why inlining transformation won't work when the indirect branching is active. Maybe it is because the inliner is not able to find the room to insert the function being inlined from the STL code into the function modified by the indirect branch obfuscator? @Naville Could you confirm that the indirect branch obfuscator runs before inline transformation? If so, can we change this behavior so that to run the indirect branch obfuscator AFTER inlining, this would resolve the problem.

[Naville]

Could you confirm that the indirect branch obfuscator runs before inline transformation?

-debug-pass=Executions

If so, can we change this behavior so that to run the indirect branch obfuscator AFTER inlining

You can

[Naville]

this would resolve the problem.

Because your current pass pipeline is stupid, you need to think about where your pass fits the best in the pipeline

[NewDwarf]

@NeHyci I think the best solution for this issue would be pipeline management so that to run indirect branch obfuscation AFTER the inliner.

[61bcdefg]

@NewDwarf I'm still trying to build NDK....It was very slow

[NewDwarf]

@NewDwarf I'm still trying to build NDK....It was very slow

If you need any help to assist to build exactly the same version of clang as NDK has, just ping me.

[61bcdefg]

@NeHyci I think the best solution for this issue would be pipeline management so that to run indirect branch obfuscation AFTER the inliner.

But how do we ensure that the Pass Plugin executes after the inliner and it works for all pipelines?

[NewDwarf]

@NeHyci I think the best solution for this issue would be pipeline management so that to run indirect branch obfuscation AFTER the inliner.

But how do we ensure that the Pass Plugin executes after the inliner and it works for all pipelines?

It's simple. As @Naville mentioned, we have to use the hidden parameter -debug-pass=Executions NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -O2 -c 2.cpp -mllvm -enable-indibran -mllvm -debug-pass=Executions Above command gives you complete statistic.

[61bcdefg]

It's simple. As @Naville mentioned, we have to use the hidden parameter -debug-pass=Executions

NDK/25/NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -O2 -c 2.cpp -mllvm -enable-indibran -mllvm -debug-pass=Executions

Above command gives you complete statistic.

I mean if we use Obfuscation Pass as the LLVM Pass Plugin, how can we make it execute after AlwaysInliner in all pipelines? I've taken a cursory look at the source code in lib/Passes and this seems impossible.

[Naville]

Injection order is execution order for Transform passes

[NewDwarf]

Statistic output produced by the clang/opt

[2023-01-11 12:32:33.323157000] 0x7fda1d70e760     Executing Pass 'Partially inline calls to library functions' on Function '_Z6secondRNSt6__ndk112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEES6_'...
[2023-01-11 12:32:33.323169000] 0x7fda1d70e760      Freeing Pass 'Partially inline calls to library functions' on Function '_Z6secondRNSt6__ndk112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEES6_'...

The order is in llvm//lib/Transforms/Scalar/PartiallyInlineLibCalls.cpp

char PartiallyInlineLibCallsLegacyPass::ID = 0;
INITIALIZE_PASS_BEGIN(PartiallyInlineLibCallsLegacyPass,
                      "partially-inline-libcalls",
                      "Partially inline calls to library functions", false,
                      false)
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
INITIALIZE_PASS_END(PartiallyInlineLibCallsLegacyPass,
                    "partially-inline-libcalls",
                    "Partially inline calls to library functions", false, false)

[Naville]

This is not the pass you are looking for? This is for library calls

[NewDwarf]

This is not the pass you are looking for? This is for library calls

Forgot to mention, I have been already working with such code

#include <string>

int second (std::string &str1, std::string &str2)
{
  if (str1.compare(str2)) printf("Not equal\n");

  return 0;
}

[Naville]

Honestly don't understand your reasoning here anymore, we'll review the actual full patch when submit it

[NewDwarf]

Honestly don't understand your reasoning here anymore, we'll review the actual full patch when submit it

I am new in compilers. ...but how LLVM works very impressed me to learn it in more details. This is why I am here. ...this is a main reason why my progress is too slow at this moment as I have to understand many concepts before in such complex project.

[Naville]

No worries, we will review your patch together. It's just information lost during our discussion here

[NewDwarf]

@Naville Could you, please, give a clue why I don't see "Enable Obfuscation" statistic in the -debug-pass=Executionsoutput? I guess, we have to get "Enable Obfuscation" in the output as the obfuscation pass is registered as INITIALIZE_PASS_BEGIN(Obfuscation, "obfus", "Enable Obfuscation", true, true)

The naive approach is just add as the dependency "PartiallyInlineLibCallsLegacyPass" in the obfuscator pass registration

char Obfuscation::ID = 0;
INITIALIZE_PASS_BEGIN(Obfuscation, "obfus", "Enable Obfuscation", true, true)
INITIALIZE_PASS_DEPENDENCY(PartiallyInlineLibCallsLegacyPass);
INITIALIZE_PASS_DEPENDENCY(AntiClassDump);
INITIALIZE_PASS_DEPENDENCY(BogusControlFlow);
...

but the main question here - is PartiallyInlineLibCallsLegacyPass always passed to the pipeline? I checked it quickly on the samples which produces no inlined code. PartiallyInlineLibCallsLegacyPass is executed in these cases as well.

[Naville]

"Enable Obfuscation" statistic in the -debug-pass=Executions output?

The output is the passname, which you change by overriding "StringRef getPassName() const" in LPM

e naive approach is just add as the dependency "PartiallyInlineLibCallsLegacyPass" in the obfuscator pass registration

Wrong approach,wrong pass

[NewDwarf]

I noticed that optimization level 0 (-O0) also inserts COMDAT's regardless of using the indirect branch obfuscation. And such code (not obfuscated by indirect branching but compiled as -O0) is linked fine.

Using the same code

//header.hpp
template <typename T> T myMax(T x, T y) { return (x > y) ? x : y; }

// foo.cpp
#include "header.hpp"
int foo(int a, int b) { return myMax(a, b); }

// bar.cpp
#include "header.hpp"
int bar(int a, int b) { return myMax(a, b); }

The bitcode with -0O and DISABLED indirect branch obfuscation looks foo.ll

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3fooii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  br i1 %7, label %8, label %10

8:                                                ; preds = %2
  %9 = load i32, i32* %3, align 4
  br label %12

10:                                               ; preds = %2
  %11 = load i32, i32* %4, align 4
  br label %12

12:                                               ; preds = %10, %8
  %13 = phi i32 [ %9, %8 ], [ %11, %10 ]
  ret i32 %13
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

and bar.ll

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3barii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  br i1 %7, label %8, label %10

8:                                                ; preds = %2
  %9 = load i32, i32* %3, align 4
  br label %12

10:                                               ; preds = %2
  %11 = load i32, i32* %4, align 4
  br label %12

12:                                               ; preds = %10, %8
  %13 = phi i32 [ %9, %8 ], [ %11, %10 ]
  ret i32 %13
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

In contrast, the code with ENABLED indirect branch obfuscation foo.ll.bad

; ModuleID = 'foo.cpp'
source_filename = "foo.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*), i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3fooii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  %8 = zext i1 %7 to i32
  %9 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i32 0, i32 %8
  %10 = load i8*, i8** %9, align 8
  %11 = load i8**, i8** %9, align 8
  indirectbr i8** %11, [label %17, label %12]

12:                                               ; preds = %2
  %13 = load i32, i32* %3, align 4
  %14 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %15 = load i8*, i8** %14, align 8
  %16 = load i8**, i8** %14, align 8
  indirectbr i8** %16, [label %22]

17:                                               ; preds = %2
  %18 = load i32, i32* %4, align 4
  %19 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %20 = load i8*, i8** %19, align 8
  %21 = load i8**, i8** %19, align 8
  indirectbr i8** %21, [label %22]

22:                                               ; preds = %17, %12
  %23 = phi i32 [ %13, %12 ], [ %18, %17 ]
  ret i32 %23
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

and bar.ll.bad

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12)]
@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*), i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3barii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  %8 = zext i1 %7 to i32
  %9 = getelementptr [2 x i8*], [2 x i8*]* @HikariConditionalLocalIndirectBranchingTable, i32 0, i32 %8
  %10 = load i8*, i8** %9, align 8
  %11 = load i8**, i8** %9, align 8
  indirectbr i8** %11, [label %17, label %12]

12:                                               ; preds = %2
  %13 = load i32, i32* %3, align 4
  %14 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %15 = load i8*, i8** %14, align 8
  %16 = load i8**, i8** %14, align 8
  indirectbr i8** %16, [label %22]

17:                                               ; preds = %2
  %18 = load i32, i32* %4, align 4
  %19 = getelementptr [3 x i8*], [3 x i8*]* @IndirectBranchingGlobalTable, i32 0, i32 2
  %20 = load i8*, i8** %19, align 8
  %21 = load i8**, i8** %19, align 8
  indirectbr i8** %21, [label %22]

22:                                               ; preds = %17, %12
  %23 = phi i32 [ %13, %12 ], [ %18, %17 ]
  ret i32 %23
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

[NewDwarf]

If we remove any references to the @IndirectBranchingGlobalTable and @HikariConditionalLocalIndirectBranchingTable from the bitcode, linking works fine. So something wrong with these tables. For example

; ModuleID = 'bar.cpp'
source_filename = "bar.cpp"
target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-none-linux-android21"

$_Z5myMaxIiET_S0_S0_ = comdat any

;@IndirectBranchingGlobalTable = private global [3 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
;@HikariConditionalLocalIndirectBranchingTable = private global [2 x i8*] [i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %17), i8* blockaddress(@_Z5myMaxIiET_S0_S0_, %12)]
;@llvm.compiler.used = appending global [2 x i8*] [i8* bitcast ([3 x i8*]* @IndirectBranchingGlobalTable to i8*), i8* bitcast ([2 x i8*]* @HikariConditionalLocalIndirectBranchingTable to i8*)], section "llvm.metadata"

; Function Attrs: mustprogress noinline optnone uwtable
define dso_local noundef i32 @_Z3barii(i32 noundef %0, i32 noundef %1) #0 {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = call noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %5, i32 noundef %6)
  ret i32 %7
}

; Function Attrs: mustprogress noinline nounwind optnone uwtable
define linkonce_odr dso_local noundef i32 @_Z5myMaxIiET_S0_S0_(i32 noundef %0, i32 noundef %1) #1 comdat {
  %3 = alloca i32, align 4
  %4 = alloca i32, align 4
  store i32 %0, i32* %3, align 4
  store i32 %1, i32* %4, align 4
  %5 = load i32, i32* %3, align 4
  %6 = load i32, i32* %4, align 4
  %7 = icmp sgt i32 %5, %6
  %8 = zext i1 %7 to i32

  ret i32 0
}

attributes #0 = { mustprogress noinline optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }
attributes #1 = { mustprogress noinline nounwind optnone uwtable "frame-pointer"="non-leaf" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="generic" "target-features"="+fix-cortex-a53-835769,+neon,+outline-atomics,+v8a" }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !6, !7, !8}
!llvm.ident = !{!9}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 8, !"branch-target-enforcement", i32 0}
!2 = !{i32 8, !"sign-return-address", i32 0}
!3 = !{i32 8, !"sign-return-address-all", i32 0}
!4 = !{i32 8, !"sign-return-address-with-bkey", i32 0}
!5 = !{i32 7, !"PIC Level", i32 2}
!6 = !{i32 7, !"PIE Level", i32 2}
!7 = !{i32 7, !"uwtable", i32 1}
!8 = !{i32 7, !"frame-pointer", i32 1}
!9 = !{!"Android (dev, based on r450784d) clang version 14.0.6 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0cca074e9238af8b4106c30add4418f6)"}

[Naville]

Again, comdat issue

[Naville]

GlobalValue::setComdat(nullptr)

[NewDwarf]

@Naville Quick question to you. I am trying to edit bitcode manually to understand better the original problem but I get such error on translating ll to bc

bar.ll.bad:36:14: error: explicit pointee type doesn't match operand's pointee type
  %11 = load i8**, i8** %9, align 8
             ^

If I correctly understood, this is because of opaque pointers. How to fix this issue manually to make llvm-as happy and don't break the code semantic?

[Naville]

How to fix this issue manually to make llvm-as happy and don't break the code semantic?

Disable opaque pointer mode

[NewDwarf]

GlobalValue::setComdat(nullptr)

The problem here is who marks the functions as comdat. I am almost sure this happens after running the indirect branch obfuscation and if traverse all functions in the module by F->setComdat(nullptr); immediately after running indirect branch obfuscation, I don't get the result as comdats will be inserted later.

[NewDwarf]

@Naville Thanks a lot for assistance. Removing comdats directly before running the indirect branch obfuscation helped! Now linking works perfect.

[NewDwarf]

@NeHyci The patch is very simple

diff --git a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
index 97ac85b49..cd6bffca2 100755
--- a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
+++ b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
@@ -66,6 +66,7 @@ struct IndirectBranch : public FunctionPass {
     if (!this->initialized)
       initialize(*Func.getParent());
     errs() << "Running IndirectBranch On " << Func.getName() << "\n";
+    Func.setComdat(nullptr);
     vector<BranchInst *> BIs;
     for (Instruction &Inst : instructions(Func))
       if (BranchInst *BI = dyn_cast<BranchInst>(&Inst))

[NewDwarf]

@Naville But, anyway, it would be nice to understand original issue as the linker is able to resolve correctly comdats of any type. When the indirect branch tables are inserted, linking of comdats modules gets broken. I tried to pass the -mllvm -opaque-pointers=0 parameter NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ -target aarch64-none-linux-android21 -mllvm -opaque-pointers=0 -O0 -c foo.cpp -emit-llvm -S -mllvm -enable-indibran -o foo.ll.bad but ll -> bc translation still doesn't work throwing the same error message

NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as foo.ll.bad 
NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as: foo.ll.bad:36:14: error: explicit pointee type doesn't match operand's pointee type (i8** vs i8*)
  %11 = load i8**, i8** %9, align 8
             ^

when I pass -mllvm -opaque-pointers=1, error message is changed

NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as foo.ll.bad 
foo.ll.bad:8:53: warning: ptr type is only supported in -opaque-pointers mode
@IndirectBranchingGlobalTable = private global [3 x ptr] [ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %12), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %17), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
                                                    ^
NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-as: foo.ll.bad:8:53: error: expected type
@IndirectBranchingGlobalTable = private global [3 x ptr] [ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %12), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %17), ptr blockaddress(@_Z5myMaxIiET_S0_S0_, %22)]
                                                    ^

[Naville]

Wrong, if you use clang, you instruct clang to emit opaque pointer IR (or not), that -mllvm only configures the LLVM level option, CFE still emits IR in default type system

[NewDwarf]

Wrong, if you use clang, you instruct clang to emit opaque pointer IR (or not), that -mllvm only configures the LLVM level option, CFE still emits IR in default type system

NDK's clang doesn't accept the parameter -Xclang -no-opaque-pointers.

[NewDwarf]

Func.setComdat(nullptr); works only for

template <typename T> T myMax(T x, T y) { return (x > y) ? x : y; }

But it doesn't work for NDK STL. It is definitely necessary to look at the "PartiallyInlineLibCallsLegacyPass" direction and run the indirect branch obfuscator only when "PartiallyInlineLibCallsLegacyPass" finish its work. @Naville Can we do this using existing obfuscator's scheduler or we need to create standalone indirect branch pass and push it in the pass pipeline after the "PartiallyInlineLibCallsLegacyPass" pass?

[Naville]

What on earth makes you think PartiallyInlineLibCallsLegacyPass is related, I'm curious

[Naville]

Func.setComdat(nullptr); works only for

Try the opposite, branchtable->setComdat(Func.getComdat())

[NewDwarf]

@Naville I had some more changes which filtered some symbols in the indirect branch pass. I removed them and checked again Func.setComdat(nullptr); I confirm that this solution works.

[NewDwarf]

@Naville Are there any clang/opt options to check LLVM transformation after the specific pass? Say, to dump the bitcode of the specific module before transformation and after transformation.

[NewDwarf]

@Naville Are there any clang/opt options to check LLVM transformation after the specific pass? Say, to dump the bitcode of the specific module before transformation and after transformation. Looks like -mllvm -print-after-all is exactly what I need!

[NewDwarf]

@NeHyci Did you have a chance to reproduce this issue?

[61bcdefg]

@NeHyci Did you have a chance to reproduce this issue?

I think this obfuscator is designed primarily for iOS/MacOS, so issues for other platforms are a low priority. And NDK takes a lot of time to build, I don't have a lot of free time.

[NewDwarf]

It takes ~1 hour to build from the scratch on my MacBook pro 15'' 2019. Each time, when you update the obfuscation specific codebase, recompiling takes about 1-2 minutes. Of cause, it is up to you, but if you need some assistance for building NDK, just let me know. As I already mentioned, below patch works for me.

diff --git a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
index 97ac85b49..cd6bffca2 100755
--- a/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
+++ b/llvm/lib/Transforms/Obfuscation/IndirectBranch.cpp
@@ -66,6 +66,7 @@ struct IndirectBranch : public FunctionPass {
     if (!this->initialized)
       initialize(*Func.getParent());
     errs() << "Running IndirectBranch On " << Func.getName() << "\n";
+    Func.setComdat(nullptr);
     vector<BranchInst *> BIs;
     for (Instruction &Inst : instructions(Func))
       if (BranchInst *BI = dyn_cast<BranchInst>(&Inst))