search

eunomia-bpf / bpftime

Userspace eBPF runtime for fast Uprobe & Syscall hook & Extensions with LLVM JIT
https://eunomia.dev/bpftime/
MIT License
687 stars 68 forks source link

[BUG] Unable to build on newer LLVM #268

Closed Officeyutong closed 2 months ago

Officeyutong commented 3 months ago

bpftime Failed to build on some newer LLVM versions, seemed to be related unexpected updates. Fix them.

See https://github.com/eunomia-bpf/bpftime/issues/266

NobinPegasus commented 2 months ago

I want to work on this. Can you provide some hints? How this should be achieved?

Officeyutong commented 2 months ago

I want to work on this. Can you provide some hints? How this should be achieved?

Just build bpftime with a newer LLVM version (such as 19), and you will see many compile errors like https://github.com/eunomia-bpf/bpftime/issues/266 . This might be related to the using of outdated APIs

NobinPegasus commented 2 months ago 
    SymbolMap extSymbols;
    for (uint32_t i = 0; i < std::size(vm->ext_funcs); i++) {
        if (vm->ext_funcs[i] != nullptr) {
            auto sym = JITEvaluatedSymbol::fromPointer(
                vm->ext_funcs[i]);
            auto symName = jit->getExecutionSession().intern(
                ext_func_sym(i));
            sym.setFlags(JITSymbolFlags::Callable |
                     JITSymbolFlags::Exported);
        --->    extSymbols.try_emplace(symName, sym);
            extFuncNames.push_back(ext_func_sym(i));
        }
    }
  template <typename KeyArg, typename... ValueArgs>
  BucketT *InsertIntoBucket(BucketT *TheBucket, KeyArg &&Key,
                            ValueArgs &&... Values) {
    TheBucket = InsertIntoBucketImpl(Key, Key, TheBucket);

    TheBucket->getFirst() = std::forward<KeyArg>(Key);
    ::new (&TheBucket->getSecond()) ValueT(std::forward<ValueArgs>(Values)...);
    return TheBucket;
  }

/home/pegasus/Documents/bpftime/vm/llvm-jit/src/llvm/llvm_jit_context.cpp:340:24: required from here /home/pegasus/Documents/llvm-project/llvm/include/llvm/ADT/DenseMap.h:577:5: error: no matching function for call to ‘llvm::orc::ExecutorSymbolDef::ExecutorSymbolDef(llvm::JITEvaluatedSymbol&)’ 577 | ::new (&TheBucket->getSecond()) ValueT(std::forward(Values)...); | ^~~~~~~~~~~~~~~~~~

    const auto tryDefineLddwHelper = [&](const char *name, void *func) {
        if (func) {
            SPDLOG_DEBUG("Defining LDDW helper {} with addr {:x}",
                     name, (uintptr_t)func);
            auto sym = JITEvaluatedSymbol::fromPointer(func);
            sym.setFlags(JITSymbolFlags::Callable |
                     JITSymbolFlags::Exported);
        --->    lddwSyms.try_emplace(
                jit->getExecutionSession().intern(name), sym);
            definedLddwHelpers.push_back(name);
        }
    };
  template <typename KeyArg, typename... ValueArgs>
  BucketT *InsertIntoBucket(BucketT *TheBucket, KeyArg &&Key,
                            ValueArgs &&... Values) {
    TheBucket = InsertIntoBucketImpl(Key, Key, TheBucket);

    TheBucket->getFirst() = std::forward<KeyArg>(Key);
    ::new (&TheBucket->getSecond()) ValueT(std::forward<ValueArgs>(Values)...);
    return TheBucket;
  }

The try_implace function:

  // Inserts key,value pair into the map if the key isn't already in the map.
  // The value is constructed in-place if the key is not in the map, otherwise
  // it is not moved.
  template <typename... Ts>
  std::pair<iterator, bool> try_emplace(KeyT &&Key, Ts &&... Args) {
    BucketT *TheBucket;
    if (LookupBucketFor(Key, TheBucket))
      return std::make_pair(makeIterator(TheBucket,
                                         shouldReverseIterate<KeyT>()
                                             ? getBuckets()
                                             : getBucketsEnd(),
                                         *this, true),
                            false); // Already in map.

    // Otherwise, insert the new element.
    TheBucket =
        InsertIntoBucket(TheBucket, std::move(Key), std::forward<Ts>(Args)...);
    return std::make_pair(makeIterator(TheBucket,
                                       shouldReverseIterate<KeyT>()
                                           ? getBuckets()
                                           : getBucketsEnd(),
                                       *this, true),
                          true);
  }

  // Inserts key,value pair into the map if the key isn't already in the map.
  // The value is constructed in-place if the key is not in the map, otherwise
  // it is not moved.
  template <typename... Ts>
  std::pair<iterator, bool> try_emplace(const KeyT &Key, Ts &&... Args) {
    BucketT *TheBucket;
    if (LookupBucketFor(Key, TheBucket))
      return std::make_pair(makeIterator(TheBucket,
                                         shouldReverseIterate<KeyT>()
                                             ? getBuckets()
                                             : getBucketsEnd(),
                                         *this, true),
                            false); // Already in map.

    // Otherwise, insert the new element.
    TheBucket = InsertIntoBucket(TheBucket, Key, std::forward<Ts>(Args)...);
    return std::make_pair(makeIterator(TheBucket,
                                       shouldReverseIterate<KeyT>()
                                           ? getBuckets()
                                           : getBucketsEnd(),
                                       *this, true),
                          true);
  }

The issue is originating from the second parameter of try_emplace(symName, sym); which is sym (llvm::JITEvaluatedSymbol sym)

NobinPegasus commented 2 months ago

@Officeyutong So what I can see is the error is originating from try_emplace function call. And what I think is has to do something with the datatype of sym. The second argument of try_emplace(). Can you provide some more guidance?

Officeyutong commented 2 months ago

@Officeyutong So what I can see is the error is originating from try_emplace function call. And what I think is has to do something with the datatype of sym. The second argument of try_emplace(). Can you provide some more guidance?

sym is the function pointer of the current defining eBPF helper. It was stored at vm->ext_funcs[i]

NobinPegasus commented 2 months ago

@Officeyutong 

[ 69%] Linking CXX executable ../../vm-llvm-example
/usr/bin/cmake: /usr/local/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
/usr/bin/ld: CMakeFiles/vm-llvm-example.dir/example/main.cpp.o: in function `_GLOBAL__sub_I_main.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x25e): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(ebpf_vm.cpp.o): in function `_GLOBAL__sub_I_ebpf_vm.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x1e): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(llvm_jit_context.cpp.o): in function `_GLOBAL__sub_I_llvm_jit_context.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x13b): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(compiler.cpp.o): in function `_GLOBAL__sub_I_compiler.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x1e): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(compiler_utils.cpp.o): in function `_GLOBAL__sub_I_compiler_utils.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x1e): undefined reference to `LLVMLinkInMCJIT'
collect2: error: ld returned 1 exit status
gmake[3]: *** [vm/llvm-jit/CMakeFiles/vm-llvm-example.dir/build.make:158: vm-llvm-example] Error 1
gmake[3]: Leaving directory '/home/pegasus/Documents/bpftime/build'
gmake[2]: *** [CMakeFiles/Makefile2:879: vm/llvm-jit/CMakeFiles/vm-llvm-example.dir/all] Error 2
gmake[2]: Leaving directory '/home/pegasus/Documents/bpftime/build'
gmake[1]: *** [Makefile:156: all] Error 2
gmake[1]: Leaving directory '/home/pegasus/Documents/bpftime/build'
make: *** [Makefile:62: release] Error 2

The changes I've made for LLVM VERSION >16

auto sym = JITEvaluatedSymbol::fromPointer(
                vm->ext_funcs[i]);
auto symbol = ::llvm::orc::ExecutorSymbolDef (::llvm::orc::ExecutorAddr (sym.getAddress()), sym.getFlags());
extSymbols.try_emplace(symName, symbol);

And inside the optimizeModule I've made this change added new handling for PassManager

static void optimizeModule(llvm::Module &M)
{
    if (LLVM_VERSION_MAJOR > 16) {
        // =====================
        // Create the analysis managers.
        // These must be declared in this order so that they are destroyed in the
        // correct order due to inter-analysis-manager references.
        LoopAnalysisManager LAM;
        FunctionAnalysisManager FAM;
        CGSCCAnalysisManager CGAM;
        ModuleAnalysisManager MAM;

        // Create the new pass manager builder.
        // Take a look at the PassBuilder constructor parameters for more
        // customization, e.g. specifying a TargetMachine or various debugging
        // options.
        PassBuilder PB;

        // Register all the basic analyses with the managers.
        PB.registerModuleAnalyses(MAM);
        PB.registerCGSCCAnalyses(CGAM);
        PB.registerFunctionAnalyses(FAM);
        PB.registerLoopAnalyses(LAM);
        PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);

        // Create the pass manager.
        // This one corresponds to a typical -O2 optimization pipeline.
        ModulePassManager MPM = PB.buildPerModuleDefaultPipeline(OptimizationLevel::O3);

        // Optimize the IR!
        MPM.run(M, MAM);
        // =====================================
    } else {
        llvm::legacy::PassManager PM;

        llvm::PassManagerBuilder PMB;
        PMB.OptLevel = 3;
        PMB.populateModulePassManager(PM);

        PM.run(M);
    }
}
Officeyutong commented 2 months ago

@Officeyutong

[ 69%] Linking CXX executable ../../vm-llvm-example
/usr/bin/cmake: /usr/local/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
/usr/bin/ld: CMakeFiles/vm-llvm-example.dir/example/main.cpp.o: in function `_GLOBAL__sub_I_main.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x25e): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(ebpf_vm.cpp.o): in function `_GLOBAL__sub_I_ebpf_vm.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x1e): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(llvm_jit_context.cpp.o): in function `_GLOBAL__sub_I_llvm_jit_context.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x13b): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(compiler.cpp.o): in function `_GLOBAL__sub_I_compiler.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x1e): undefined reference to `LLVMLinkInMCJIT'
/usr/bin/ld: ../libvm-bpf.a(compiler_utils.cpp.o): in function `_GLOBAL__sub_I_compiler_utils.cpp':
/home/pegasus/Documents/llvm-project/llvm/include/llvm/ExecutionEngine/MCJIT.h:35:(.text.startup+0x1e): undefined reference to `LLVMLinkInMCJIT'
collect2: error: ld returned 1 exit status
gmake[3]: *** [vm/llvm-jit/CMakeFiles/vm-llvm-example.dir/build.make:158: vm-llvm-example] Error 1
gmake[3]: Leaving directory '/home/pegasus/Documents/bpftime/build'
gmake[2]: *** [CMakeFiles/Makefile2:879: vm/llvm-jit/CMakeFiles/vm-llvm-example.dir/all] Error 2
gmake[2]: Leaving directory '/home/pegasus/Documents/bpftime/build'
gmake[1]: *** [Makefile:156: all] Error 2
gmake[1]: Leaving directory '/home/pegasus/Documents/bpftime/build'
make: *** [Makefile:62: release] Error 2

The changes I've made for LLVM VERSION >16

auto sym = JITEvaluatedSymbol::fromPointer(
              vm->ext_funcs[i]);
auto symbol = ::llvm::orc::ExecutorSymbolDef (::llvm::orc::ExecutorAddr (sym.getAddress()), sym.getFlags());
extSymbols.try_emplace(symName, symbol);

And inside the optimizeModule I've made this change added new handling for PassManager

static void optimizeModule(llvm::Module &M)
{
    if (LLVM_VERSION_MAJOR > 16) {
        // =====================
        // Create the analysis managers.
        // These must be declared in this order so that they are destroyed in the
        // correct order due to inter-analysis-manager references.
        LoopAnalysisManager LAM;
        FunctionAnalysisManager FAM;
        CGSCCAnalysisManager CGAM;
        ModuleAnalysisManager MAM;

        // Create the new pass manager builder.
        // Take a look at the PassBuilder constructor parameters for more
        // customization, e.g. specifying a TargetMachine or various debugging
        // options.
        PassBuilder PB;

        // Register all the basic analyses with the managers.
        PB.registerModuleAnalyses(MAM);
        PB.registerCGSCCAnalyses(CGAM);
        PB.registerFunctionAnalyses(FAM);
        PB.registerLoopAnalyses(LAM);
        PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);

        // Create the pass manager.
        // This one corresponds to a typical -O2 optimization pipeline.
        ModulePassManager MPM = PB.buildPerModuleDefaultPipeline(OptimizationLevel::O3);

        // Optimize the IR!
        MPM.run(M, MAM);
        // =====================================
    } else {
        llvm::legacy::PassManager PM;

        llvm::PassManagerBuilder PMB;
        PMB.OptLevel = 3;
        PMB.populateModulePassManager(PM);

        PM.run(M);
    }
}

This seems to be we missed some LLVM libraries when linking LLVM. Try adding missed parts https://github.com/eunomia-bpf/bpftime/blob/8beba35449d21b52f14a637174faeab4b3866539/vm/llvm-jit/CMakeLists.txt#L49