kwai / blaze

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.
Apache License 2.0
1.3k stars 121 forks source link

`RUSTFLAGS='-C target-cpu=native'` causes jvm crash when compile and runtime environments differ #584

Closed XorSum closed 1 month ago

XorSum commented 2 months ago

Describe the bug

The CI machine used for compiling Blaze and the machines in the cluster used for running Blaze are different, resulting in a JVM crash when running.

2024-09-23 11:57:33.692 [Stage 0:(0 + 8) / 8][Stage 1:(0 + 0) / 1][Stage 2:(0 + 0) / 4]
------ initializing blaze native environment ------
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00007f2bbad0d7ec, pid=380, tid=0x00007f2bbf533700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libblaze-3882176084054705220.tmp+0x3307ec]  _$LT$blaze..logging..SimpleLogger$u20$as$u20$log..Log$GT$::log::hd1b9037843c55282+0x5c
#
# Core dump written. Default location: 

To Reproduce Steps to reproduce the behavior:

  1. compile and run blaze on different cpu

Expected behavior do not crash when running

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

richox commented 2 months ago

i suggest we use target-cpu=native as a default configuration since it can bring extra performance profits with cpu that supports newer simd instructions (like avx512). however we can add some configurations to allow users to compile with a specified cpu feature.

XorSum commented 2 months ago

i suggest we use target-cpu=native as a default configuration since it can bring extra performance profits with cpu that supports newer simd instructions (like avx512). however we can add some configurations to allow users to compile with a specified cpu feature.

I have modified the pull request to make RUSTFLAGS configurable. If the user does not set the RUSTFLAGS variable, its default value will be -C target-cpu=native; if the user sets the RUSTFLAGS variable, the user's value will be used.