kwai / blaze

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.
Apache License 2.0
968 stars 90 forks source link

Why is scala's logInfo unable to output logs? #503

Closed Au-Miner closed 1 day ago

Au-Miner commented 6 days ago

Describe the bug Why do I only print Rust logs when I run code on Spark333 and execute SQL commands in Sparkshell? How do I print logs related to Scala?

To Reproduce

  1. generate jar package.
    SHIM=spark333
    MODE=release
    mvn package -P"${SHIM}" -P"${MODE}"
  2. run spark-shell
    /home/qcsun/wql_zx_files/spark333/bin/spark-shell --name run_test \
    --master local[4] --deploy-mode client \
    --jars /home/qcsun/wql_zx_files/blaze-engine/target/blaze-engine-spark333-release-2.0.9.1-SNAPSHOT.jar

    3.execute some code

    import org.apache.spark.sql.SparkSession
    val spark = SparkSession.builder().appName("Spark SQL Example").getOrCreate()
    spark.sql("CREATE DATABASE IF NOT EXISTS test_db")
    spark.sql("USE test_db")
    spark.sql("DROP TABLE IF EXISTS test_table")
    spark.sql("""
    CREATE TABLE IF NOT EXISTS test_table (
    id INT,
    value DOUBLE
    )
    USING PARQUET
    """)
    val data = spark.range(100000).map(id => (
    id,
    Math.random() * 100,
    )).toDF("id", "value")
    data.write.mode("overwrite").insertInto("test_table")
    sql("SELECT AVG(value) FROM test_table").show()
  3. I found only rust log can be printed. So why?!

Screenshots

image

Desktop (please complete the following information):

Au-Miner commented 6 days ago

I have found that modifying any code in Scala through the above process cannot properly affect the execution of Spark shell. I am certain that my jar package and other settings are correct

Au-Miner commented 6 days ago

Even if I modify BlazeSparkSessionExtension to the following content, it still cannot affect the execution of the spark shell. I am certain that the jar package has been updated

class BlazeSparkSessionExtension extends (SparkSessionExtensions => Unit) with Logging {
  Shims.get.initExtension()

  override def apply(extensions: SparkSessionExtensions): Unit = {
    return
  }
}
richox commented 2 days ago

scala logs are configured by conf/log4j2.properties, you can use a minimal conf:

rootLogger.level = info
rootLogger.appenderRef.stdout.ref = console

appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
Au-Miner commented 2 days ago

scala logs are configured by conf/log4j2.properties, you can use a minimal conf:

rootLogger.level = info
rootLogger.appenderRef.stdout.ref = console

appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex

Thank you very much.