search

IntelLabs / Auto-Steer

Auto-Steer
MIT License
36 stars 7 forks source link

Spark inference mode error #11

Open wanweiqiangintel opened 6 months ago

wanweiqiangintel commented 6 months ago

After I run training mode with Spark database and get results/spark.sqlite successfully, then run the inference mode and get the error: image I found the method get_plan_preprocessor just implemented in Presto, should I implement it in Spark?

christophanneser commented 6 months ago

Thank you for opening the issue. Yes, currently, the plan preprocessor has just been implemented for Presto. I would highly appreciate it if you could implement the preprocessor also for Spark query plans. Thank you!

wanweiqiangintel commented 6 months ago

@christophanneser, Thank you for your comments! I am wondering if there is a way to reproduce the performance gain in SparkSQL TPC-DS workload as shown in the paper without implementing preprocessor?

christophanneser commented 4 months ago

You can run AutoSteer's training mode, which does not require the query plan preprocessor. Unfortunately, I must not share SparkSQL's exact configuration as it was/is internally used.