eclipse-aspectj / aspectj

Other
295 stars 85 forks source link

[Spark On Kubernetes] Aspectj Load-Time Weaving Spark API failure #318

Open melin opened 1 week ago

melin commented 1 week ago

Verify that the user has read and write permissions on the table. Use Aspectj Load-Time Weaving to block the sparksession.sql method. Gets the sql to execute. Parse the sql and verify permissions(https://github.com/melin/superior-sql-parser)

Spark On Yarn aspectj can successfully weave spark api methods. spark app jar Startup classloader is AppClassLoader.

spark on kuberbernete, spark app jar Startup classloader is MutableURLClassLoader. spark api methods cannot be woven into the spark jar. The Spark JAR is added by the AppClassLoader.

image

[AppClassLoader@18b4aac2] info AspectJ Weaver Version 1.9.22.1 built on Friday May 10, 2024 at 23:23:16 PDT
[AppClassLoader@18b4aac2] info register classloader sun.misc.Launcher$AppClassLoader@18b4aac2
[AppClassLoader@18b4aac2] info using configuration file:/yarn/nm/filecache/3130/spark-jobserver-driver-3.5.0.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkCatalogAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkDataFrameAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkSessionAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkContextAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.HdfsApiAspectj
kriegaex commented 1 week ago

I do not understand the error description, and there is no reproducer. Please improve the issue in these two respects to avoid it getting closed as invalid. Thank you.