search

eclipse-aspectj / aspectj

Other
303 stars 86 forks source link

[Spark On Kubernetes] Aspectj Load-Time Weaving Spark API failure #318

Closed melin closed 1 month ago

melin commented 1 month ago

Verify that the user has read and write permissions on the table. Use Aspectj Load-Time Weaving to block the sparksession.sql method. Gets the sql to execute. Parse the sql and verify permissions(https://github.com/melin/superior-sql-parser)

Spark On Yarn aspectj can successfully weave spark api methods. spark app jar Startup classloader is AppClassLoader.

spark on kuberbernete, spark app jar Startup classloader is MutableURLClassLoader. spark api methods cannot be woven into the spark jar. The Spark JAR is added by the AppClassLoader.

image

[AppClassLoader@18b4aac2] info AspectJ Weaver Version 1.9.22.1 built on Friday May 10, 2024 at 23:23:16 PDT
[AppClassLoader@18b4aac2] info register classloader sun.misc.Launcher$AppClassLoader@18b4aac2
[AppClassLoader@18b4aac2] info using configuration file:/yarn/nm/filecache/3130/spark-jobserver-driver-3.5.0.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkCatalogAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkDataFrameAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkSessionAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.SparkContextAspectj
[AppClassLoader@18b4aac2] info register aspect io.github.melin.jobserver.spark.driver.aspectj.HdfsApiAspectj
kriegaex commented 1 month ago

I do not understand the error description, and there is no reproducer. Please improve the issue in these two respects to avoid it getting closed as invalid. Thank you.

kriegaex commented 1 month ago

No reaction, closing as invalid. Feel free to add information, then we can consider reopening the issue.

melin commented 1 month ago

No reaction, closing as invalid. Feel free to add information, then we can consider reopening the issue.

thank you, Problem solved

kriegaex commented 1 month ago

@melin, then why didn't you say so before? If you create an issue here, causing work for others, please be polite enough to explain what was the root cause of the problem and how exactly you solved it, so everybody can learn about it for future reference, if a similar problem comes up. Thank you.