search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
175 stars 90 forks source link

Provide Spark 3.4 Support for Spline w/ Backwards Compatibility #793

Open rycowhi opened 3 months ago

rycowhi commented 3 months ago

Addressing #705 ,

TODO:

sonarcloud[bot] commented 3 months ago

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

wajda commented 3 months ago

@rycowhi Thanks a million for your effort. Guys let me take a look at those event pairs more closely and I'll get back to you later. I'm a bit out of capacity at the moment, but will try to find some time during the next day or two.

devsaik commented 3 months ago

Just checking on status of this PR. We are waiting on this feature. @wajda any updates.

wajda commented 3 months ago

Just checking on status of this PR. We are waiting on this feature. @wajda any updates.

Not yet, but it's high on my todo list for this week.

luke3481 commented 3 months ago

@wajda thanks for taking a look at @rycowhi's PR. Let me know if you'll be able to get to it this week - we're also waiting on this feature.

kaimalsreeja commented 3 months ago

@wajda Looks like this feature is in high demand. Our team is also watching the PR merge, highly appreciate if you could review this as early as possible. Thank you in advance.

wajda commented 3 months ago

Sorry folks, I've still been buried in work on another high-priority project, and have yet to obtain time allocation approval for this one. The Spline project was essentially put on hold by the company last year, that's the reason why it received almost no support for the past X months. I'm not giving up, but I can't really spend my free time on this any more, so trying to get some official allocation from the employer.

ramonje5 commented 1 month ago

Sorry folks, I've still been buried in work on another high-priority project, and have yet to obtain time allocation approval for this one. The Spline project was essentially put on hold by the company last year, that's the reason why it received almost no support for the past X months. I'm not giving up, but I can't really spend my free time on this any more, so trying to get some official allocation from the employer.

Hello! Is there any news regarding this Spark 3.4 support? I've been using Spline since a long time and the fact that there is no support for Scala 2.13 is going to affect a lot of projects and maybe the need to find a new listener.

If there is any way I could contribute to this feature, or to speed things up, please let me know!

Thanks!

Edit: I've just seen Upgrade org.scala-lang:scala-library from 2.12.17 to 2.13.14 #806

wajda commented 1 month ago

@ramonje5 thank you for the message. Unfortunately our company has yet to sort out its plans and book of work for Spline. Currently the team has zero capacity to work on it.

If there is any way I could contribute to this feature, or to speed things up, please let me know!

Thanks!

Edit: I've just seen https://github.com/AbsaOSS/spline-spark-agent/pull/806

The PR #806 is automatically created by Snyk and won't work. Please ignore it. Upgrading to Scala 2.13 cannot be a simple change like that, it would require adding another vertical into the build matrix, so to speak. Basically the agent is built for every supported Sclala+Spark version combination.

If you want to help that would be awesome if you take this current PR, test it, address @rycowhi's points in the related issue #705. When testing succeeds and those points are addressed we can merge and release this PR.

rycowhi commented 1 month ago

@ramonje5 thank you for the message. Unfortunately our company has yet to sort out its plans and book of work for Spline. Currently the team has zero capacity to work on it.

If there is any way I could contribute to this feature, or to speed things up, please let me know! Thanks! Edit: I've just seen #806

The PR #806 is automatically created by Snyk and won't work. Please ignore it. Upgrading to Scala 2.13 cannot be a simple change like that, it would require adding another vertical into the build matrix, so to speak. Basically the agent is built for every supported Sclala+Spark version combination.

If you want to help that would be awesome if you take this current PR, test it, address @rycowhi's points in the related issue #705. When testing succeeds and those points are addressed we can merge and release this PR.

At least for the versions maintained off of the existing cross-builds is there anything else to look into? I believe the only outstanding piece was seeing that there are now two (different commands) emitted off of a CTAS in 3.4 but have confirmed this is a new Spark behavior (by seeing the same, multiple events happen on a barebones query listener). Is the outstanding piece to see how that would look in the Spline UI, or?