search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
185 stars 95 forks source link

change to 9 parameters constructor #823

Closed XavierZYXue closed 2 months ago

XavierZYXue commented 3 months ago

fixes #821 to switch the 9 parameters constructor, could you please review it?

cerveada commented 3 months ago

Hello, thanks for your contribution.

Please correct me if I am wrong, but this fixes the issues for the new version of BigQuery, but also it will break it for the older versions.

Another thing I notice is that in the linked Big Query code the method has 8 params, but here you use 9. How come?

XavierZYXue commented 3 months ago

@cerveada hi, first of appreciated your response, I revised this logic because it was changed to 9 parameters after this PR, and affetced all versions after 0.26.0 that is to say, it will be applicable to all versions after 0.26.0

cerveada commented 3 months ago

That is ok, but we should keep it compatible with the old version as well. We already do this in other plugins. Usually some combination of retries and reflection is sufficient to support both.

You can see some examples of it here: core/src/main/scala/za/co/absa/spline/harvester/plugin/embedded/DataSourceV2Plugin.scala

XavierZYXue commented 3 months ago

@cerveada I've added different constructors separately are compatible with all Spark BigQuery connector please review again?

XavierZYXue commented 2 months ago

any comment for this after I changed? and why the pipeline always timeout? do you know the reason

sonarcloud[bot] commented 2 months ago

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

rkrumins commented 2 months ago

Amazing, thank you very much @cerveada and @cerveada