search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
185 stars 95 forks source link

Questions regarding Spline Agent #679

Closed zacayd closed 1 year ago

zacayd commented 1 year ago

Hi team I uses your solution on my application on databricks on azure and i have issues with the fact that i need to write some code of scala to get the name of the running notebook and cluster to the exection plan 1.is there is a plan to add the code to get the databrciks params to the execution plan? 2.Do you have a stable code of spline agent? if not- what is the frequency of change 3.How can we get the spline code and can push to the cloud?

thanks alot

wajda commented 1 year ago

We do not have any plans to implement Databricks specific features, although we do our best to ensure at least core functionality is supported. Nevertheless we continue thinking about what we can improve on Databricks support without introducing any hard dependency on the platform.

zacayd commented 1 year ago

Thanks

Do you have a stable code of spline agent? if not- what is the frequency of change? How can we get the spline code and can push to the cloud?

wajda commented 1 year ago

Every release is as stable as it can be from the developer point of view. We do not have a dedicated QA team though. Just use the latest released version and it should be fine. They are all generally backward compatible. If you want to use a development version you can also safely do it as long as you are not using it for production. You will have to build it yourself though.

wajda commented 1 year ago

what is the frequency of change

there is no stable release schedule. We release a new version when we are confident and feel comfortable doing so. Usually when a bug is found in a released version we release an update ASAP.

wajda commented 1 year ago

How can we get the spline code and can push to the cloud?

  1. git clone https://github.com/AbsaOSS/spline-spark-agent.git
  2. https://github.com/AbsaOSS/spline-spark-agent#building
zacayd commented 1 year ago

yes and How can i push it to cloud of Azure?

cerveada commented 1 year ago

and How can i push it to cloud of Azure?

I don't understand the question. Are you asking us how to do git push?

zacayd commented 1 year ago

push to a cloud in azure so I can see it on Databricks

cerveada commented 1 year ago

You can build the agent from the code and deploy it on the Databricks.

https://github.com/AbsaOSS/spline-getting-started/tree/main/spline-on-databricks

If you want to build it yourself, you just upload the jar instead of using maven to get it. You can upload any additional artifacts the same way.

zacayd commented 1 year ago

but can i build only version 1.0.4?

wajda commented 1 year ago

Pull the 'release/1.0.4' tag from git and build it as normal.

cerveada commented 1 year ago

I don't recommend using older versions, since there are several bugs that were fixed in newer versions. So at least 1.0.7 or ideally 1.1.0.

But all versions are tagged, so you can see them in here.

wajda commented 1 year ago

Closing thus issue as it doesn't contain any bug report or feature request. Please use discussion board for questions instead of github issues.