search

JenspederM / kedro-databricks

A Databricks Plugin for Kedro
MIT License
13 stars 5 forks source link

How to make it easier to switch conf/data location #34

Open JenspederM opened 3 months ago

JenspederM commented 3 months ago

When users don't have access to dbfs, they should be able to provide another path to which conf/data is uploaded.

What to do here?

JenspederM commented 3 months ago

My first thought was to simply add an argument to deploy which lets you set the upload path. However,the location is also used for the logging path in bundle...

So how to ensure alignment between these two functions? It almost seems as if the best way to go forward would be to extend the default configuration, maybe reserve the kedro_databricks keyword the conf/base/databricks.yml file. What do you think?

theoclipse commented 1 month ago

Hello,

I am also trying to deploy through kedro-databricks plugin. I have the same problem than you: Trying to deploy a dev conf => it overides stagging conf (it is on the same databricks instance)

Would you have some suggestion to do that?

JenspederM commented 1 month ago

I have an idea of the issues, but not how to resolve it. :(

theoclipse commented 1 month ago

Ok, I will try to propose you an update next week :)