Add new draft of "The ways of integrating Kedro with Databricks"

kedro-org / kedro-devrel

Kedro developer relations team use this for content creation ideation and execution

Apache License 2.0

0 stars 3 forks source link

Add new draft of "The ways of integrating Kedro with Databricks" #104

Closed stichbury closed 10 months ago

stichbury commented 11 months ago

Working on a blog post with Diego Lira

stichbury commented 11 months ago

Hi @diegoliraQB A few comments from @noklam in the Kedro team to review please. One other point he makes is the following:

"Is it a pure Spark workflow, if the workflow requires moving data to local to compute? It usually has great performance hit (and issues with moving data locally is a compliance concern IIRC)" and I wondered if you could consider this and maybe make a note on the article that you are doing this, so those with compliance requirements should be mindful of it.