search

kedro-org / kedro-devrel

Kedro developer relations team use this for content creation ideation and execution
Apache License 2.0
0 stars 3 forks source link

Create a blog post about stream processing with Kedro #79

Closed stichbury closed 1 year ago

stichbury commented 1 year ago

Work with the associated team to convert some markdown into a blog post (technical) that covers the detail that any complementary post on Medium omits.

deepyaman commented 1 year ago

Lest I forget about it when the time comes, I think it's important to note the limitations of the current implementation, and what you can do with it. I think the takeaways for the reader should be:

The takeaway should not be:

Specifically for Spark, being able to deploy a streaming application is predicated on being able to deploy a job to a cluster (see https://github.com/kedro-org/kedro/issues/529), which is something that has been requested for a while but isn't support AFAIK. (The alternative may be to support cluster deployment.)

stichbury commented 1 year ago

This is now with the SME team to review and expand. The draft is here https://github.com/kedro-org/kedro-devrel/pull/96