search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8k stars 792 forks source link

Support Serverless Google Cloud Functions #268

Open gregorskii opened 4 years ago

gregorskii commented 4 years ago

Any plans to support Google Cloud functions similar to the way Lambda is supported?

If you have an idea of what would need to be done to do this I can potentially help with a PR.

Thanks.

Jeffail commented 4 years ago

Hey @gregorskii,

Definitely, I haven't got much experience with Google Cloud so I was reluctant to take this on myself. The plumbing work should already be there from the lambda implementation, so it should hopefully just be a case of following the same process.

First step would be to create a package similar to https://github.com/Jeffail/benthos/tree/master/lib/serverless/lambda, which does the execution handler wrapping around the serverless library.

Then add a cmd package similar to https://github.com/Jeffail/benthos/tree/master/cmd/serverless/benthos-lambda

Finally, add the build and archive stages for the cmd package to https://github.com/Jeffail/benthos/blob/master/.goreleaser.yml and it should then get packaged in future releases.

I'm happy to help out with an unfinished PR but I'd need you to do the deployment testing.

gregorskii commented 4 years ago

Thank you I will review these notes and see if I can help.

Appreciate the direction.

gregorskii commented 4 years ago

An update on this. It runs quite well in Google Cloud Run as a regular docker image. Cloud Run has similar costs/complexity to Cloud Functions.

I believe Cloud Run suits my use case, up to you if it makes sense to leave this open. I do think others would benefit from it.

Jeffail commented 4 years ago

Thanks for the update @gregorskii, I'll leave this open as it'd be cool to have as an option at some point.