Enhance tremor with connectors for the Google Cloud Platform

[darach]

Add support for the GCP suite of APIs to tremor as connectors ( source, sink, linked transport ) minimally to offer support for GCS and PubSub, but with reuse in mind for extending API support to the GCP suite of APIs in tremor over time.

Describe the problem you are trying to solve

Minimally enhance tremor with support for GCS and PubSub.

Describe the solution you'd like

Ideally the plumbing and integration for GCP and its suite of APIs would be a separate rust crate / project usable by the rust community at large. One possible approach to implementation would be adding a rust code generation target to Google's own API client generation toolkit: https://github.com/googleapis/gapic-generator

Notes

Recommended skills - A basic working knowledge of rust and java with some exposure to tools and frameworks that generate bindings against declarative API specifications to produce source code. A basic working knowledge of CI, particulary github actions.

Update

A suggested set of milestones:

• [x] Write a tremor sink that can write to GCS
• [x] Write a tremor source that can read from GCS
• [x] Write a tremor source that consumes from Google PubSub
• [x] Write a tremor sink that can publish to Google PubSub
• [x] Write a walkthrough guide for setting and using GCS
• [x] Write a walkthrough guide for setting up and using Pubsub
• x] Write a tremor sink that can publish to Google Cloud Logging
• [x] Write a walkthrough guide for GCL Cloud Logging
• [x] Update tremor docs
• [ ] Prepare a presentation on the connectivity for the tremor community

Possible extensions and evolutions ( optional ):

• Extend gapic-builder with rust support and refactor connectors
  • This extension would also involve the java programming language and contributing to gapic-builder
• Investigate acknowledgements in the context of GCP connectors
• Investigate guaranteed delivery in the context of GCP connectors
• Investigate circuit breakers in the context of GCP connectors

Useful Resources:

• https://docs.rs/gcp_auth/0.4.0/gcp_auth/
• https://docs.rs/google-pubsub1/1.0.14+20200627/google_pubsub1/
• https://crates.io/crates/cloud-pubsub/0.2.0
• https://byron.github.io/google-apis-rs/
• https://github.com/Byron/google-apis-rs
• https://crates.io/crates/gcs
• https://github.com/EmbarkStudios/tame-gcs
• https://cloud.google.com/pubsub/docs
• https://cloud.google.com/storage

[imskr]

I would to work on this issue. Please assign me :)

[Licenser]

Hi there :D, thank you so much for your interest!

Feel free to hop by our discord to chat.

[imskr]

sure @licenser

[SantamRC]

Hello @Licenser @darach, This is Santam. I have basic understanding of Rust and Google Cloud Platform. I am interested to work on this issue.

[darach]

Hi @SantamRC,

Thank you for your interest. Feel free to hop by our discord to chat ( https://bit.ly/tremor-discord )

Cheers,

Darach.