rvowles
commented
1 year ago Hiya @rupeshpoojary !
The docker-compose is basically what you need to function - it tells FeatureHub that it should be using Pubsub - but you will need your infrastructure (terraform, pulumi, crossplane, etc) to actually create the topics and subscriptions before hand. The only issue is that you need to ensure that it is allowed to create subscriptions because each Edge instance needs to create a brand new subscriber to the topic coming out of the Management Repository.
If we look at the config in the docker stuff, there is some things you would never use, e.g. cloudevents.pubsub.local.* is never used in "real" GCP because these exist to connect to the local pubsub and create the topics/subscriptions and pair them together. The topics listed in "cloudevents.pubsub.local.topics" you need to create in your infrastructure code, the "cloudevents.pubsub.local.sub-pairs" holds subcriber/topic pairs your infrastructure code needs to attach.
All of the rest of the configuration indicates what names are going to be used, so your infrastructure code will need to feed these into your values.yaml file in the k8s chart.
We use GCP Pub/Sub in our SaaS version and these are the settings we use as well.
# this indicates it will use pubsub
cloudevents.pubsub.enabled=true
# this is your GCP project name
cloudevents.pubsub.project=featurehub
cloudevents.pubsub.min-backoff-delay-seconds=20
# this is a comma separated list of keys, it will take each key and look for cloudevents.enricher.pubsub.channel.<key> to find the name of the subscription it needs to publish on (it expects a topic)
cloudevents.enricher.pubsub.channels=webhooks
cloudevents.enricher.pubsub.channel.webhooks=featurehub-enriched-events-webhooks
# this is "dacha2" pubsub has to have a subscription to listen for the original feature update coming from MR, it can't use the original name
cloudevents.enricher.pubsub.subscription-name=enricher-updates-sub
# this is "edge" - listening to the cloudevents.enricher.channel-name
cloudevents.enricher.pubsub.enriched-subscription-name=featurehub-enriched-events
cloudevents.edge-mr.pubsub.topic-name=featurehub-edge-updates
cloudevents.mr-edge.pubsub.topic-name=featurehub-mr-edge
# featurehub must dynamically create subscribers for Edge and Dacha2 as PubSub is not actually... PubSub
cloudevents.mr-edge.pubsub.subscription-prefix=featurehub-edge-listener
cloudevents.stats.pubsub.topic-name=featurehub-stats
cloudevents.mr-dacha2.pubsub.topic-name=featurehub-mr-dacha2
cloudevents.mr-dacha2.pubsub.subscription-prefix=featurehub-dacha2-listener
# don't use any of this in your configuration, however it informs you of how you need to connect them together
# start Google PubSub emulator config
# this is internal FeatureHub code that pre-creates these topics on the emulator, it shouldn't be used in production code
cloudevents.pubsub.local.host=pubsub:8075
cloudevents.pubsub.local.topics=featurehub-mr-dacha2,featurehub-mr-edge,featurehub-edge-updates,featurehub-stats,featurehub-enriched-events-webhooks,enriched-feature-data
# dacha2 and edge make their own subscribers
cloudevents.pubsub.local.sub-pairs=featurehub-edge-updates-mr-sub=featurehub-edge-updates,featurehub-enriched-events=featurehub-enriched-events-webhooks,enricher-updates-sub=featurehub-mr-edge
# end Google PubSub emulator config
rupeshpoojary
Hi team, we are integrating feature-hub into our ecosystem for all our product line, we wanted to integrate the google cloud hosted pub-sub into the feature-hub(hem chart) which we have deployed into our GKE cluster, currently the official documentation shows the implementation of a docker-compose based solution which has a pub-sub emulator as a container and passing the details of the pub-sub in the environment for edge and MR.
How do i integrate/pass the details for pub-sub that is hosted in GCP, Please help me how do i integrate it to connect to google cloud pub-sub.