Support apache pulsar for buildcluster queue

[s-yoshika]

What happened: We are trying to introduce buildcluster feature to our environment. It requires message queue components and Screwdriver.cd supports only rabbitmq as the message queue for it. Also, the source code itself is hard-coded to use rabbitmq. We don't have full managed rabbitmq internally but have apache pulsar instead.

What you expected to happen: Scredriver.cd supports apache pulsar as buildcluster queue.

How to reproduce it:

[s-yoshika]

We started to create prototype just replace rabbitmq to pulsar in queue-service and buildcluster-queue-worker. we confirms it works simply to start build pod via pulsar. But we found pulsar node client doesn't enough to support full features of buildlcuster.

• pulsar node client doesn't support to get redelively count which is used in buildcluster-queue-worker to make build failed with too many redelivery
  • I posted the issue to pulsar node client: https://github.com/apache/pulsar-client-node/issues/98

[jithine]

cc @parthasl

[parthasl]

Cool @s-yoshika Yes, as of now queue service push and queue worker services are tied to Rabbitmq. We have plans to evaluate Apache Pulsar and AWS SQS options. Will keep this issue updated.

[jithine]

We will have to reevaluate this in context of https://github.com/screwdriver-cd/screwdriver/issues/2550 since we are looking a flexible queuing solution which can support multiple use cases. We are leaning towards Kafka for this, so not sure we should keep pulsar an option at all.

cc @pritamstyz4ever

[tk3fftk]

@jithine Would you tell me the reason why Kafka is selected to use? Are Apache Pulsar and AWS SQS (and probably just use RabbitMQ as Amazon MQ) are evaluated? I'd like to know some comparison details on GitHub issue if it's possible:eyes:

[pritamstyz4ever]

@tk3fftk I created this matrix with some of the features considered before selecting an Amazon MSK setup. We did proof-of-concept projects with different combinations of the queue and consumer and tested the solutions for scaling, high-throughput, ease of setup, security.

Consumer\	Queue
AWS Lambda	Self Managed RabbitMq	Amazon MQ	Self Managed Apache Kafka	AWS MSK	AWS SQS	AWS SNS
Polling Based	Yes	Yes	No	No	Yes	No
Pub/Sub Based	No	No	Yes	Yes	No	Yes
Lambda Event-Source Mapping Support	No	Yes	Yes	Yes	Yes	Yes
Cross-Account Support	No	No(consumer and producer need to be in the same account)	Yes	Yes	Yes	Yes
Cross account IAM Permissions required at both consumer and producer account(SD account)	No	NA	No	No	Yes	Yes
Subscription to the individual topic for each consumer	No	No	Yes	Yes	No	Yes(we need to create multiple Standard SNS topics Subscribed by lambda)
Outbound data from queue to cross-account VPC privately and securely	Yes(but needs VPC peering in a long polling approach which cannot scale to multiple account solution)	NA	Yes	Yes	No(data sent over the internet)	No(data sent over the internet)
Messaging Protocol	Yes	Yes	Yes	Yes	No uses HTTP. There will be a latency associated with SQS API HTTP calls	No uses HTTP. There will be a latency associated with SNS API HTTP calls
AutoScaling	Yes	Yes	Yes	Yes	Yes	Yes
Topics and Exchanges support	Yes	yes	Yes	Yes	No	Yes(partially)
Unidirectional data flow from Consumer	No	No	Yes	Yes	No	No
High Throughput	Yes	Yes	Yes	Yes	No	Yes

Considerations:

1. Multiple consumers are running in different AWS accounts/VPCs
2. Consumer resources will have a limited footprint and be easy to setup
3. Screwdriver Account will have no direct access to Consumer account and can only publish the message to the queue
4. Data flow will be unidirectional with a consumer-controlled approach reading messages from a secure queue endpoint