Varadhi : RESTBus - A message bus implementation with a REST interface

Introduction

Varadhi is the open source version of a similar FK internal project, a message bus implementation with a REST interface. Varadhi supports both P2P (point-to-point) and Publish-Subscribe messaging models. The idea behind RESTBus is to take your HTTP API stack and automatically transform them into service bus driven, queue/pub-sub, message oriented endpoints. The communication between Varadhi and endpoints will be on HTTP.

Varadhi is in active development and has been in production since last 10 years inside Flipkart as the backbone of async REST communication between various microservices sending billions of messages daily.

With Varadhi we hope others would be able to benefit from our learnings of a decade.

Full Documentation

See the Wiki for concepts, use cases, architecture details, API spec and other detailed information.

Build

./gradlew build test

Integration Tests

./gradlew copyDependencies copyConfigForE2E -x test

docker build . --file setup/docker/Dockerfile --tag varadhi.docker.registry/varadhi:latest --build-arg

docker compose --profile test -f setup/docker/compose.yml up -d --wait --wait-timeout 180

./gradlew testE2E

Dependencies

To provide the required functionality, Varadhi takes dependencies on various tech stack. Current development environment supports below option for these.

Run the following to start the below dependencies.

docker compose --profile dev -f setup/docker/compose.yml up -d --wait --wait-timeout 180

OR

Message Broker

Varadhi needs a message broker for message persistence and delivery. Apache Pulsar is used as underlying message broker. For the development environment users can use containerised Pulsar in a standalone mode. Details can be found here.

Persistent Store

For storing metadata about various Varadhi entities, a datastore is needed. Zookeeper is used as global datastore to persist json formatted entities. For the development environment, containerised Zookeeper can be used in a standalone mode. Details can be found here.

Varadhi Server

Finally, to run the Varadhi server, provide the custom zk & pulsar endpoints at server/src/main/resources/config.overrides and then do ./gradlew run

If you are using the dev profile to start the zk and pulsar, then simply do ./gradlew run. No config overrides are required.

k8s Deployment

cd setup/helm/varadhi

helm install varadhi-server . -f values/common.values.yaml -f values/local.server.values.yaml

helm install varadhi-controller . -f values/common.values.yaml -f values/local.controller.values.yaml

Modules

• entities: It has all the entities used by the spi module and varadhi apis.
• spi: It has all the interfaces related to persistence and messaging.
• pulsar: It contains messaging-spi implementation using Apache Pulsar.
• core: It contains all the core logic of Varadhi and is relied upon by various other sub-modules.
• server: It is the entry point of Varadhi. It houses all server related logic and binds together all other modules.

Releases

[ TBD ]

Roadmap

Status: Initial Review Completed.

See the Wiki page Roadmap for details.

Communications

[ TBD ]

Want to Contribute ?

Refer to Contributing.

You can also reachout to sahil.chachan@flipkart.com or k.dhruv@flipkart.com.

Bugs and Feedback

[ TBD ]

Blogs

Effective Failure Handling in Flipkart’s Message Bus

LICENSE

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0