Pgwatch3 RPC Receivers

This repository contains the essential components to build your own Remote Sinks for Pgwatch v3. You can find the basic structure to create a Sink(or Receiver as we call it in this repo) which is basically a RPC Server that the pgwatch RPC Client interacts with.

The primary goal of this repository is to provide you with the building blocks to get started with your own implementations and also to provide some examples of places where measurements from pgwatch can be used. You can find some of our example implementations in the cmd folder.

Checkout PgWatch to get started with this project.

Architecture

The Remote Sinks work using RPC protocol. Some cool advantages of using RPC are:

• PgWatch is not concerned about the actual sink implementation. You can literally do anything with the measurements delievered to you by pgwatch and share messages per function call if requried.
• The sink implementations can be easily developed in Go, which has support for most of the storage formats out there and is pretty easy to write and work with.

The RPC receiver is treated as the default sink formats and no special changes are required in your pgwatch setup.

To use a RPC sink you can start pgwatch with the argument: --sink=rpc://<host>:<port>.

Running Custom RPC Sinks

By default the RPC Server for your sink listens at 0.0.0.0 with the specified port number. To start the any of the given receivers you can use:

# Parquet Receiver
go run ./cmd/parquet_receiver --port=<port_number_for_sink> --rootFolder=<location_on_disk>

# Kafka Receiver
go run ./cmd/kafka_prod_receiver --port=<port_number_for_sink> --kafkaHost=<host_address_of_kafka> --autoadd=<true/false>

Now once your receiver is up you can setup pgwatch as follows:

go run ./cmd/pgwatch --sources=<postgres://postgres@localhost:5432/postgres> --sink=rpc://<ip/hostname_of_your_sink>:<port_where_recv_is_listening>

Viola! You havea seemless integration between pgwatch and your custom sink. Try out our various implementations to get a feel of how these receivers feel with your custom pgwatch instances.

Developing custom sinks

To develop your own sinks you can utilize the template that we have used for each of the example receivers. You can copy the main.go file as it is and use the Receiver interface to write your own the UpdateMeasurements functions. You'll need to import the package github.com/destrex271/pgwatch3_rpc_server/sinks to utilize the components to build your own sinks.

You can refer our exmaple sinks to develop your own sinks or extend these for your usecases:

• CSV Receiver: Store measurements in CSV files
• Kafka Receiver: Stream measurements using kafka
• Parquet Receiver: Store measurements in parquet files
• Clickhouse Receiver: Store measurements in OLAP databases like Clickhouse for analytics
• Llama Receiver: Gain performance insights and recommendations based on your measurements using the power of tinyllama
• S3 Receiver: Store measurements in AWS S3

To get a simple to use template with no additional configurations, you can use the text receiver example provided along with the other receiver examples.