Terraform Provider for Pulsar

Authored by [StreamNative], this repository includes the source code and documentation for a Terraform provider for managing Apache Pulsar entities such as clusters, tenants, namespaces, topics, sources, and sinks.

Prerequisites

• Terraform 0.12.0 or later
• Go 1.16 or later (to build the provider plugin)

Installation

• From terraform registry

  This provider has been published in the Terraform Registry:

  terraform {
  required_providers {
    pulsar = {
      version = "0.1.3"
      source = "registry.terraform.io/streamnative/pulsar"
    }
  }
  }

• From source code

  • Clone this repository and cd into the directory
  • Run make build, it will generate a binary file named terraform-provider-pulsar
  • Copy this terraform-provider-pulsar binary file to your terraform plugin directory based on your OS:

• Run make build-dev, it will build the binary and copy it to the plugin directory automatically.

Version Compatibility

The compatibility matrix from terraform-provider-pulsar to pulsar as shown below:

Note: It's not strictly tested. Please report issues when you encounter any bugs.

Using the Provider

Configurations

Example

Example provider with apache pulsar cluster, running locally with authentication disabled.

provider "pulsar" {
  web_service_url = "http://localhost:8080"
  token           = "my_auth_token"
}

provider "pulsar" {
  web_service_url           = "https://localhost:8443"
  tls_trust_certs_file_path = "./ca.pem"
  tls_key_file_path         = "./key.pem"
  tls_cert_file_path        = "./cert.pem"
}

Properties

Resources

pulsar_cluster

A resource for managing Apache Pulsar Clusters, can update various properties for a given cluster.

Example

provider "pulsar" {
  web_service_url = "http://localhost:8080"
}

resource "pulsar_cluster" "my_cluster" {
  cluster = "eternals"

  cluster_data {
    web_service_url    = "http://localhost:8080"
    broker_service_url = "http://localhost:6050"
    peer_clusters      = ["skrulls", "krees"]
  }
}

Properties

pulsar_tenant

A resource for managing Pulsar Tenants, can update admin roles and allowed clusters for a tenant.

Example

provider "pulsar" {
  web_service_url = "http://localhost:8080"
}

resource "pulsar_tenant" "my_tenant" {
  tenant           = "thanos"
  allowed_clusters = ["pulsar-cluster-1"]
  admin_roles      = ["godmode"]
}

Properties

pulsar_namespace

A resource for creating and managing Apache Pulsar Namespaces, can update various properties for a given namespace.

Example

provider "pulsar" {
  web_service_url = "http://localhost:8080"
}

resource "pulsar_cluster" "test_cluster" {
  cluster = "skrulls"

  cluster_data {
    web_service_url    = "http://localhost:8080"
    broker_service_url = "http://localhost:6050"
    peer_clusters      = ["standalone"]
  }
}

resource "pulsar_tenant" "test_tenant" {
  tenant           = "thanos"
  allowed_clusters = [pulsar_cluster.test_cluster.cluster, "standalone"]
}

resource "pulsar_namespace" "test" {
  tenant    = pulsar_tenant.test_tenant.tenant
  namespace = "eternals"

  enable_deduplication = true

  // If defined partially, plan would show difference
  // however, none of the mising optionals would be changed
  namespace_config {
    anti_affinity                  = "anti-aff"
    max_consumers_per_subscription = "50"
    max_consumers_per_topic        = "50"
    max_producers_per_topic        = "50"
    message_ttl_seconds            = "86400"
    replication_clusters           = ["standalone"]
  }

  dispatch_rate {
    dispatch_msg_throttling_rate  = 50
    rate_period_seconds           = 50
    dispatch_byte_throttling_rate = 2048
  }

  subscription_dispatch_rate {
    dispatch_msg_throttling_rate  = 50
    rate_period_seconds           = 50
    dispatch_byte_throttling_rate = 2048
  }

  retention_policies {
    retention_minutes    = "1600"
    retention_size_in_mb = "10000"
  }

  backlog_quota {
    limit_bytes  = "10000000000"
    limit_seconds = "-1"
    policy = "consumer_backlog_eviction"
    type = "destination_storage"
  }

  persistence_policies {
    bookkeeper_ensemble                   = 1   // Number of bookies to use for a topic, default: 0
    bookkeeper_write_quorum               = 1   // How many writes to make of each entry, default: 0
    bookkeeper_ack_quorum                 = 1   // Number of acks (guaranteed copies) to wait for each entry, default: 0
    managed_ledger_max_mark_delete_rate   = 0.0 // Throttling rate of mark-delete operation (0 means no throttle), default: 0.0
  }

  permission_grant {
    role    = "some-role"
    actions = ["produce", "consume", "functions"]
  }
}

Properties

namespace_config nested schema

The schema_compatibility_strategy can take the following values:

• AutoUpdateDisabled
• Backward
• Forward
• Full
• AlwaysCompatible
• BackwardTransitive
• ForwardTransitive
• FullTransitive

pulsar_topic

A resource for creating and managing Apache Pulsar Topics, can update partitions for a given partition topic.

Example

provider "pulsar" {
  web_service_url = "http://localhost:8080"
}

resource "pulsar_topic" "sample-topic-1" {
  tenant     = "public"
  namespace  = "default"
  topic_type = "persistent"
  topic_name = "partition-topic"
  partitions = 4                     # partitions > 0 means this is a partition topic

  permission_grant {
    role    = "some-role"
    actions = ["produce", "consume", "functions"]
  }

  retention_policies {
    retention_time_minutes = 1600
    retention_size_mb = 20000
  }
}

resource "pulsar_topic" "sample-topic-2" {
  tenant     = "public"
  namespace  = "default"
  topic_type = "persistent"
  topic_name = "non-partition-topic"
  partitions = 0                     # partitions = 0 means this is a non-partition topic

  permission_grant {
    role    = "some-role"
    actions = ["produce", "consume", "functions"]
  }

  retention_policies {
    retention_time_minutes = 1600
    retention_size_mb = 20000
  }
}

Properties

pulsar_function

A resource for creating and managing Apache Pulsar Functions.

Example

provider "pulsar" {
  web_service_url = "http://localhost:8080"
}

resource "pulsar_function" "function-1" {
  provider = pulsar

  name = "function1"
  tenant = "public"
  namespace = "default"
  parallelism = 1

  processing_guarantees = "ATLEAST_ONCE"

  jar = "/Downloads/apache-pulsar-2.10.1/examples/api-examples.jar"
  classname = "org.apache.pulsar.functions.api.examples.WordCountFunction"

  inputs = ["public/default/input1", "public/default/input2"]

  output = "public/default/test-out"

  subscription_name = "test-sub"
  subscription_position = "Latest"
  cleanup_subscription = true
  skip_to_latest = true
  forward_source_message_property = true
  retain_key_ordering = true
  auto_ack = true
  max_message_retries = 101
  dead_letter_topic = "public/default/dlt"
  log_topic = "public/default/lt"
  timeout_ms = 6666

  secrets = jsonencode(
  {
    "SECRET1": {
       "path": "sectest"
       "key": "hello"
    }
  })
  custom_runtime_options = jsonencode(
  {
      "env": {
          "MILLBRAE": "HILLCREST"
      }
  })
}

Properties

pulsar_source

A resource for creating and managing Apache Pulsar Sources.

Example

provider "pulsar" {
  web_service_url = "http://localhost:8080"
}

resource "pulsar_source" "source-1" {
  provider = pulsar

  name = "source-1"
  tenant = "public"
  namespace = "default"

  archive = "testdata/pulsar-io/pulsar-io-file-2.10.4.nar"

  destination_topic_name = "source-1-topic"

  processing_guarantees = "EFFECTIVELY_ONCE"

  configs = "{\"inputDirectory\":\"opt\"}"

  cpu = 2
  disk_mb = 20480
  ram_mb = 2048
}

Properties

pulsar_sink

A resource for creating and managing Apache Pulsar Sinks.

Example

provider "pulsar" {
  web_service_url = "http://localhost:8080"
}

resource "pulsar_sink" "sample-sink-1" {
  provider = pulsar

  name = "sample-sink-1"
  tenant = "public"
  namespace = "default"
  inputs = ["sink-1-topic"]
  subscription_position = "Latest"
  cleanup_subscription = false
  parallelism = 1
  auto_ack = true

  processing_guarantees = "EFFECTIVELY_ONCE"

  archive = "testdata/pulsar-io/pulsar-io-jdbc-postgres-2.10.4.nar"
  configs = "{\"jdbcUrl\":\"jdbc:clickhouse://localhost:8123/pulsar_clickhouse_jdbc_sink\",\"password\":\"password\",\"tableName\":\"pulsar_clickhouse_jdbc_sink\",\"userName\":\"clickhouse\"}"
}

Properties

Importing existing resources

All resources could be imported using the standard terraform way.

Example

terraform import pulsar_cluster.standalone standalone

Testing the Provider

• Change directory to the project </path/to/provider/terraform-provider-pulsar>
• In order to test the provider, you can run make test
• In order to run the full suite of Acceptance tests, run make testacc

Note: Acceptance tests create real resources, and often cost money to run.

Contributing

Contributions are warmly welcomed and greatly appreciated! Please read the contribution guidelines for more details.

Issues on GitHub are intended to be related to bugs or feature requests with the provider codebase. For a list of community resources to ask questions about Terraform, see https://www.terraform.io/docs/extend/community/index.html.

Licence

This library is licensed under the terms of the Apache License 2.0 and may include packages written by third parties which carry their own copyright notices and license terms.

About StreamNative

Founded in 2019 by the original creators of Apache Pulsar, StreamNative is one of the leading contributors to the open-source Apache Pulsar project. We have helped engineering teams worldwide make the move to Pulsar with StreamNative Cloud, a fully managed service to help teams accelerate time-to-production.