logstash-plugins / logstash-integration-kafka

Kafka Integration for Logstash, providing Input and Output Plugins
Apache License 2.0
32 stars 60 forks source link

Adds schema registry's truststore and keystore settings #137

Closed andsel closed 1 year ago

andsel commented 1 year ago

Release notes

Add schema registry's setting for keystore and truststore.

What does this PR do?

This commit mainly exposes location password and type settings for schema registry's secret and key stores. It brings those configuration options, if available, and directly forward down to the Kafka's SerDes library and Manticore client. Introduces a script named setup_keystore_and_truststore.sh to setup keystore and truststore used in integration tests. Furthermore it reworks a little bit the bash scripts to setup Kafka and Schema Registry in integration test to avoid download of artifacts if they are already locally downloaded.

Why is it important/What is the impact to the user?

With this PR the user is able to configure trust and key stores to be used to connect and interact specifically with Schema Registry. These stores are different from the one configured for the Kafka client.

Checklist

Author's Checklist

How to test this PR locally

To test this PR a couple of security settings has to be done. The best way to setup Kafka and Schema registry with key store is to launch the kafka_test_setup.sh and ./start_schema_registry.sh. The script has the job to create keystore for schema registry, export it certificate and install on a truststore used by the client pipeline. All this security artifacts are available under the./tls_repository directory in plugin's clone.

  1. clone the repo and launch:
    ./kafka_test_setup.sh
    ./start_schema_registry.sh
  2. install a schema in the registry, for example
    curl --location --request POST 'http://localhost:8081/subjects/logstash_test-value/versions' \
    --header 'Content-Type: application/vnd.schemaregistry.v1+json' \
    --data-raw '{"schema": "{\"namespace\": \"io.confluent.examples.clients.basicavro\", \"type\": \"record\", \"name\": \"Payment\", \"fields\": [     {\"name\": \"id\", \"type\": \"string\"},     {\"name\": \"amount\", \"type\": \"double\"} ]}"}'
    • if you want to use the HTTPS, add to the curl
      --cacert /path/to/logstash-integration-kafka/tls/schema_reg_certificate.pem   
    • point to port 8083 and switch protocol tohttps
  3. in a local Logstash configure the Gemfile to point to the plugin:
    • adds in Gemfile
      gem "logstash-integration-kafka", :path => "/path/to/logstash-integration-kafka"
    • install it
      bin/logstash-plugin install --no-verify
  4. execute a pipeline with leverages the schema registry trust store setting:
    
    input {
    kafka {
    bootstrap_servers => ["localhost:9092"]
    topics => ["logstash_test"]
    group_id => "logstash"
    consumer_threads => 2
    schema_registry_url => "https://localhost:8083/"
    schema_registry_ssl_truststore_location => "/path/to/logstash-integration-kafka/tls/clienttruststore.jks"
    schema_registry_ssl_truststore_password => "changeit"
    key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
    }
    }

output { stdout { codec => rubydebug } }


5. verify no errors happen in log

## Related issues

<!-- Recommended
Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.

- Closes #123
- Relates #123
- Requires #123
- Superseeds #123
-->
- Closes #101 
- Relates https://github.com/confluentinc/schema-registry/pull/957

<!-- Recommended
Explain here the different behaviors that this PR introduces or modifies in this project, user roles, environment configuration, etc.

If you are familiar with Gherkin test scenarios, we recommend its usage: https://cucumber.io/docs/gherkin/reference/
-->

## Screenshots

<!-- Optional
Add here screenshots about how the project will be changed after the PR is applied. They could be related to web pages, terminal, etc, or any other image you consider important to be shared with the team.
-->

## Logs

<!-- Recommended
Paste here output logs discovered while creating this PR, such as stack traces or integration logs, or any other output you consider important to be shared with the team.
-->