= Knative connectors
Knative eventing connectors based on https://camel.apache.org/camel-kamelets/[Apache Camel Kamelets].
Each connector project is able to act as a source or sink for the Knative eventing broker.
The projects create container images that get referenced in a Knative IntegrationSource or IntegrationSink custom resource.
== Create new connectors
You can use a Maven archetype to generate new connector modules in this project. The archetype should give you a good start into the new connector project as it generates the basic folder structure and some source files.
You need to build the Maven archetype locally first before using it for the first time:
The command creates the Maven archetype artifact on your local machine, so you can use it in the Maven archetype:generate command:
NOTE: The Maven archetype archetype-source creates a new IntegrationSource connector. Please use archetype-sink to create a new IntegrationSink connector project.
This command starts the archetype generation in interactive mode.
In interactive mode the prompt asks you to provide some more information such as the Kamelet name that you want to use in this kn-connector source.
Also, you may want to customize the Apache Camel component name that is used in the Kamelet (e.g. aws-s3 as Kamelet name and aws2-s3 as Apache Camel component name).
After the archetype:generate command has finished you will find a new project folder that is ready to be built with Maven.
You can also avoid the interactive mode when specifying all input parameters:
Each connector requires a set of Camel Quarkus dependencies in the Maven POM.
After the 1st build of the new Maven project you can review a generated list of dependencies in target/metadata/dependencies.json.
This generated info gives you a list of dependencies that you should consider adding to the Maven project POM.
Also, the metadata folder includes the information about supported Kamelet properties (target/metadata/properties.json).
You may review this list of properties and decide which one of these the IntegrationSource CRD should include in its spec.
The generated info includes all property specifications such as enumeration constraints or formats.
NOTE: Each connector project generates an AsciiDoc file in its project root folder (properties.adoc) which holds a list of all supported Kamelet properties. This includes the environment variable name that you may set as part of a connector deployment in order to overwrite one of these Kamelet properties
NOTE: The connector project contains a generated properties specification (properties.yaml) that you can use as a base for adding the connector to the IntegrationSource CRD specification as typed API.
The generation of the info and spec files is done automatically when the connector Maven project is built.
== Build the container image
Each kn-connector project uses Quarkus in combination with Apache Camel, Kamelets and Maven as a build tool.
You can use the following Maven commands to build the container image.
The container image uses the project version and image group defined in the Maven POM.
You can customize the image group with -Dquarkus.container-image.group=my-group.
By default, the container image looks like this:
The project leverages the Quarkus Kubernetes and container image extensions so you can use Quarkus properties and configurations to customize the resulting container image.
See these extensions for details:
== Push the container image
Pushes the image to the image registry defined in the Maven POM. The default registry is https://quay.io/[quay.io].
You can customize the registry with -Dquarkus.container-image.registry=localhost:5001 (e.g. when connecting to local Kind cluster).
In case you want to connect with a local cluster like Kind or Minikube you may also need to set -Dquarkus.container-image.insecure=true.
== Kubernetes manifest
The build produces a Kubernetes manifest in (target/kubernetes/kubernetes.yml).
This manifest holds all resources required to run the application on your Kubernetes cluster.
You can customize the Kubernetes resources in src/main/kubernetes/kubernetes.yml.
This is being used as a basis and Quarkus will generate the final manifest in target/kubernetes/kubernetes.yml during the build.
== Deploy to Kubernetes
You can deploy the application to Kubernetes with:
This deploys the application to the current Kubernetes cluster that you are connected to (e.g. via kubectl config set-context --current and kubectl config view --minify).
You may change the target namespace with -Dquarkus.kubernetes.namespace=my-namespace.
== Configuration
Each Kamelet defines a set of properties. The user is able to customize these properties when running a connector deployment.
=== Environment variables
You can customize the properties via environment variables on the deployment:
The environment variables that overwrite properties on the Kamelet source follow a naming convention:
The name represents the name of the Kamelet as defined in the https://camel.apache.org/camel-kamelets/[Kamelet catalog].
The environment variables may be set as part of the Kubernetes deployment or as an alternative on the Knative ContainerSource:
apiVersion: sources.knative.dev/v1 kind: ContainerSource metadata: name: kamelet-source namespace: knative-samples spec: template: spec: containers:
You can also set the environment variable on the running deployment:
=== ConfigMap and Secret refs
You may also mount a configmap/secret to overwrite Kamelet properties with values from the configmap/secret resource.
As the Kamelet properties are configured viw environment variables on the ContainerSource you can also use values referencing a configmap or secret.
apiVersion: sources.knative.dev/v1 kind: ContainerSource metadata: name: kamelet-source namespace: knative-samples spec: template: spec: containers:
The example above references a secret called my-secret and loads the keys aws.s3.accessKey and aws.s3.secretKey.
=== ConfigMap/Secret via Apache Camel property function
You can also load and reference the values of the configmap/secret in the environment variables following Apache Camel expressions:
Given a configmap named my-aw-s3-source-config in Kubernetes that has two entries:
You can reference the values of the configmap in the environment variables like this:
The configmap property function in Apache Camel follows this general syntax:
This means you can also set a default value in case the configmap should not be present.
The configmap and secret based configuration requires to add a volume and volume-mount configuration to the connector deployment.
apiVersion: apps/v1 kind: Deployment metadata: name: timer-source spec: selector: matchLabels: app.kubernetes.io/name: timer-source app.kubernetes.io/version: 1.0-SNAPSHOT template: spec: containers:
Camel is able to resolve the configmap mount path given in the volume mount.
The mount path is configurable via application.properties in the connector project:
The mount path configured on the Kubernetes deployment should match the configuration in the application.properties.
Instead of settings the mount paths statically in the application.properties you can also set these via environment variables on the
Kubernetes deployment.
The same mechanism applies to mounting and configuring Kubernetes secrets. The syntax for referencing a secret value via Apache Camel property function is as follows:
This means you can overwrite Kamelet properties with the values from the secret like this:
== CloudEvent attributes
== Source attributes
Each connector produces/consumes events in CloudEvent data format. The connector uses a set of default values for the CloudEvent attributes:
You can customize the CloudEvent attributes with setting environment variables on the deployment.
You can set the CE_OVERRIDE attributes on a running deployment.
You may also use the SinkBinding K_CE_OVERRIDES environment variable set on the deployment.
=== Sink attributes
Each connector sink consumes events in CloudEvent data format. By default, the connector receives all events on the Knative broker.
You may want to specify filters on the CloudEvent attributes so that the connector selectively consumes events from the broker. Just configure the Knative trigger to filter based on attributes:
The trigger for example filters the events by its type ce-type=dev.knative.connector.event.timer.
== Secure Knative transports
Knative brokers may use TLS encrypted transport options as described in https://knative.dev/docs/eventing/features/transport-encryption/#overview
This means that Event producers need to use proper SSL authentication to connect to Https Knative broker endpoints with cluster-internal CA certificates.
The IntegrationSource may use a volume mount with the cluster-internal CA certificates being injected.
The integration source needs to enable the SSL client via environment variables and set the path to the injected CA certs and PEM files:
This enables the SSL options on the Http client that connects with the broker endpoint. The SSL client support provides these environment variables:
|=== |EnvVar |Description
|CAMEL_KNATIVE_CLIENT_SSL_ENABLED
|Enable/disable SSL options on the Http client. Default value is false.
|CAMEL_KNATIVE_CLIENT_SSL_VERIFY_HOSTNAME
|Enable/disable hostname verification. Default value is true.
|CAMEL_KNATIVE_CLIENT_SSL_KEY_PATH |Path to the key store options configuring a list of private key and its certificate based on Privacy-enhanced Electronic Email (PEM) files.
|CAMEL_KNATIVE_CLIENT_SSL_KEY_CERT_PATH |Paths to client key CA certificates. Supports multiple paths as comma delimited String value.
|CAMEL_KNATIVE_CLIENT_SSL_KEYSTORE_PATH |Java keystore (.jks) or (.p12) path as an alternative to using PEM files.
|CAMEL_KNATIVE_CLIENT_SSL_KEYSTORE_PASSWORD |Keystore password. Value can be set via secretKeyRef.
|CAMEL_KNATIVE_CLIENT_SSL_TRUST_CERT_PATH |Paths to trust CA certificates. Supports multiple paths as comma delimited String value.
|CAMEL_KNATIVE_CLIENT_SSL_TRUSTSTORE_PATH |Java truststore (.jks) or (.p12) path as an alternative to using PEM files.
|CAMEL_KNATIVE_CLIENT_SSL_TRUSTSTORE_PASSWORD |Truststore password. Value can be set via secretKeyRef.
|===
As you can see the SSL client support provides multiple ways to configure keystore and truststore options. It is recommended to set keystore/truststore passwords vie secretKeyRef on the IntegrationSource spec. When no truststore configuration is given the SSL client support defaults to using trust all options.
== Dependencies
The required Camel dependencies need to be added to the Maven POM before building and deploying. You can use one of the Kamelets available in the https://camel.apache.org/camel-kamelets/[Kamelet catalog] as a source or sink in this connector.
Typically, the Kamelet is backed by a Quarkus Camel extension component dependency that needs to be added to the Maven POM. The Kamelets in use may list additional dependencies that we need to include in the Maven POM.
== Custom Kamelets
Creating a new kn-connector project is very straightforward. You may copy one of the sample projects and adjust the reference to the Kamelets.
Also, you can use the Camel JBang kubernetes export functionality to generate a Maven project from a given Pipe YAML file.
This generates a Maven project that you can use as a starting point for the kn-connector project.
The connector is able to reference all Kamelets that are part of the https://camel.apache.org/camel-kamelets/[default Kamelet catalog].
In case you want to use a custom Kamelet, place the *.kamelet.yaml file into src/main/resources/kamelets.
The Kamelet will become part of the built container image, you can just reference the Kamelet in the Pipe YAML file as a source or sink.
== More configuration options
For more information about Apache Camel Kamelets and their individual properties see https://camel.apache.org/camel-kamelets/.
For more detailed description of all container image configuration options please refer to the Quarkus Kubernetes extension and the container image guides: