search

Vantiq / vantiq-extension-sources

Contains source code for VANTIQ Extension sources as well as SDKs for building these sources.
Other
13 stars 6 forks source link

readme

vantiq-extension-sources

Note: Use Extension Sources requires VANTIQ Version 1.23 or later.

Please ensure that the VANTIQ instance with which you are working has been updated to this version or later.

Repository Overview

This repository contains the source code for various VANTIQ Extension Sources as well as SDKs for building these sources. VANTIQ Extension Sources provide a means by which VANTIQ sources can operate outside of (but related to) a VANTIQ installation.

VANTIQ Extension Sources support the creation of VANTIQ source types. Within a VANTIQ system, a source is the means by which the VANTIQ system communicates with other systems. Each VANTIQ source has a type or implementation. The source type determines defines the type(s) of system(s) with which the source can communicate. The VANTIQ system has a variety of source types included, but individual enterprises' situations will require connectivity to other systems. These other systems may be other commercial or standard systems, or they may be a home-grown system. In any case, the set of source types needs to be extensible.

VANTIQ Extension Sources provide this extensibility. Working with the VANTIQ system, an extension source can be constructed and operated outside the VANTIQ installation. This allows these sources to fully participate in VANTIQ edge or full installations.

Repository Contents & Conventions

The various directories within this repository represent either SDKs to build extension sources or extension sources themselves. By convention, SDKs will end with sdk.

Each directory will contain a README (or README.md) file that describes the contents, and either contains or directs the reader toward appropriate documentation. Each directory should contain a LICENSE (or LICENSE.md) file that outlines the license for the code as well as any packages used.

Bugs, Issues, Enhancement requests should be reported using the GitHub issues tab. In any such report or request, please make clear the extension source or SDK to which the issue applies. Including the top level directory in the issue's Title is most expeditious way of doing so. See issue #33 for an example.

Some extensions sources present will be written and supported by VANTIQ; others are contributed by other parties.

In general, branches other than master are considered development or experimental branches. Modulo any warnings or caveats in the README, things in the master branch should be considered usable.

The repository is set up to require reviews of pull requests (PRs) going into master. The primary purpose here is to maintain some level of consistency.

Extension Sources Overview

Extension sources are used to allow VANTIQ to talk to various other systems. Supported operations include

It is important to understand that the interpretation of these operations is determined completely by the extension source designer/developer.

For example, when sending a message to an extension source, it might be the case that depending upon some parameter in the message, the extension source might save the data, invoke some operation, or perform some other action that is appropriate for the data system that source represents.

Overall Architecture

The VANTIQ system maintains managers for each source type. All extension sources are defined with a base type of EXTENSION (see Defining a Type/Implementation). The extension source manager is responsible for the interaction between the extension sources and the remainder of the VANTIQ system.

Extension source connect to the VANTIQ system using websockets. To enable extension sources behind firewalls, etc., the extension source must initiate connection to the VANTIQ system. This connection must be authenticated before the extension source operations begin.

Once authenticated, there are six (6) operations that define the extension source protocol. These operations are as follows.

What follows is a general description of how things work. For details in Java, please see the Java SDK.

Messages

Note: When using the Java SDK, generally you may not need this level of detail.

Messages are sent using an ExtensionServiceMessage. This message contains the following properties.

Response messages adopt the HTTP Codes by convention. 1xx indicate continuing work, 2xx is success, 300 or greater is an error.

Operations

connectExtension Operation

When an extension source is activated in the VANTIQ server, it is not connected. Connection happens when the extension source contacts the VANTIQ system. At that time, it establishes a session with the VANTIQ system, and further interactions take place in the context of that session.

In a connectExtension operation, the following properties are to be provided.

reconnectRequired

The reconnectRequired message is sent when something has occurred that requires the source to reconnect. The most common case for this is an alteration in the source's configuration.

Regardless of the reason, when this message is received, the source should being the connection process again, starting at the connectExtension operation.

The websocket connection should still be up, and the connection authenticated.

For this operation, the following properties are provided.

Note that if you are using the Java SDK, you can set autoReconnect to true and this process will be handled automatically, provided that you have a configHandler set.

configureExtension

Upon connection, the first operation sent by VANTIQ will be the configureExtension operation. This operation will send the configuration document to the extension source, allowing it to perform whatever internal configuration it deems appropriate.

Properties provided in this operation are as follows.

Note: At this time, the only data required for extension sources is the configuration document. At some point in the future, one or more additional properties may be required. See note in Defining a Type/Implementation regarding configuration document structure.

publish Operation

The publish operation on the source results when a VANTIQ application publishes something to the source. As noted above, the interpretation of the publish operation is entirely up to the developer of the extension source. The publish operation has no explicit semantics required.

The publish operation provides the following properties.

notification Operation

The notification operation is data sent from the extension source to the VANTIQ system. It will appear in VANTIQ as a message arriving from a particular source. Within VANTIQ, a rule can be created to react to the message.

The notification message requires the following properties to be set.

query Operation

The query operation is the only request-response style invocation. We say request-response style invocation because it is not implemented as a request-response (implying synchronicity).

In this case, the EXTENSION source will receive a query operation. The following properties will be provided.

The query is taken from the WITH clause of the SELECT statement in VANTIQ. For example, given the query

SELECT * FROM SOURCE opcExample as row WITH
            queryStyle : "NodeId",
            nsu : "urn:eclipse:milo:hello-world",
            nodeIdentifier : "HelloWorld/ScalarTypes/String"
    {
       ...
    }

the object property would contain the JSON string

{
    "queryStyle" : "NodeId",
    "nsu" : "urn:eclipse:milo:hello-world",
    "nodeIdentifier" : "HelloWorld/ScalarTypes/String"
}

In response, the query will respond with a HTML-style response. This is a JSON message consisting of

If the status returned is an error, the reply address header & body should still be returned. Failure to return this will result in the query (on the VANTIQ) side waiting for the query to time out.

In the case of an error, the body should contain an array of objects, each of which contains the following properties:

An example might be

[
    { “messageCode”: “com.example.sources.mysource.invalidQuery”,
      “messageTemplate”: 
      “The query: {0} is invalid for sources of type {1}”,
      “parameters”: [“hi there”, “mysource”]
    }
]

Overall Protocol

The following diagram represents the overall flow of messages between an extension source and VANTIQ.

Extension Source Protocol Flow Diagram

Defining a Type/Implementation

Creating the Source Type

When creating an Extension source, you must first create the source type or implementation. This is done by defining a file containing a JSON document with the following properties.

To make the source type known to VANTIQ, use the VANTIQ CLI command load sourceimpls.

For example, suppose we are creating an extension source of type "EXAMPLE". To do so, we would first create the file exampleImpl.json with the following contents:

{
   "name" : "EXAMPLE",
   "baseType" : "EXTENSION",
   "verticle" : "service:extensionSource",
   "config" : {}
}

Then, we would use the VANTIQ CLI to load that source implementation.

vantiq -s <profileName> load sourceimpls exampleImpl.json

where <profileName> is replaced by the VANTIQ profile name, and exampleImpl.json identifies the file containing the definition to be loaded.

Source Implementation documents may also include baseConfigProperties- a list of strings that represent source configuration options for the connector in question. By including this field, the listed configuration properties will automatically be made available for configuration by a Vantiq Assembly. Typically, the source config options included in the baseConfigProperties are the required properties for any instance of the connector. The following example demonstrates how to include the 'baseConfigProperties':

{
   "name" : "EXAMPLE",
   "baseType" : "EXTENSION",
   "verticle" : "service:extensionSource",
   "config" : {},
   "baseConfigProperties": [
        "exampleConfigOption",
        "nested.config.example1",
        "nested.config.example2",
   ] 
}

Once that type is loaded, you can create a source of that type. This is done by first selecting the EXAMPLE type for the source,

Creating a source, step 1

and then providing the configuration.

Creating a source, step 2

Creating the configuration document

You are strongly encouraged to put the type-specific part of your configuration in a separate property within the configuration, preferably named to match your extension source type in some way. In the future, VANTIQ may need to include some general properties in this configuration, so keeping things separated now will help in future proofing.

In our example above, for our EXAMPLE extension source type, we have put the relevant configuration in the exampleConfig property within the configuration document.

Note that the format and contents of this document (at least the type-specific part) are to be understood by the extension source implementation. Specifically, this JSON document is the configuration document delivered as part of the configureExtension operation.

Developer Notes

To develop or build within this environment, we use gradle to build.

Some of the connectors require other software to build. Generally, these are things that are not available as downloads via gradle dependencies, or are things that are specific to each use. For example, building the JDBC connector requires that the developer building the connector provide the appropriate JDBC driver to make the database connection. In most cases, there is some environment variable that ends with _LOC (e.g. JDBC_DRIVER_LOC) that specifies the location of the driver.

To allow developers to work on the connector in which they are interested without downloading all possible options, we have parameterized the top-level settings.gradle file. The settings.gradle file determines the scope of the build. Connectors requiring a driver are included in the overall build only if the associated driver location environment variable is present. Otherwise, the connector is ignored for the build.

Building the connectors require a Java compiler. To build everything, you'll need at least Java 11. Most connectors are at the Java 8 level, but the Camel Component & Connectors require Java 11.

Building Docker Images

The connectors in this repository contain gradle tasks that can be used to build Docker Images for each connector and push those images to the developer's Docker Registry. In order to use these tasks, the developer must include the following configuration option in their gradle.properties file or on the gradle command line, along with some optional parameters:

Note that the repositoryName and, most likely, connectorSpecificInclusions, will be most appropriate on the gradle command line. Otherwise, the repositoryName will be used for all connectors built, and that will be overwritten by the last one built.

With the required properties in place, the tasks can then be executed as follows:

Build Image

From the root directory of this repo, run the following command (this example builds the JDBC Connector)

./gradlew jdbcSource:buildConnectorImage

Or, you can just run the following task:

./gradlew jdbcSource:buildImages

Push Image

From the root directory of this repo, run the following command (this example pushes the JDBC Connector image)

./gradlew jdbcSource:pushConnectorImage

Or, you can just run the following task:

./gradlew jdbcSource:pushImages

Deploying the connector images in the Vantiq IDE

Once you have built and published the docker image for a given connector (as described above), you can then deploy it into a Kubernetes Cluster directly from the Vantiq IDE. This process is described in its entirety here, including both the prerequisite Kubernetes Cluster setup, and an example that deploys the JDBC Connector.