TDengine is a highly efficient platform to store, query, and analyze time-series data. It is specially designed and optimized for IoT, Internet of Vehicles, Industrial IoT, IT Infrastructure and Application Monitoring, etc. It works like a relational database, such as MySQL.
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
kafka-connect-tdengine is a Kafka Connector for real-time data synchronization from Kafka to TDengine
This connector guarantees that records from the Kafka topic are delivered at least once.
The connector supports running one or more tasks. You can specify the number of tasks in the tasks.max configuration parameter. This can lead to huge performance gains when a large amount of data need to be parsed.
This connector supports multiple input formats, such as line,telnet,json. Users can explicitly define mappings for types in indices. When a mapping is not explicitly defined, TDengine can determine field types from data, however, some types such as timestamp and decimal, may not be correctly inferred. To ensure that the types are correctly inferred, the connector provides json format to infer a mapping from the schemas of Kafka messages.
To build a development version you'll need a recent version of Kafka as well as a set of upstream Confluent projects, which you'll have to build from their appropriate snapshot branch. See the Confluent quickstart for guidance on this process.
git clone git@github.com:taosdata/kafka-connect-tdengine.gitMaven using the standard lifecycle phases.mvn clean package.zip file will be produced in the /target/components/packages/ folder after the process has run..zip into a directory specified in the plugin.path of your connect worker's configuration properties file. See the Confluent instructions for further information on this step.This project is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE.