taosdata / TDengine

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios
https://tdengine.com
GNU Affero General Public License v3.0
23.46k stars 4.87k forks source link

参数绑定方式写入数据 bug #22366

Open habma1 opened 1 year ago

habma1 commented 1 year ago

Bug Description Java 使用参数绑定方式对单表每次写入一条数据,预计要写入100w条数据,在写入几十万条数据时,Java 程序进程无异常抛出便被强行终止

Screenshots 企业微信截图_7a55dde4-415d-4eec-a58c-30b77537be56

Environment

Additional Context 刚开始模拟对一百万张表每张表插入1000-50000条数据出现该问题,后发现在写入操作达到一定次数之后(几十万次)Java程序会被终止。复现问题简化版本程序 `package com.example;

import com.taosdata.jdbc.TSDBPreparedStatement;

import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.sql.Statement; import java.util.ArrayList;

public class StmtInsert { private static final int RECORDS_PER_TABLE = 600_000;

public static void main(String[] args) throws SQLException {
    insertData();
}

private static void insertData() throws SQLException {
    try (final Connection conn = getConnection()) {
        createTable(conn);
        final String psql = "INSERT INTO ? USING meters TAGS(?, ?) VALUES(?, ?, ?, ?)";

        try (final TSDBPreparedStatement pst = (TSDBPreparedStatement) conn.prepareStatement(psql)) {
            System.out.println("-> 开始插入数据 <-");

            int counter = 0;
            for (int i = 0; i < RECORDS_PER_TABLE; i++) {
                pst.clearBatch();

                final String line = getRowData(i);
                final String[] ps = line.split(",");

                // bind table name and tags
                pst.setTableName(ps[0]);
                pst.setTagString(0, ps[5]);
                pst.setTagInt(1, Integer.valueOf(ps[6]));

                // bind values
                pst.setTimestamp(0, toArray(Long.valueOf(ps[1])));
                pst.setFloat(1, toArray(Float.valueOf(ps[2])));
                pst.setInt(2, toArray(Integer.valueOf(ps[3])));
                pst.setFloat(3, toArray(Float.valueOf(ps[4])));
                pst.columnDataAddBatch();

                pst.columnDataExecuteBatch();

                counter++;
                if (counter % 5000 == 0) {
                    System.out.println("------------已插入数据 " + counter);
                }
            }

            System.out.println("执行完毕");
        }
    }
}

private static Connection getConnection() throws SQLException {
    System.out.println("-> 建立连接 <-");
    String jdbcUrl = "jdbc:TAOS://localhost:6030?user=root&password=taosdata";

    return DriverManager.getConnection(jdbcUrl);
}

private static void createTable(Connection conn) throws SQLException {
    System.out.println("-> 创建数据库和超级表 <-");
    try (Statement stmt = conn.createStatement()) {
        stmt.execute("DROP DATABASE IF EXISTS power");
        stmt.execute("CREATE DATABASE IF NOT EXISTS power KEEP 3650");
        stmt.executeUpdate("USE power");
        stmt.execute("CREATE STABLE IF NOT EXISTS meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) " +
                "TAGS (location BINARY(64), groupId INT)");
    }
}

private static String getRowData(int i) {
    long time = 1691478964333L + i;

    return "d0," + time + ",10.30000,219,0.31000,California.SanFrancisco,2";
}

private static <T> ArrayList<T> toArray(T v) {
    ArrayList<T> result = new ArrayList<>();
    result.add(v);

    return result;
}

}`

sangshuduo commented 1 year ago

监控一下内存占用是不是资源没释放被系统杀掉了?

zuodengchao commented 1 year ago

NSERT INTO ? USING meters TAGS

SQL拼接的方式插入,会不会有SQL注入的风险呢?taos-jdbcdriver是不是内部做了优化呢?

Xnollis commented 1 year ago

可能是taos.dll的资源泄漏没有释放引起的,已提过issue,但还未修复 https://github.com/taosdata/TDengine/issues/22599