@tdengine/rest@3.0.2 REST API concurrently inserting enough rows throws error "connect ENOBUFS"

Bug描述

@tdengine/rest@3.0.2 REST API 并发迁移40张sqlite数据表，每张表有1千万条数据，当前仅使用了每张表的1000条数据进行并发插入，仅相当于40 * 1000的并发量，每次验证只有约15张表完成了1000条的插入任务，其余的都发生了报错：

超级表

CREATE STABLE s (f_ts timestamp, f_value bigint, f_open float, f_upper float, f_lower float, f_close float, f_price float, f_high float, f_low float) TAGS (f_code NCHAR(10), f_name NCHAR(10));

自动创建表插入数据

INSERT INTO db_quant.d_000659 USING db_quant.d (f_code, f_name) TAGS ('000659', '000659') (f_ts, f_price, f_amount, f_turnover, f_count, f_type) VALUES ('2024-01-24 11:12:21', 2.15, 2, 430, 1, 1)
INSERT INTO db_quant.d_000659 USING db_quant.d (f_code, f_name) TAGS ('000659', '000659') (f_ts, f_price, f_amount, f_turnover, f_count, f_type) VALUES ('2024-01-24 11:12:30', 2.15, 4, 860, 2, 1)

迁移脚本

TD.ts

import { connect, options, type TDengineCursor } from '@tdengine/rest'

let database: TDengineCursor

export async function TD () {
  if (!database) {
    options.path = '/rest/sql'
    options.host = 'td1'
    options.port = 6041
    const conn = connect(options)
    database = conn.cursor()
  }
  return database
}

index.ts

import { readDirectoryFiles, targetDir } from './read'
import Database from 'better-sqlite3'
import { TD } from './TD'
import path from 'path'

async function update (file) {
  const startTime = performance.now()
  console.log(`Updating database in ${file}`)
  const db = new Database(file)
  const stm = db.prepare('SELECT * FROM t_stocks_details limit 1000')
  const taos = await TD()
  const requestMap = stm.all().map(async (item: any) => {
    const insertSql = `INSERT INTO db_quant.d_${item.code} USING db_quant.d (f_code, f_name) TAGS ('${item.code}', '${item.code}') (f_ts, f_price, f_amount, f_turnover, f_count, f_type)
                           VALUES ('${[item.date.replace(/\//g, '-'), item.time].join(' ')}', ${item.price}, ${item.amount}, ${item.turnover}, ${item.count}, ${item.type})`
    console.log({ insertSql })
    return taos.query(insertSql)
  })
  const allSettled = await Promise.allSettled(requestMap)
  const result = JSON.stringify({
    db: path.basename(file),
    takes: performance.now() - startTime,
    allSettled
  })
  console.log(`Updated database in ${file}`, result)
  return result
}

async function main () {
  console.log({ targetDir })
  const files = await readDirectoryFiles(targetDir)
  return Promise.allSettled(files.map(async file => update(file)))
}

main().then(console.log).catch(console.error)

执行命令

ts-node index.ts

成功日志：

{"status":"fulfilled","value":{"_code":0,"_rows":1,"_column_meta":[{"columnName":"affected_rows","typeName":"INT","size":4}],"_data":[[1]]}},

失败日志：

{"status":"rejected","reason":{"message":"request to http://td1:6041/rest/sql failed, reason: connect ENOBUFS 172.30.192.35:6041 - Local (undefined:undefined)","type":"system","errno":"ENOBUFS","code":"ENOBUFS"}}

预期行为

完成全部迁移工作。

截屏日志

环境

客户端环境

设备名称    DESKTOP-E61ENE7
处理器 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz   4.00 GHz
机带 RAM  64.0 GB
设备 ID   9BD07667-B3AF-4D3D-81A1-CF8BDEF96AE0
产品 ID   00391-70000-00000-AA073
系统类型    64 位操作系统, 基于 x64 的处理器
笔和触控    没有可用于此显示器的笔或触控输入

版本  Windows 10 专业工作站版
版本号 22H2
安装日期    ‎2022/‎6/‎20
操作系统内部版本    19045.4170
体验  Windows Feature Experience Pack 1000.19054.1000.0

客户端nodejs环境

$ node -v
v18.13.0

服务器ubuntu环境

root@DESKTOP-E61ENE7:~# lshw
desktop-e61ene7
    description: Computer
    width: 64 bits
    capabilities: smp vsyscall32
  *-core
       description: Motherboard
       physical id: 0
     *-memory
          description: System memory
          physical id: 1
          size: 32GiB
     *-cpu
          product: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
          vendor: Intel Corp.
          physical id: 2
          bus info: cpu@0
          version: 6.94.3
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves md_clear flush_l1d arch_capabilities
          configuration: microcode=4294967295
     *-scsi:0
          description: SCSI storage controller
          product: Virtio console
          vendor: Red Hat, Inc.
          physical id: 3
          bus info: pci@1eab:00:00.0
          version: 01
          width: 64 bits
          clock: 33MHz
          capabilities: scsi msix bus_master cap_list
          configuration: driver=virtio-pci latency=64
          resources: iomemory:90-8f iomemory:90-8f iomemory:90-8f irq:0 memory:9ffe00000-9ffe00fff memory:9ffe01000-9ffe01fff memory:9ffe02000-9ffe02fff
        *-virtio0 UNCLAIMED
             description: Virtual I/O device
             physical id: 0
             bus info: virtio@0
             configuration: driver=virtio_console
     *-display
          description: 3D controller
          product: Microsoft Corporation
          vendor: Microsoft Corporation
          physical id: 4
          bus info: pci@5eaa:00:00.0
          version: 00
          width: 32 bits
          clock: 33MHz
          capabilities: bus_master cap_list
          configuration: driver=dxgkrnl latency=0
          resources: irq:0
     *-generic
          description: System peripheral
          product: Virtio file system
          vendor: Red Hat, Inc.
          physical id: 0
          bus info: pci@e5ea:00:00.0
          version: 01
          width: 64 bits
          clock: 33MHz
          capabilities: msix bus_master cap_list
          configuration: driver=virtio-pci latency=64
          resources: iomemory:e0-df iomemory:e0-df iomemory:c0-bf irq:0 memory:e00000000-e00000fff memory:e00001000-e00001fff memory:c00000000-dffffffff
        *-virtio1 UNCLAIMED
             description: Virtual I/O device
             physical id: 0
             bus info: virtio@1
             configuration: driver=virtiofs
     *-pnp00:00
          product: PnP device PNP0b00
          physical id: 5
          capabilities: pnp
          configuration: driver=rtc_cmos
     *-scsi:1
          physical id: 6
          logical name: scsi0
        *-disk:0
             description: SCSI Disk
             product: Virtual Disk
             vendor: Linux
             physical id: 0.0.0
             bus info: scsi@0:0.0.0
             logical name: /dev/sda
             version: 1.0
             size: 388MiB
             capabilities: extended_attributes large_files huge_files extents ext2 initialized
             configuration: ansiversion=5 filesystem=ext2 logicalsectorsize=512 sectorsize=512 state=clean
        *-disk:1
             description: Linux swap volume
             product: Virtual Disk
             vendor: Msft
             physical id: 0.0.1
             bus info: scsi@0:0.0.1
             logical name: /dev/sdb
             version: 1
             serial: 8ae81d30-4fa9-4f3e-9e6d-cda34fc9a8c7
             size: 8GiB
             capacity: 8GiB
             capabilities: swap initialized
             configuration: ansiversion=5 filesystem=swap logicalsectorsize=512 pagesize=4096 sectorsize=4096
        *-disk:2
             description: EXT4 volume
             product: Virtual Disk
             vendor: Linux
             physical id: 0.0.2
             bus info: scsi@0:0.0.2
             logical name: /dev/sdc
             logical name: /
             logical name: /mnt/wslg/distro
             logical name: /snap
             version: 1.0
             serial: 51b56668-88d9-4fdd-ac8d-befdbad73e8d
             size: 1TiB
             capabilities: journaled extended_attributes large_files huge_files dir_nlink recover 64bit extents ext4 ext2 initialized
             configuration: ansiversion=5 created=2024-02-22 01:41:55 filesystem=ext4 lastmountpoint=/distro logicalsectorsize=512 modified=2024-03-30 20:43:07 mount.fstype=ext4 mount.options=rw,relatime,discard,errors=remount-ro,data=ordered mounted=2024-03-30 20:43:07 sectorsize=4096 state=mounted
  *-usbhost:0
       product: USB/IP Virtual Host Controller
       vendor: Linux 5.15.146.1-microsoft-standard-WSL2 vhci_hcd
       physical id: 1
       bus info: usb@1
       logical name: usb1
       version: 5.15
       capabilities: usb-2.00
       configuration: driver=hub slots=8 speed=480Mbit/s
  *-usbhost:1
       product: USB/IP Virtual Host Controller
       vendor: Linux 5.15.146.1-microsoft-standard-WSL2 vhci_hcd
       physical id: 2
       bus info: usb@2
       logical name: usb2
       version: 5.15
       capabilities: usb-3.00
       configuration: driver=hub slots=8 speed=5000Mbit/s
  *-network
       description: Ethernet interface
       physical id: 3
       logical name: eth0
       serial: 00:15:5d:1c:44:71
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=hv_netvsc driverversion=5.15.146.1-microsoft-standard-W duplex=full firmware=N/A ip=172.30.192.35 link=yes multicast=yes speed=10Gbit/s
root@DESKTOP-E61ENE7:~#

TDengine版本

root@DESKTOP-E61ENE7:~# taos
Welcome to the TDengine Command Line Interface, Client Version:3.2.2.0
Copyright (c) 2023 by TDengine, all rights reserved.

  ********************************  Tab Completion  ************************************
  *   The TDengine CLI supports tab completion for a variety of items,                 *
  *   including database names, table names, function names and keywords.              *
  *   The full list of shortcut keys is as follows:                                    *
  *    [ TAB ]        ......  complete the current word                                *
  *                   ......  if used on a blank line, display all supported commands  *
  *    [ Ctrl + A ]   ......  move cursor to the st[A]rt of the line                   *
  *    [ Ctrl + E ]   ......  move cursor to the [E]nd of the line                     *
  *    [ Ctrl + W ]   ......  move cursor to the middle of the line                    *
  *    [ Ctrl + L ]   ......  clear the entire screen                                  *
  *    [ Ctrl + K ]   ......  clear the screen after the cursor                        *
  *    [ Ctrl + U ]   ......  clear the screen before the cursor                       *
  **************************************************************************************

Server is Community Edition.

taos>

taosdata / TDengine

@tdengine/rest@3.0.2 REST API concurrently inserting enough rows throws error "connect ENOBUFS" #25219

Bug描述

超级表

自动创建表插入数据

迁移脚本

执行命令

成功日志：

失败日志：

预期行为

截屏日志

环境

客户端环境

客户端nodejs环境

服务器ubuntu环境

TDengine版本