Prometheus对接TDengine显示HTTP status 400 Bad Request

现象

根据Prometheus | TDengine 文档 | 涛思数据 (taosdata.com) 文档说明配置Prometheus后查看数据时提示 remote_read: remote server http://10.0.0.12:6041/prometheus/v1/remote_write/prometheus returned HTTP status 400 Bad Request: proto: wrong wireType = 0 for field Labels

具体现象如下图

环境

操作系统信息

Ubuntu 20.04.4 LTS arm64架构

prometheus版本

root@prometheus01:~# /usr/local/bin/prometheus --version 
prometheus, version 2.45.3 (branch: HEAD, revision: a90afda1448aae879aa8bbf401cb4b27273a9723)
  build user:       root@e5cb2e00f325
  build date:       20240124-14:33:10
  go version:       go1.21.6
  platform:         linux/arm64
  tags:             netgo,builtinassets,stringlabels

尝试过的TDengine版本

TDengine-server-3.0.3.0-Linux-arm64

TDengine-server-3.2.2.0-Linux-arm64

提交Issue时TDengine版本

root@TDengine01:~# /usr/bin/taosd -V
community version: 3.0.3.0 compatible_version: 3.0.0.0
gitinfo: 4c6983592deede5ac17955f9dabc136ea6b84487
buildInfo: Built at 2023-03-08 17:58

集群安装步骤

配置/etc/hosts

10.0.0.12 TDengine01
10.0.0.14 TDengine02
10.0.0.15 TDengine03
10.0.0.16 TDengine04

下载安装包

#wget https://www.taosdata.com/assets-download/3.0/TDengine-server-3.2.2.0-Linux-arm64.tar.gz
wget https://www.taosdata.com/assets-download/3.0/TDengine-server-3.0.3.0-Linux-arm64.tar.gz

tar -zxvf TDengine-server-3.0.3.0-Linux-arm64.tar.gz
cd TDengine-server-3.0.3.0
./install.sh

修改TDengine集群配置文件

各节点配置文件中的firstEp 配置保持一致；fqdn为各节点主机名

vim /etc/taos/taos.cfg
firstEp                   TDengine01:6030
fqdn                      TDengine01
serverPort                6030

启动服务

systemctl restart taosd
systemctl restart taosadapter
systemctl status taosd
systemctl status taosadapter

创建集群以及创建prometheus库

taos
show dnodes;

CREATE DNODE "TDengine02:6030";
CREATE DNODE "TDengine03:6030";
CREATE DNODE "TDengine04:6030";
show dnodes;

create database prometheus;
use prometheus;
show stables;

修改root密码

# 修改root密码
SHOW USERS;
ALTER USER root PASS '****';

prometheus.yml

global:
  scrape_interval: 15s 
  evaluation_interval: 15s 
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["10.0.0.11:9100", "10.0.0.13:9100"]
remote_write:
  - url: "http://10.0.0.12:6041/prometheus/v1/remote_write/prometheus"
    basic_auth:
      username: root
      password: ****
    remote_timeout: 30s
    queue_config:
        capacity: 100000
        max_shards: 1000
        max_samples_per_send: 1000
        batch_send_deadline: 5s
        min_backoff: 30ms
        max_backoff: 100ms
remote_read:
  - url: "http://10.0.0.12:6041/prometheus/v1/remote_write/prometheus"
    basic_auth:
      username: root
      password: ****
    remote_timeout: 10s
    read_recent: true

日志

taosadapter_2024_02_19_00_00.log的日志内容如下：

02/19 11:01:15.030943 00004774 TAOS_ADAPTER info "| 202 |    7.650313ms |       10.0.0.11 | POST | /prometheus/v1/remote_write/prometheus " model=web req_id=0x%!x(<nil>)
02/19 11:01:25.001134 00004774 TAOS_ADAPTER info "| 202 |   24.601835ms |       10.0.0.11 | POST | /prometheus/v1/remote_write/prometheus " model=web req_id=0x%!x(<nil>)
02/19 11:01:30.013160 00004774 TAOS_ADAPTER info "| 202 |    9.393775ms |       10.0.0.11 | POST | /prometheus/v1/remote_write/prometheus " model=web req_id=0x%!x(<nil>)
02/19 11:01:35.062002 00004774 TAOS_ADAPTER info "| 202 |     230.847µs |       10.0.0.11 | POST | /prometheus/v1/remote_write/prometheus " model=web req_id=0x%!x(<nil>)
02/19 11:01:36.575192 00004774 TAOS_ADAPTER info "| 400 |    1.024584ms |       10.0.0.11 | POST | /prometheus/v1/remote_write/prometheus " model=web req_id=0x%!x(<nil>)
02/19 11:01:37.034934 00004774 TAOS_ADAPTER info "| 400 |      96.072µs |       10.0.0.11 | POST | /prometheus/v1/remote_write/prometheus " model=web req_id=0x%!x(<nil>)
02/19 11:01:37.197585 00004774 TAOS_ADAPTER info "| 400 |      71.241µs |       10.0.0.11 | POST | /prometheus/v1/remote_write/prometheus " model=web req_id=0x%!x(<nil>)
02/19 11:01:37.393727

我不太清楚产生状态码为400的原因

附件为详细的taosadapter_2024_02_19_00_00.log

taosadapter_2024_02_19_00_00.log

taosdata / TDengine

Prometheus对接TDengine显示HTTP status 400 Bad Request #24766

现象

环境

集群安装步骤

prometheus.yml

日志