search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
9.05k stars 1.82k forks source link

Old transactions are not terminated after FE/BE IPs are changed #52515

Open gmuraleekrishna opened 3 weeks ago

gmuraleekrishna commented 3 weeks ago

Steps to reproduce the behavior (Required)

Change FE and BE IPs and restart after some transactions are 'COMMITTED'.

Expected behavior (Required)

Old transactions are killed/terminated and not shown in show proc '/transactions/'.

Transaction from expired or old BEs/FEs should be terminated.

Real behavior (Required)

> show compute nodes;
+---------------+-------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+--------+---------------+----------+----------+-------------------+------------+------------+---------------------------------------------------+----------------+-------------+----------+-------------------+-----------+
| ComputeNodeId | IP          | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | ErrMsg | Version       | CpuCores | MemLimit | NumRunningQueries | MemUsedPct | CpuUsedPct | DataCacheMetrics                                  | HasStoragePath | StarletPort | WorkerId | WarehouseName     | TabletNum |
+---------------+-------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+--------+---------------+----------+----------+-------------------+------------+------------+---------------------------------------------------+----------------+-------------+----------+-------------------+-----------+
| 44030         | 172.19.0.31 | 9050          | 9060   | 8040     | 8060     | 2024-11-01 09:11:25 | 2024-11-01 09:41:40 | true  | false                | false                 |        | 3.3.4-56bcf6f | 32       | 95.586GB | 0                 | 0.89 %     | 0.0 %      | Status: Normal, DiskUsage: 0B/0B, MemUsage: 0B/0B | true           | 9070        | 14002    | default_warehouse | 270       |
| 44116         | 172.19.0.32 | 9050          | 9060   | 8040     | 8060     | 2024-11-01 09:01:13 | 2024-11-01 09:41:40 | true  | false                | false                 |        | 3.3.4-56bcf6f | 32       | 95.586GB | 0                 | 1.58 %     | 0.0 %      | Status: Normal, DiskUsage: 0B/0B, MemUsage: 0B/0B | true           | 9070        | 15001    | default_warehouse | 398       |
| 39900         | 172.19.0.33 | 9050          | 9060   | 8040     | 8060     | 2024-11-01 09:01:13 | 2024-11-01 09:41:40 | true  | false                | false                 |        | 3.3.4-56bcf6f | 32       | 95.586GB | 0                 | 6.27 %     | 3.1 %      | Status: Normal, DiskUsage: 0B/0B, MemUsage: 0B/0B | true           | 9070        | 14001    | default_warehouse | 396       |
+---------------+-------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+--------+---------------+----------+----------+-------------------+------------+------------+---------------------------------------------------+----------------+-------------+----------+-------------------+-----------+
> show proc '/transactions/10152/';
+----------+--------+
| State    | Number |
+----------+--------+
| running  | 1389   |
| finished | 0      |
+----------+--------+
> show proc '/transactions/10152/running';   
================ SKIPPED LINES ==============================
| 4081          | spark-e4910b6e-9697-46f3-a0c4-d9069478d1e2                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 15:18:12 | 2024-10-25 15:18:13 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4074          | spark-e04c75bd-f6f8-42b0-9eb2-2b74575af1d9                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 15:03:12 | 2024-10-25 15:03:14 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4063          | spark-612e97bf-9809-4f84-8354-65555a66aea9                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 14:48:12 | 2024-10-25 14:48:14 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4053          | spark-10fd6df2-0a12-4ff1-9ffa-7be2c8b7df7e                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 14:33:12 | 2024-10-25 14:33:14 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4045          | spark-5b2f8ed9-1a8a-4b26-ae55-f9d4071dbfb5                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 14:18:12 | 2024-10-25 14:18:13 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4035          | spark-cc82ad4f-1bd6-4aa4-bfaa-6fe50288be93                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 14:03:12 | 2024-10-25 14:03:14 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4027          | spark-7da8523d-a019-476e-a28c-7bb6311227bc                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 13:48:12 | 2024-10-25 13:48:15 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4019          | spark-a112f8fd-144a-4532-8763-6d093eba217c                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 13:33:12 | 2024-10-25 13:33:14 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4011          | spark-0bf608ee-0973-4ae8-b88f-296b0fd09017                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 13:18:12 | 2024-10-25 13:18:14 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 4003          | spark-e30b555f-7d25-4e45-9038-7faacdf727b8                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 13:03:12 | 2024-10-25 13:03:13 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 3995          | spark-03c09d45-9661-4481-95ff-051faaae27c2                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 12:48:11 | 2024-10-25 12:48:13 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 3982          | spark-99ab03b5-4b2a-4681-9c3a-477a5a36f418                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 12:33:11 | 2024-10-25 12:33:13 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 3966          | spark-3b31e958-95a5-4304-ab3d-f0a8ead5379a                         | BE: 172.19.0.8  | COMMITTED         | BACKEND_STREAMING | 2024-10-25 12:18:11 | 2024-10-25 12:18:13 | NULL        | NULL       |        | 0                  | -1         | 900000    |        |
| 3912          | COMPACTION_10152-10154-10832-1729829332103                         | FE: 172.19.0.5  | COMMITTED         | LAKE_COMPACTION   | 2024-10-25 12:08:52 | 2024-10-25 12:18:06 | NULL        | NULL       |        | 0                  | -1         | 86400000  |        |
+---------------+--------------------------------------------------------------------+-----------------+-------------------+-------------------+---------------------+---------------------+-------------+------------+--------+--------------------+------------+-----------+--------+

StarRocks version (Required)

3.3.4-56bcf6f

tracymacding commented 2 weeks ago

How do you change your ip？

gmuraleekrishna commented 2 weeks ago

Hi @tracymacding. I am deploying Starrocks using docker containers. Hence, when I start the containers in different order the IP of the FE's and CNs can change. But the transactions are retained with old IP references.

wyb commented 2 weeks ago

any errors related the running transactions in leader fe log?