search

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
37.07k stars 5.83k forks source link

Tiflash is out of memory on huge select with simple data transformation #32320

Closed 0xdeafbeef closed 2 months ago

0xdeafbeef commented 2 years ago

Bug Report

1. Minimal reproduce step

  1. Create table.

    create table messages
(
message_hash  varbinary(32)                                  not null
    primary key,
src_workchain tinyint                                        not null,
src_address   varbinary(32)                                  null,
dst_workchain tinyint                                        not null,
dst_address   varbinary(32)                                  null,
message_type  enum ('Internal', 'ExternalIn', 'ExternalOut') not null,
message_value bigint unsigned                                not null,
ihr_fee       bigint unsigned                                not null,
fwd_fee       bigint unsigned                                not null,
import_fee    bigint unsigned                                not null,
created_lt    bigint unsigned                                not null,
created_at    int unsigned                                   not null,
body          mediumblob                                     null,
state_init    mediumblob                                     null,
bounced       tinyint(1)                                     not null,
bounce        tinyint(1)                                     not null
);

  2. Fill it with 100m+ rows of data.

  3. Replicate table to 1 tiflash instanse.

  4. Execute query: 

    select
   created_at,
   concat(src_workchain, ':', lower(hex(src_address))) as src,
   concat(dst_workchain, ':', lower(hex(dst_address))) as dst,
   message_value
from messages
where created_at > 0
and message_value > 0;

2. What did you expect to see? (Required)

Exported data

3. What did you see instead (Required)

[2022-02-12 16:41:35] [HY000][1105] Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 1048576 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_351):
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 1048576 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_717), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 1048576 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_595), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 65536 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_190), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 1048576 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_264), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 16368 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_555), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 1048576 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_282), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 4194304 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_216), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 1048576 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_397), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 4194304 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_319), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 4194304 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_160), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 4194304 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_421), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 1048576 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_282), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 4194304 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_524), e.what() = DB::Exception:
[2022-02-12 16:41:35] Code: 0, e.displayText() = DB::Exception: Memory limit (for query) exceeded: would use 10.00 GiB (attempt to allocate chunk of 4194304 bytes), maximum: 10.00 GiB: (while reading from DTFile: /var/tidb/tiflash-9000/data/t_981/stable/dmf_625), e.what() = DB::Exception

4. What is your TiDB version? (Required)

Release Version: v5.2.2 Edition: Community Git Commit Hash: da1c21fd45a4ea5900ac16d2f4a248143f378d18 Git Branch: heads/refs/tags/v5.2.2 UTC Build Time: 2021-10-20 06:08:33 GoVersion: go1.16.4 Race Enabled: false TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 Check Table Before Drop: false

Forcing db to use tikv fixes the problem, but it's strange that it fails on tiflash.

dveeden commented 2 years ago

/component tiflash

0xdeafbeef commented 2 years ago

It's working correctly in 6.2

0xdeafbeef commented 2 years ago

@jebter, should I close the issue?

jebter commented 2 years ago

@zanmato1984 PTAL