search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.26k stars 5.84k forks source link

Lightning encounters OOM when importing Parquet files #49234

Open lyzx2001 opened 11 months ago

lyzx2001 commented 11 months ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Lightning imports 100 parquet files, each has a size of 278.1MB and consists of 1,000,000 rows.

2. What did you expect to see? (Required)

No error.

3. What did you see instead (Required)

fatal error: runtime: out of memory

4. What is your TiDB version? (Required)

v7.5.0

ti-chi-bot[bot] commented 11 months ago

@lyzx2001: The label(s) sig/affects-7.5 cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/pingcap/tidb/issues/49234#issuecomment-1849380127): >/sig affects-7.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
lyzx2001 commented 11 months ago

/label affects-7.5

lyzx2001 commented 11 months ago

/label affects-7.4 affects-7.3

ti-chi-bot[bot] commented 11 months ago

@lyzx2001: The label(s) sig/affects-7.5 cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/pingcap/tidb/issues/49234#issuecomment-1849380127): >/sig affects-7.5 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
lyzx2001 commented 11 months ago

/label affects-7.4

ti-chi-bot[bot] commented 11 months ago

@lyzx2001: The label(s) affects-7.4 cannot be applied. These labels are supported: fuzz/sqlancer, challenge-program, compatibility-breaker, first-time-contributor, contribution, good first issue, correctness, duplicate, proposal, security, ok-to-test, needs-ok-to-test, needs-more-info, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5.

In response to [this](https://github.com/pingcap/tidb/issues/49234#issuecomment-1849383882): >/label affects-7.4 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
lyzx2001 commented 11 months ago

/label affects-7.1

lyzx2001 commented 11 months ago

/label affects-6.5

lyzx2001 commented 11 months ago

/label affects-6.1

lyzx2001 commented 11 months ago

/label affects-5.4

lyzx2001 commented 11 months ago

/label affects-5.3

ti-chi-bot[bot] commented 11 months ago

@lyzx2001: The label(s) affects-5.3 cannot be applied. These labels are supported: fuzz/sqlancer, challenge-program, compatibility-breaker, first-time-contributor, contribution, good first issue, correctness, duplicate, proposal, security, ok-to-test, needs-ok-to-test, needs-more-info, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5.

In response to [this](https://github.com/pingcap/tidb/issues/49234#issuecomment-1849389622): >/label affects-5.3 Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.