hydradatabase / hydra

Hydra: Column-oriented Postgres. Add scalable analytics to your project in minutes.
https://www.hydra.so
Apache License 2.0
2.8k stars 73 forks source link

Fix an out-of-memory when inserting large values #253

Open japinli opened 5 months ago

japinli commented 5 months ago

The InsertTupleAndEnforceConstraints() constructed tuple on-the-fly, and on other reference, so we can safely set shouldFree to true for ExecStoreHeapTuple() to avoid duplicate memory allocation.

Here is my environment:

$ uname -a
Linux dell-dev 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       3.6Gi       2.0Gi       403Mi       2.0Gi       3.3Gi
Swap:             0B          0B          0B

The test case comes from regression test columnar_vaccum.sql.

[local]:3458609 postgres=# CREATE EXTENSION columnar ;
CREATE EXTENSION
[local]:3458609 postgres=# CREATE TABLE t (id serial, data text) USING columnar;
ERROR:  relation "t" already exists
[local]:3458609 postgres=# CREATE TABLE t1 (id serial, data text) USING columnar;
CREATE TABLE
[local]:3458609 postgres=# INSERT INTO t1 SELECT 1, repeat('a', 255000000);
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
The connection to the server was lost. Attempting reset: Failed.
: !?>

Here is system logs.

$ sudo journalctl -xe
Apr 02 18:20:59 dell-dev kernel: Out of memory: Killed process 3458609 (postgres) total-vm:3236140kB, anon-rss:2546688kB, file-rss:128kB, shmem-rss:14336kB, UID:1000 pgtables:5168k>
Apr 02 18:20:59 dell-dev systemd[1]: session-2.scope: A process of this unit has been killed by the OOM killer.
░░ Subject: A process of session-2.scope unit has been killed by the OOM killer.
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A process of unit @UNIT has been killed by the Linux kernel out-of-memory (OOM)
░░ killer logic. This usually indicates that the system is low on memory and that
░░ memory needed to be freed. A process associated with session-2.scope has been determined
░░ as the best process to terminate and has been forcibly terminated by the
░░ kernel.
░░
░░ Note that the memory pressure might or might not have been caused by session-2.scope.
Apr 02 18:21:01 dell-dev systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit NetworkManager-dispatcher.service has successfully entered the 'dead' state.
Apr 02 18:21:24 dell-dev sudo[3458984]:    japin : TTY=pts/15 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/journalctl -xe

As the issue pertains to the machine's memory, I am unable to give a way to test it.