Possible memory leak while running copy_to_distributed_table

[samay-sharma]

Today I was running some experiments with 5.0 and ran into a possible memory leak.

I was trying to load data into a hash partitioned table using the copy_to_distributed_table script. The process ran for sometime and then the postgres backend was killed. I observed the memory using top and it rose until it hit 100% and then got killed. The INSERTs wen't through successfully till that point and I see around 49k rows out of the total 1.1 million row dataset.

The steps to reproduce are:

CREATE TABLE; select master_create_distributed_table(); select master_create_worker_shards(); Run the copy to distributed table script

I ran it using the following command: /home/ec2-user/install_postgres95/bin/copy_to_distributed_table -C /data/sportradar/player_plays_combined.csv player_plays_combined_dist

I didn't run into that issue while loading data with 4.1.

I tried debugging the issue for couple of hours but couldn't find it. I also tried to run valgrind but failed to do it properly.

[samay-sharma]

We ran into this issue during a customer engagement.

[marcocitus]

I'm seeing it to when trying to copy customer_reviews_1998.csv into a table with 1000 shards.

The error I'm getting after partially completing the inserts is:

LOG:  server process (PID 50307) was terminated by signal 9: Killed
DETAIL:  Failed process was running: COPY  pg_temp.customer_reviews_1000_insert_proxy FROM STDIN with (OIDS false, FREEZE false, FORMAT csv)
LOG:  terminating any other active server processes

[marcocitus]

dmesg shows the following:

[5640841.845229] Out of memory: Kill process 50307 (postgres) score 10 or sacrifice child
[5640841.848111] Killed process 50307 (postgres) total-vm:1967384kB, anon-rss:601664kB, file-rss:17936kB