search

toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.16k stars 180 forks source link

Exit code 137 (out of memory) in daemon mode #154

Closed jdposthuma closed 3 years ago

jdposthuma commented 3 years ago

PGSync version: 2.1.1

Postgres version: 12.x

Elasticsearch version: 7.10

Redis version: 6.0

Python version: 3.7, 3.8

Problem Description: When syncing a table (root has ~1.3 million records, child has 10 records) in daemon mode (pgsync -d). It works initially, but then bogs down and is killed with exit code 137 (out of memory). I've been able to replicate this on a container with 4GB of memory.

How to avoid this?

Error Message (if any): 

`Killed`
toluaina commented 3 years ago

Oh boy.

I have added support for streaming bulk index mode. This is however not the default. Streaming bulk is more lenient on memory usage. You can enable it by defining

ELASTICSEARCH_STREAMING_BULK=True

jdposthuma commented 3 years ago

This was wholly caused by the issues surrounding #164 and #162. Because those each provide more specific root causes, I suggest this be closed.

ELASTICSEARCH_STREAMING_BULK=True did not positively affect performance.

toluaina commented 3 years ago

Closing this