toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.1k stars 172 forks source link

Crashes on parsing big double values #527

Closed bartoszpijet closed 3 weeks ago

bartoszpijet commented 4 months ago

PGSync version: Version: 3.1.0

Postgres version: PostgreSQL 15.1 (Ubuntu 15.1-1.pgdg20.04+1) on aarch64-unknown-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit

Elasticsearch/OpenSearch version: 2.11.1

Redis version: Redis server v=6.0.16 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=a3fdef44459b3ad6

Python version: Python 3.10.12

Problem Description:

Floats with big values are crashing sync since postgres is sending values like "2e+09" and LOGICAL_SLOT_SUFFIX regex is not parsing them properly causing the code to parse float('2e') which does not work. I changed this regex so it does work for my case, but idk if it would work for other cases.

LOGICAL_SLOT_SUFFIX = re.compile(
    '\s(?P<key>"?\w+"?)\[(?P<type>[\w\s]+)\]:(?P<value>[\w\'"\-\\.\\+]+)'
)