duckdb / postgres_scanner

https://duckdb.org/docs/extensions/postgres
MIT License
244 stars 40 forks source link

postgres_scanner causes memory leak #158

Open ila opened 10 months ago

ila commented 10 months ago

What happens?

Hi, not sure if this is interesting for you but I get this by executing the following series of steps on DuckDB 0.9.2 (official version, no changes made by me, postgres scanner 0.9.2 downloaded from your sources) compiled in debug, Ubuntu 23.10:

D load '/home/ila/postgres_scanner.duckdb_extension';
D ATTACH '' AS postgres_db (TYPE postgres);
Error: IO Error: Unable to connect to Postgres at : connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?

D .q

=================================================================
==49548==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 1024 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2de997 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
    duckdb/duckdb#1 0x7f0d03d9199f in makeEmptyPGconn (/home/ila/postgres_scanner.duckdb_extension+0x19199f) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 16384 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2defdf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    duckdb/duckdb#1 0x7f0d03d91a61 in makeEmptyPGconn (/home/ila/postgres_scanner.duckdb_extension+0x191a61) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 16384 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2defdf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    duckdb/duckdb#1 0x7f0d03d91a46 in makeEmptyPGconn (/home/ila/postgres_scanner.duckdb_extension+0x191a46) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 1024 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2defdf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    duckdb/duckdb#1 0x7f0d03d9759a in connectOptions2 (/home/ila/postgres_scanner.duckdb_extension+0x19759a) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 512 byte(s) in 2 object(s) allocated from:
    #0 0x7f0d1e2defdf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    duckdb/duckdb#1 0x7f0d03da996d in initPQExpBuffer (/home/ila/postgres_scanner.duckdb_extension+0x1a996d) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 512 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2defdf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    duckdb/duckdb#1 0x7f0d03d91a7c in makeEmptyPGconn (/home/ila/postgres_scanner.duckdb_extension+0x191a7c) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 110 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2de997 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
    duckdb/duckdb#1 0x7f0d03d88b8c in pg_getaddrinfo_all (/home/ila/postgres_scanner.duckdb_extension+0x188b8c) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 57 byte(s) in 11 object(s) allocated from:
    #0 0x7f0d1e2841a8 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
    duckdb/duckdb#1 0x7f0d03d98a09 in PQconnectStart (/home/ila/postgres_scanner.duckdb_extension+0x198a09) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2de997 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
    duckdb/duckdb#1 0x7f0d03d88b71 in pg_getaddrinfo_all (/home/ila/postgres_scanner.duckdb_extension+0x188b71) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2de997 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
    duckdb/duckdb#1 0x7f0d03d96fd6 in connectOptions2 (/home/ila/postgres_scanner.duckdb_extension+0x196fd6) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 5 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2defdf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    duckdb/duckdb#1 0x7f0d03d971f6 in connectOptions2 (/home/ila/postgres_scanner.duckdb_extension+0x1971f6) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 5 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2841a8 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
    duckdb/duckdb#1 0x7f0d03d9724c in connectOptions2 (/home/ila/postgres_scanner.duckdb_extension+0x19724c) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

Indirect leak of 4 byte(s) in 1 object(s) allocated from:
    #0 0x7f0d1e2841a8 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
    duckdb/duckdb#1 0x7f0d03d972dd in connectOptions2 (/home/ila/postgres_scanner.duckdb_extension+0x1972dd) (BuildId: 7ae6dfb9400febdb8712071fd94f7bb04491d4f2)

SUMMARY: AddressSanitizer: 36109 byte(s) leaked in 24 allocation(s).

Process finished with exit code 1

Feel free to close this if you think this situation is unique or have already fixed it in newer versions (which I can't really try since I can't find a newer packaged version of the scanner).

To Reproduce

load '/home/ila/postgres_scanner.duckdb_extension';
ATTACH '' AS postgres_db (TYPE postgres);
.q

OS:

Ubuntu 23.04

DuckDB Version:

0.9.2

DuckDB Client:

shell

Full Name:

Ilaria Battiston

Affiliation:

CWI

Have you tried this on the latest main branch?

I have tested with a release build (and could not test with a main build)

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

markhalonen commented 5 months ago

I also had a memory leak issue. I'm providing a situation where this tool is not used for just ad-hoc data analysis: Our use-case is to export a single-tenant DuckDB from a multi-tenant Postgres so our user can do in-browser bi. So we use it in a production web server in-process in Node, and the memory leaks stack up. So we got around the memory leaks by using node fork, which is no big deal, just wanted to provide a use-case where memory leaks are not completely ignorable.