Postgres replication

[panosfol]

This PR contains the following:

• pg_connect: Connection to the origin and the target database.
• replicate: Utilize the pg_dump and pg_restore commands to replicate the database.
• Utilizing the log.h library.

[charmitro]

valgrind --leak-check=full --show-leak-kinds=all ./cnc -f test.yaml -v
==1745997== Memcheck, a memory error detector
==1745997== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==1745997== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==1745997== Command: ./cnc -f test.yaml -v
==1745997== 
==1745997== Conditional jump or move depends on uninitialised value(s)
==1745997==    at 0x484663C: strncat (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x10A033: replicate (postgres.c:195)
==1745997==    by 0x109581: execute_db_operations (db.c:34)
==1745997==    by 0x10968A: main (main.c:45)
==1745997== 
==1745997== Conditional jump or move depends on uninitialised value(s)
==1745997==    at 0x4846647: strncat (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x10A033: replicate (postgres.c:195)
==1745997==    by 0x109581: execute_db_operations (db.c:34)
==1745997==    by 0x10968A: main (main.c:45)
==1745997== 
==1746000== Syscall param execve(envp[i]) points to uninitialised byte(s)
==1746000==    at 0x4998917: execve (syscall-template.S:120)
==1746000==    by 0x10A1D1: replicate (postgres.c:220)
==1746000==    by 0x109581: execute_db_operations (db.c:34)
==1746000==    by 0x10968A: main (main.c:45)
==1746000==  Address 0x1fff000040 is on thread 1's stack
==1746000==  in frame #1, created by replicate (postgres.c:109)
==1746000== 
==1746002== Syscall param execve(envp[i]) points to uninitialised byte(s)
==1746002==    at 0x4998917: execve (syscall-template.S:120)
==1746002==    by 0x10A41B: replicate (postgres.c:250)
==1746002==    by 0x109581: execute_db_operations (db.c:34)
==1746002==    by 0x10968A: main (main.c:45)
==1746002==  Address 0x1fff000040 is on thread 1's stack
==1746002==  in frame #1, created by replicate (postgres.c:109)
==1746002== 
test.yaml
construct
VERBOSE:connect_pg  Origin-database connection: Success!
VERBOSE:connect_pg  Target-database connection: Success!
VERBOSE:replicate   Checking if $PG_BIN environmental variable.
VERBOSE:replicate   $PG_BIN was not found, default path /usr/bin/ will be used.
VERBOSE:replicate   Starting `pg_dump` process.
VERBOSE:replicate   Starting `pg_restore` process.
VERBOSE:replicate   Database Replication was successful!
==1745997== 
==1745997== HEAP SUMMARY:
==1745997==     in use at exit: 233 bytes in 5 blocks
==1745997==   total heap usage: 59,760 allocs, 59,755 frees, 4,888,693 bytes allocated
==1745997== 
==1745997== 8 bytes in 1 blocks are still reachable in loss record 1 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x50192C9: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50193A1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5005265: gss_acquire_cred (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x48961CF: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x488005B: PQconnectPoll (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x48809ED: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997== 
==1745997== 24 bytes in 1 blocks are still reachable in loss record 2 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x5019273: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50193A1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5005265: gss_acquire_cred (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x48961CF: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x488005B: PQconnectPoll (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x48809ED: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997== 
==1745997== 48 bytes in 1 blocks are still reachable in loss record 3 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x51B7F0C: krb5int_setspecific (in /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1)
==1745997==    by 0x50192E5: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50193A1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5005265: gss_acquire_cred (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x48961CF: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x488005B: PQconnectPoll (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997== 
==1745997== 73 bytes in 1 blocks are still reachable in loss record 4 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x49627F9: strdup (strdup.c:42)
==1745997==    by 0x501938F: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x502D0B1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5031D33: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997== 
==1745997== 80 bytes in 1 blocks are still reachable in loss record 5 of 5
==1745997==    at 0x48455EF: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x109489: execute_db_operations (db.c:17)
==1745997==    by 0x10968A: main (main.c:45)
==1745997== 
==1745997== LEAK SUMMARY:
==1745997==    definitely lost: 0 bytes in 0 blocks
==1745997==    indirectly lost: 0 bytes in 0 blocks
==1745997==      possibly lost: 0 bytes in 0 blocks
==1745997==    still reachable: 233 bytes in 5 blocks
==1745997==         suppressed: 0 bytes in 0 blocks
==1745997== 
==1745997== Use --track-origins=yes to see where uninitialised values come from
==1745997== For lists of detected and suppressed errors, rerun with: -s
==1745997== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

[charmitro]

valgrind --leak-check=full --show-leak-kinds=all ./cnc -f test.yaml -v
==1745997== Memcheck, a memory error detector
==1745997== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==1745997== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==1745997== Command: ./cnc -f test.yaml -v
==1745997== 
==1745997== Conditional jump or move depends on uninitialised value(s)
==1745997==    at 0x484663C: strncat (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x10A033: replicate (postgres.c:195)
==1745997==    by 0x109581: execute_db_operations (db.c:34)
==1745997==    by 0x10968A: main (main.c:45)
==1745997== 
==1745997== Conditional jump or move depends on uninitialised value(s)
==1745997==    at 0x4846647: strncat (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x10A033: replicate (postgres.c:195)
==1745997==    by 0x109581: execute_db_operations (db.c:34)
==1745997==    by 0x10968A: main (main.c:45)
==1745997== 
==1746000== Syscall param execve(envp[i]) points to uninitialised byte(s)
==1746000==    at 0x4998917: execve (syscall-template.S:120)
==1746000==    by 0x10A1D1: replicate (postgres.c:220)
==1746000==    by 0x109581: execute_db_operations (db.c:34)
==1746000==    by 0x10968A: main (main.c:45)
==1746000==  Address 0x1fff000040 is on thread 1's stack
==1746000==  in frame #1, created by replicate (postgres.c:109)
==1746000== 
==1746002== Syscall param execve(envp[i]) points to uninitialised byte(s)
==1746002==    at 0x4998917: execve (syscall-template.S:120)
==1746002==    by 0x10A41B: replicate (postgres.c:250)
==1746002==    by 0x109581: execute_db_operations (db.c:34)
==1746002==    by 0x10968A: main (main.c:45)
==1746002==  Address 0x1fff000040 is on thread 1's stack
==1746002==  in frame #1, created by replicate (postgres.c:109)
==1746002== 
test.yaml
construct
VERBOSE:connect_pg    Origin-database connection: Success!
VERBOSE:connect_pg    Target-database connection: Success!
VERBOSE:replicate Checking if $PG_BIN environmental variable.
VERBOSE:replicate $PG_BIN was not found, default path /usr/bin/ will be used.
VERBOSE:replicate Starting `pg_dump` process.
VERBOSE:replicate Starting `pg_restore` process.
VERBOSE:replicate Database Replication was successful!
==1745997== 
==1745997== HEAP SUMMARY:
==1745997==     in use at exit: 233 bytes in 5 blocks
==1745997==   total heap usage: 59,760 allocs, 59,755 frees, 4,888,693 bytes allocated
==1745997== 
==1745997== 8 bytes in 1 blocks are still reachable in loss record 1 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x50192C9: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50193A1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5005265: gss_acquire_cred (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x48961CF: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x488005B: PQconnectPoll (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x48809ED: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997== 
==1745997== 24 bytes in 1 blocks are still reachable in loss record 2 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x5019273: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50193A1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5005265: gss_acquire_cred (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x48961CF: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x488005B: PQconnectPoll (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x48809ED: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997== 
==1745997== 48 bytes in 1 blocks are still reachable in loss record 3 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x51B7F0C: krb5int_setspecific (in /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1)
==1745997==    by 0x50192E5: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50193A1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5005265: gss_acquire_cred (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x48961CF: ??? (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997==    by 0x488005B: PQconnectPoll (in /usr/lib/x86_64-linux-gnu/libpq.so.5.15)
==1745997== 
==1745997== 73 bytes in 1 blocks are still reachable in loss record 4 of 5
==1745997==    at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x49627F9: strdup (strdup.c:42)
==1745997==    by 0x501938F: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50194E1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50179EC: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50184CF: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x502D0B1: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5031D33: ??? (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x50049D1: gss_add_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997==    by 0x5004FC8: gss_acquire_cred_from (in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2)
==1745997== 
==1745997== 80 bytes in 1 blocks are still reachable in loss record 5 of 5
==1745997==    at 0x48455EF: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==1745997==    by 0x109489: execute_db_operations (db.c:17)
==1745997==    by 0x10968A: main (main.c:45)
==1745997== 
==1745997== LEAK SUMMARY:
==1745997==    definitely lost: 0 bytes in 0 blocks
==1745997==    indirectly lost: 0 bytes in 0 blocks
==1745997==      possibly lost: 0 bytes in 0 blocks
==1745997==    still reachable: 233 bytes in 5 blocks
==1745997==         suppressed: 0 bytes in 0 blocks
==1745997== 
==1745997== Use --track-origins=yes to see where uninitialised values come from
==1745997== For lists of detected and suppressed errors, rerun with: -s
==1745997== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

Nevermind this, was testing the wrong patch.

[charmitro]

I did not review it, but with a very quick look I believe the pipes of the second command (pg_restore) are not redirecting due to them being closed at that time.

[panosfol]

I did not review it, but with a very quick look I believe the pipes of the second command (pg_restore) are not redirecting due to them being closed at that time.

I tested it and it seems to work, not sure if there is an edge case that might fail. I get both the verbose output of pg_dump and pg_restore.

[charmitro]

Panagiotis Foliadis @.***> writes:

I did not review it, but with a very quick look I believe the pipes of the second command (pg_restore) are not redirecting due to them being closed at that time.

I tested it and it seems to work, not sure if there is an edge case that might fail. I get both the verbose output of pg_dump and pg_restore.

"read_buffer" is where the output should come from, right? If that is the case, you can test it by adding a debug word in the "printf" of the function.

I can see it for pg_dump but not for pg_restore.

printf("%s %s", __func__, buffer);

[panosfol]

Panagiotis Foliadis @.***> writes: I did not review it, but with a very quick look I believe the pipes of the second command (pg_restore) are not redirecting due to them being closed at that time. I tested it and it seems to work, not sure if there is an edge case that might fail. I get both the verbose output of pg_dump and pg_restore. "read_buffer" is where the output should come from, right? If that is the case, you can test it by adding a debug word in the "printf" of the function. I can see it for pg_dump but not for pg_restore. c printf("%s %s", __func__, buffer);

Ok I see what happened, you are right. The output of pg_restore was being printed but not through the buffer. Shoud I use another pipe?

[charmitro]

Ok I see what happened, you are right. The output of pg_restore was being printed but not through the buffer. Shoud I use another pipe?

I suggest re-opening pipe with "pipe(pipefd)" and it'll be fine.

[charmitro]

Let's fix these sparse warnings/errors before the actual review:

$ sparse src/main.c src/config.c src/postgres.c src/db.c -Iinclude/ -I/usr/include/postgresql/ -Iinclude/db/
src/main.c:15:45: warning: Using plain integer as NULL pointer
src/main.c:16:35: warning: Using plain integer as NULL pointer
src/postgres.c:25:18: warning: non-ANSI function declaration of
function 'construct_pg'
src/postgres.c:25:5: warning: symbol 'construct_pg' was not
declared. Should it be static?
src/postgres.c:57:54: warning: Using plain integer as NULL pointer
src/postgres.c:166:17: warning: Using plain integer as NULL pointer
src/postgres.c:180:17: warning: Using plain integer as NULL pointer
src/postgres.c:215:27: warning: Using plain integer as NULL pointer
src/postgres.c:254:27: warning: Using plain integer as NULL pointer

Install sparse via (it is already installed on our development machine):

        $ sudo apt install sparse