Add `--debugging`, `--unattended-debugging` and `--tds_version` parameters

[jcarnu]

--debugging display PostgreSQL backend PID and waits for key to be pressed allowing gdb --pid=<PID> for debugging purpose (in another session). It also prints the SQL sentences being executed and displays display more precise informations on test crash location using psycopg2.extensions.Diagnostics class

--unattended-debugging does almost the same as debugging, but it doesn't pause, and writes the PostgreSQL logs to console.

--tds_version allows to specify tds version at test time (both in mssql testing and pgsql testing)

This PR will also now print PostgreSQL logs if there are errors.

[jenkins-juliogonzalez]

Test FAILed.

[jcarnu]

Retest this , please

[jenkins-juliogonzalez]

Test PASSed.

[jcarnu]

@juliogonzalez great, I didn't see where the problem was in calling run_tests, now I've seen it :sweat_smile: .

Any comments are more than welcome. I tried to be cleaner in my impl. Maybe run_tests() last param should be show_details than debugging but it's kind of detail.

Next stop tomorrow morning for me.

[jenkins-juliogonzalez]

Test PASSed.

[jenkins-juliogonzalez]

Test PASSed.

[jcarnu]

@juliogonzalez I had time to do the things. Let me know if it's okaye. You've got a collaborator access to my repo.

Sorry not to be more available.

For PGConf.eu maybe next year or maybe could we meet in fosdem in end january/beginning of february ?

[jenkins-juliogonzalez]

Test FAILed.

[jenkins-juliogonzalez]

Test PASSed.

[jenkins-juliogonzalez]

Test FAILed.

[jenkins-juliogonzalez]

Test FAILed.

[jenkins-juliogonzalez]

Test FAILed.

[jenkins-juliogonzalez]

Test FAILed.

[juliogonzalez]

Not my day, it seems. I just need to get this working and print dmesg and postgres log when there are errors or unattended_debugging is enabled.

[jenkins-juliogonzalez]

Test FAILed.

[jenkins-juliogonzalez]

Test PASSed.

[juliogonzalez]

Do not merge the PR please, I am getting the code for the logs ready.

[jenkins-juliogonzalez]

Test PASSed.

[juliogonzalez]

Ok, so this is how it looks like when we use --unattended-debugging:

https://jenkins.juliogonzalez.es/job/tds_fdw-build-free/109/DISTRO=ubuntu16.04,PG_VER=12,label=docker/console https://jenkins.juliogonzalez.es/job/tds_fdw-build-free/109/DISTRO=centos6,PG_VER=12,label=docker/console

As you can see, it prints the queries, and the logs.

For --debugging it will not print the logs (the developer can have a look, and maybe he already has tail tracking them.

Printing logs requires using a PostgreSQL user with SUPERUSER access. If the user is not SUPERUSER there will be an error about it, but that will not fail the script.

In case there are errors, the logs are always printed unless --debugging is being used (useful for the CI, to help having a first look).

[jenkins-juliogonzalez]

Test PASSed.

[juliogonzalez]

@GeoffMontee, @jcarnu, when this PR is approved, let me merge. I will do it with the "squash and merge" option, and will adjust the message. It doesn't make sense having all the commits at master.

[juliogonzalez]

@jcarnu have a look, give it a try, and let me know if we are ready to merge (can't add it you as reviewer).

[jcarnu]

As you know, it will take days for me to have time to check. My training finalization at pgconf is eating all my time.

I will be able to have time maybe on wednesday, not before.

[GeoffMontee]

This looks great to me. I love seeing the new debugging information in the logs. Thanks, @jcarnu and @juliogonzalez!

This gives me an idea for another improvement that we could make to the CI eventually: creating core dumps when PostgreSQL crashes and automatically extracting back traces from them. This would probably make it easier to debug crashes like the one we saw in #213. This would probably require at least the changes listed below. What do you think about that, @juliogonzalez?

Changes Required in ci-build

• We would have to make sure that ci-build compiles tds_fdw with the -ggdb option specified in PG_CPPFLAGS, so that tds_fdw is built with debugging symbols.

Changes Required in ci-setup

• We would have to make sure that ci-setup installs debuginfo packages for PostgreSQL.

For example:

sudo yum install postgresql12-debuginfo

• We would have to make sure that ci-setup grants unlimited size core dumps to the PostgreSQL process.

For OSes that use systemd, that would probably look like this:

sudo tee /etc/systemd/system/postgresql-12.service.d/limitcore.conf <<EOF
[Service]

LimitCORE=infinity
EOF
sudo systemctl daemon-reload

For other OSes, that would probably look like this:

sudo tee /etc/security/limits.conf.d/postgres_core.conf <<EOF
postgres soft core unlimited
postgres hard core unlimited
EOF

• We would also have to make sure that ci-setup sets up some other parameters related to core dumps.

For example:

sudo tee /etc/sysctl.d/postgres_core.conf <<EOF
# Set the path to the core dumps
kernel.core_pattern = /core_dumps

# Add the PID to the end of the file name
kernel.core_uses_pid = 1

# Allow setuid processes to dump core. Is this necessary for Postgres?
fs.suid_dumpable = 2
EOF

• We would also make sure that ci-setup creates any paths that we depend on.

For example:

mkdir /core_dumps
chmod 0777 /core_dumps

Changes Required in tds_fdw

• We would have to change tests/postgresql-tests.py in tds_fdw to make it detect PostgreSQL crashes. Maybe it could scan the PostgreSQL log for lines like this?:

2019-10-02 02:28:48.702 UTC [50] LOG:  server process (PID 292) was terminated by signal 11: Segmentation fault

• If tests/postgresql-tests.py detects a PostgreSQL crash, then it would have to get the value of kernel.core_pattern:

For example:

sysctl kernel.core_pattern

• When tests/postgresql-tests.py has the value of kernel.core_pattern, it could check the path for core dumps.

• When tests/postgresql-tests.py finds a core dump, it could get all backtraces from it.

For example:

sudo gdb --batch --eval-command="thread apply all bt full" $(which postmaster) ${core_file_path}

[juliogonzalez]

Yes, it all makes sense. Can you create a new issue with all the content, and I will make some adjustments about what needs to be changed there? :-)

[jcarnu]

I quickly reviewed this patch. It seems to be fine. Thanks for improvement. It can be merged for me.