Closed grncdr closed 8 months ago
larskanis
commented
10 months ago Can you please post the output of the segfault - most notably the ruby and C stack trace? How often does the segfault appear? Is it deterministic? Can you provide a sample script for reproduction?
MannyGrill
commented
10 months ago We started having this issue today with our test automation suite as well (it was working last week). I've also tried setting PGGSSENCMODE="disable" but that did not fix our problem.
Details:
ruby 3.2.21.5.4I can get this to reproduce 100% with our project whenever we try to connect to the DB. I was able to narrow it down to this function here:
def self.open_db_connection(db_type)
PG.connect(db_type)
endThe error message we get back:
ruby(97247,0x1d85a9000) malloc: double free for ptr 0x10fa13c00
ruby(97247,0x1d85a9000) malloc: *** set a breakpoint in malloc_error_break to debugI tried wrapping it in a rescue, but it always bombs out before a more helpful error message is returned. I'm uncomfortable setting breakpoints this deep, but hopefully, this information helps.
EDIT: One more thing to add (not sure if applicable) - our test automation suite is still running smoothly in our GitLab CI process (I assume they use Linux for their boxes), it is only having issues locally on my machine
grncdr
commented
10 months ago Oh thank god I'm not the only one, it's been a frustrating few days. 😄
I will try to build a small reproduction this morning.
grncdr
commented
10 months ago Ok here are details to reproduce.
Test script:
#!/usr/bin/env ruby
require 'bundler/inline'
PG_VERSION = ENV['PG_GEM_VERSION'] || '1.5.4'
gemfile do
source 'https://rubygems.org'
gem 'pg', PG_VERSION
end
# Expects environment variables PGHOST, PGUSER, PGPASSWORD, PGDATABASE etc.
PG.connect do |conn|
result = conn.exec('select 1 as one')
puts result[0]
endI've used this script with Ruby 3.1, 3.2, and 3.3, and all of them crash 100% reliably when connecting to RDS. The script never crashes when connecting to local Postgres instances (tested against latest releases of Postgres 13 through 15).
ohTHATaaronbrown
commented
10 months ago This only seems to hit and affect my mac (running Sonoma 14.2.1 currently). The same ruby code that crashes on my mac, when running on windows server and windows workstation runs fine.
By contrast with the OP of this issue, I am connecting to an Aurora PG cluster. All other (non-pg gem) means of connecting to this same cluster connect fine from the same mac where the pg gem crashes on connect. The crash occurs with every version of the pg gem i've tried, going all the way back to 1.2.3.
grncdr
commented
10 months ago Hey @ohTHATaaronbrown thanks for adding that. So far all of those affected are running Sonoma and connecting to RDS (both aurora and non-aurora Postgres).
I realize now that the crash report doesn't really include a C-stack, just some symbols. The Ruby stack points at the call to finish on this line: https://github.com/ged/ruby-pg/blob/5c105f227ac2b32331c0006b109688f6c7c9ed64/lib/pg/connection.rb#L693-L697 ... which indicates to me that the problem actually occurs earlier. The crash report points to PQFinish which is being called here https://github.com/ged/ruby-pg/blob/5c105f227ac2b32331c0006b109688f6c7c9ed64/ext/pg_connection.c#L537
It seems that something goes wrong before the pgconn pointer is properly initialized, and then this cleanup code causes the double free. Interestingly enough, libpqxx added protection for this scenario just 3 days ago: https://github.com/jtv/libpqxx/pull/780/files#diff-8540699dd68a5427a8f97a8d3a8a0999eef6ec1a8678470f73490c4570e3cc38R748-R753
grncdr
commented
10 months ago Another update from a colleague, also using an M1 macbook, also on Sonoma. His connections to RDS were still working until today, when he installed neovim using brew, apparently triggering a brew auto-update. Unfortunately, he closed the terminal and didn't realize the problem until later.
After encountering segfaults, he ran brew install libpq and that resolved the issue for him. I've tried the same, but it made no difference for me.
ohTHATaaronbrown
commented
10 months ago Ahhh. I already had libpq installed via homebrew, but brew upgrade libpq solved the issue for me as well. Note that my macbook pro is intel-based, not Apple silicon.
MannyGrill
commented
10 months ago Looks like upgrading libpq via brew also worked for me as well
grncdr
commented
10 months ago Ok, I have also resolved the issue locally by installing postgresql from Homebrew and rebuilding pg with the option --with-pg-config=/opt/homebrew/opt/postgresql@14/bin/pg_config.
I believe my problem is that the copy of postgres (and pg_config) I had installed via asdf does not not include this patch from Homebrew.
Not really a pg issue in the end, but hopefully this helps somebody else with a similar setup.
smicdeshapriya
commented
10 months ago Thanks for all the hints. I was able to resolve it just by -- brew upgrade postgresql in my mac Somona 14.2.1
I've recently started to experience segfaults when connecting to an RDS instance from my MacBook.
Some details:
ruby 3.1.4p2231.2.3and1.5.4.In my testing, I have been unable to reproduce the issue when running different Postgres versions locally. That is, everything works with any combination of pg gem and server version.
However, when I connect to an RDS instance (standard Postgres, not Aurora, version 13.10) I receive a segfault due to a double-free ("pointer being freed was not allocated"). Setting
PGGSSENCMODE="disable"does not fix the issue. Connecting viapsqldoes work. In trying to narrow things down, the only thing I can see that might be relevant is that the connection to RDS uses TLS, whereas my local connections do not.