Regression in ruby 3 update

[sonalkr132]

We had to revert the ruby 3 update because memory usage kept increasing with time https://github.com/rubygems/rubygems.org/commit/1c1584d4d2d22d6a5511069ea9042dd2ceb2ca5b

Steps to Reproduce

1. run image with ruby 2.7 (may need rake db:create db:migrate)
   • sudo docker run -e NEW_RELIC_AGENT_ENABLED=false -e RAILS_ENV=production -e SECRET_KEY_BASE=1234 -e DATABASE_URL=postgresql://localhost --net host quay.io/rubygems/rubygems.org:ruby2 -- unicorn_rails -E production -c /app/config/unicorn.conf
   • sudo docker stats
   • create a file for load test

     $ cat <<EOF >some.txt
     GET http://localhost:3000
     Host: rubygems.org
     X-Forwarded-Proto: https
     EOF

   • run load test (download) and check stats: vegeta attack -duration=120s -targets some.txt | tee results.bin | vegeta report
2. repeat same with ruby3 image
   • sudo docker run -e NEW_RELIC_AGENT_ENABLED=false -e RAILS_ENV=production -e SECRET_KEY_BASE=1234 -e DATABASE_URL=postgresql://localhost --net host quay.io/rubygems/rubygems.org:ruby3 -- unicorn_rails -E production -c /app/config/unicorn.conf
   • sudo docker stats
     • run load test and check stats: vegeta attack -duration=120s -targets some.txt | tee results.bin | vegeta report

Expected Behavior

memory usage should become constant after a while, as seen on ruby2 image

Current Behavior

memory usage keeps increasing on ruby3. brief flat lines are part where the load test was not running.

Possible Solution

¯_(ツ)_/¯

Additional Context

I had tried taking heap dump from prod as explained in this blog, however, heapy diff was showing that size of everything had increased. Perhaps things are not getting GCed?

$ heapy diff ruby-heap3.dump  ruby-heap4.dump
Allocated IMEMO 575 objects of size 23000/1433974 (in bytes) at: /usr/local/bundle/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/query_cache.rb:101
Allocated HASH 265 objects of size 44520/1433974 (in bytes) at: /usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/map/non_concurrent_map_backend.rb:16
Allocated DATA 265 objects of size 19080/1433974 (in bytes) at: /usr/local/bundle/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/map/mri_map_backend.rb:14
Allocated OBJECT 241 objects of size 17352/1433974 (in bytes) at: /app/config/initializers/gem_version_monkeypatch.rb:11
Allocated ARRAY 241 objects of size 9800/1433974 (in bytes) at: /usr/local/lib/ruby/site_ruby/3.0.0/rubygems/version.rb:395
Allocated ARRAY 241 objects of size 9800/1433974 (in bytes) at: /usr/local/lib/ruby/site_ruby/3.0.0/rubygems/version.rb:374
Allocated STRING 240 objects of size 9600/1433974 (in bytes) at: /usr/local/lib/ruby/site_ruby/3.0.0/rubygems/version.rb:220
Allocated ARRAY 95 objects of size 4280/1433974 (in bytes) at: /usr/local/bundle/gems/actionview-6.1.4.1/lib/action_view/template/resolver.rb:152
Allocated OBJECT 95 objects of size 3800/1433974 (in bytes) at: /usr/local/bundle/gems/actionview-6.1.4.1/lib/action_view/template/resolver.rb:83
....

[hsbt]

@sonalkr132 Thanks, I'll share this regression to the Ruby core team.

[sonalkr132]

that would be great, thanks. let me know if anyone needs help setting up rubygems.org locally or reproducing this.

[peterzhu2118]

I looked into this. I can confirm that https://github.com/ruby/ruby/commit/98ac62de5cb03efec0fb32684c61c0d4df692e5a fixes the issue. However, it's not in a 3.0 release yet. I'll ask around for a timeline of when 3.0.3 will be released.

[peterzhu2118]

As you might have heard, Ruby 3.0.3 is out! It contains the fix for the memory leak in Hash#transform_keys!. Unfortunately, a backport in 3.0.3 has another bug, which breaks bootsnap. The latest version of bootsnap (1.9.3) has a workaround. Please give 3.0.3 a try and let us know if you run into any issues with it!

[hsbt]

rubygems.org used (or will use) Ruby 3.0.3 at #2876 .

@peterzhu2118 Thanks for your investigation.