Investigate/profile RAM usage

[jrochkind]

Our app has for a while used relatively large amounts of RAM for a Rails app.

Some months ago, with 3 puma workers with 3 threads each, we were exhausting the 2.5G of RAM on a Heroku performance-m dyno -- which is pretty large usage.

A couple months ago we turned on "jemalloc" use on heroku, tracked in #2377, which looked to save us around ~20% of RAM use, and we stopped getting the heroku R14 errors.

However, with the recent migration to Rails 7.1, our RAM usage has jumped, and we've started getting R14 errors ago. There is no general community-known reason why a Rails 7.1 upgrade should increase RAM usage.

That is heroku RAM metric dashboard. You can see there the bump in RAM usage that corresponded to Rails 7.1 deployment.

I think before and after commit SHA's are cc8e8404 (most recent Rails 7.0 deploy) and 3afd9c51 (first Rails 7.1 deploy)

We want to find out:

1. In general, what is using so much RAM?
2. Specifically, what changed in the apps to use more RAM with Rails 7.1 upgrade?

[jrochkind]

@bensheldon on reddit generously offered to pair with me to figure it out, and also recommended derailed_benchmarks

Ben, if you see this, I can do some initial derailed_benchmarks on my own, so we are a step ahead working together... but I'm not sure how to run derailed with the bin/production-rails technique you had recommended for easily runnable local Rails in production. I do have runnable bin/production-rails working. How do I get derailed to use it?

[bensheldon]

👋🏻

oops, sorry I didn't describe the entire process:

1. Get bin/production-rails working. That way there's a repeatable and durable (in code) and verifiable/inspectable (can boot the console/app and verify that it's loading as expected) set of steps for booting in production mode.
2. ...then, unfortunately derailed only works with ENV variables. So you need to take the ENV configuration you isolated in Step 1, and put it on the command-line, as well as identify any non-ENV-related global variables and convert them to ENV-modifiable ones.
3. Then you can run derailed

(arguably there are better technical ways to accomplish both at the same time, but I've found this has been an ok process in my consulting engagements)

[jrochkind]

OK, haven't yet figured out best way to run bundle exec derailed exec perf:mem with my custom env needed for production, but can run bundle:mem, let's see what we see.

Rails 7.1 (3afd9c514)

bundle exec derailed bundle:mem

``` TOP: 157.1563 MiB rails/all: 59.7031 MiB action_mailbox/engine: 31.3125 MiB action_mailbox: 31.3125 MiB action_mailbox/mail_ext: 31.2969 MiB action_mailbox/mail_ext/address_equality.rb: 29.4844 MiB mail/elements/address: 29.4844 MiB mail/parsers/address_lists_parser: 29.4531 MiB mail: 1.8125 MiB (Also required by: TOP) mail/field: 0.4531 MiB net/smtp: 0.3281 MiB mail/multibyte: 0.3125 MiB rails: 15.8281 MiB (Also required by: active_record/railtie, active_model/railtie, and 21 others) rails/application: 12.7344 MiB active_support/encrypted_configuration: 5.625 MiB active_support/encrypted_file: 5.6094 MiB active_support/message_encryptor: 4.7031 MiB (Also required by: rails/secrets) active_support/messages/codec: 4.625 MiB (Also required by: active_support/message_verifier) /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/metadata: 4.4531 MiB active_support/json: 4.3125 MiB (Also required by: active_model/serializers/json, active_support/multibyte/chars) active_support/json/encoding: 3.9375 MiB active_support/core_ext/object/json: 3.9219 MiB (Also required by: active_support/core_ext/object, active_record/token_for) active_support/core_ext/time/conversions: 3.6094 MiB (Also required by: active_support/core_ext/date_time/conversions, active_support/core_ext/time/calculations) active_support/values/time_zone: 3.6094 MiB (Also required by: active_support/core_ext/date_time/conversions, active_support/time_with_zone) tzinfo: 3.5781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/tzinfo-2.0.6/lib/tzinfo/string_deduper: 1.9688 MiB concurrent: 1.9688 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/tzinfo-2.0.6/lib/tzinfo/data_source, sprockets/manifest, and 4 others) concurrent/promises: 0.4688 MiB concurrent/configuration: 0.4063 MiB (Also required by: concurrent/scheduled_task, concurrent/options, and 3 others) concurrent/executors: 0.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/tzinfo-2.0.6/lib/tzinfo/annual_rules: 1.1875 MiB active_support/json/decoding: 0.375 MiB json: 0.375 MiB (Also required by: active_support/core_ext/object/json, /Users/jrochkind/.gem/ruby/3.2.2/gems/vite_ruby-3.3.4/lib/vite_ruby/config.rb, and 39 others) tempfile: 0.875 MiB (Also required by: rails/secrets, rack/utils, and 15 others) tmpdir: 0.75 MiB (Also required by: aws-sdk-s3/file_downloader, execjs/external_runtime) fileutils: 0.6094 MiB (Also required by: rack/utils, /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-test-2.1.0/lib/rack/test/uploaded_file, and 5 others) active_support/key_generator: 4.7656 MiB openssl: 4.7656 MiB (Also required by: active_support/message_encryptor, active_support/message_verifier, and 28 others) openssl.so: 2.5625 MiB /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/openssl/ssl: 1.9844 MiB ipaddr: 0.375 MiB (Also required by: active_support/core_ext/object/json, /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/endpoints/url, and 4 others) socket: 0.3125 MiB (Also required by: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/openssl/ssl, net/protocol, and 18 others) yaml: 1.2813 MiB (Also required by: active_support/encrypted_configuration, rails/secrets, and 15 others) psych: 1.2188 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/psych-5.1.1.1/lib/psych/visitors: 0.4844 MiB active_support/deprecation: 0.4844 MiB (Also required by: active_support/rails, view_component/deprecation) active_support: 2.0781 MiB (Also required by: active_support/railtie, active_support/i18n_railtie, and 21 others) active_support/dependencies/autoload: 1.0938 MiB (Also required by: view_component) active_support/inflector/methods: 1.0938 MiB (Also required by: active_support/core_ext/time/conversions, active_support/core_ext/date_time/conversions, and 6 others) active_support/inflections: 1.0469 MiB (Also required by: active_support/inflector) active_support/inflector/inflections: 0.8906 MiB (Also required by: active_support/inflector) active_support/i18n: 0.8594 MiB (Also required by: active_support/inflector/transliterate, abstract_controller) i18n: 0.5781 MiB (Also required by: ransack/translate) i18n/exceptions: 0.3438 MiB active_support/logger: 0.4063 MiB active_support/logger_silence: 0.375 MiB active_support/logger_thread_safe_level: 0.3125 MiB (Also required by: active_support/logger) action_dispatch/railtie: 0.3594 MiB (Also required by: action_controller/railtie) active_support/railtie: 0.3125 MiB active_record/railtie: 5.4375 MiB (Also required by: active_storage/engine, action_mailbox/engine) active_record: 2.8594 MiB (Also required by: active_storage, attr_json, and 6 others) arel: 1.6875 MiB arel/visitors: 0.7813 MiB arel/visitors/to_sql: 0.4531 MiB arel/nodes: 0.4219 MiB action_controller/railtie: 2.4688 MiB (Also required by: active_storage/engine, rails/all, and 4 others) action_controller: 2.1094 MiB (Also required by: responders) action_controller/metal/strong_parameters: 1.4688 MiB rack/test: 1.1563 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-test-2.1.0/lib/rack/test/cookie_jar: 0.4063 MiB rack/utils: 0.375 MiB (Also required by: sprockets/context, sprockets/sassc_processor, and 2 others) action_controller/metal/exceptions: 0.625 MiB (Also required by: action_dispatch/journey/formatter, action_dispatch/routing/route_set) action_view/railtie: 0.3438 MiB (Also required by: rails/all) active_storage/engine: 3.0 MiB (Also required by: action_mailbox/engine, action_text/engine) active_storage: 1.7656 MiB marcel: 1.6719 MiB marcel/magic: 1.6094 MiB marcel/tables: 1.5625 MiB action_dispatch/routing/route_set: 0.6094 MiB action_dispatch/journey: 0.375 MiB action_dispatch/journey/router: 0.3125 MiB action_text/engine: 2.0781 MiB action_text: 2.0781 MiB nokogiri: 2.0625 MiB (Also required by: loofah, traject/marc_reader, and 3 others) /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml: 0.5 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/extension: 0.5 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/version/info) /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/3.2/nokogiri: 0.5 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/css: 0.4375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4: 0.4219 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html) rails/test_unit/railtie: 1.7031 MiB rails/test_unit/line_filtering: 1.7031 MiB rails/test_unit/runner: 1.7031 MiB rake/file_list: 0.7969 MiB rails/test_unit/test_parser: 0.75 MiB ripper: 0.7188 MiB (Also required by: rails/source_annotation_extractor, irb, and 2 others) action_cable/engine: 0.3281 MiB citeproc/ruby: 10.3906 MiB citeproc: 5.4531 MiB citeproc/variable: 2.6406 MiB citeproc/item: 1.0938 MiB citeproc/names: 0.7813 MiB citeproc/date: 0.4688 MiB csl: 4.6719 MiB (Also required by: csl/styles) csl/schema: 1.8125 MiB csl/info: 0.5313 MiB csl/style: 0.4375 MiB aws-sdk-cloudwatchevents: 10.2813 MiB aws-sdk-core: 8.5781 MiB (Also required by: aws-sdk-cloudwatchlogs, aws-sdk-mediaconvert, and 2 others) /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/xml: 3.375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/xml/parser: 3.0938 MiB aws-sdk-core/xml/parser/engines/oga: 3.0313 MiB oga: 3.0313 MiB (Also required by: prawn_html/html_parser) oga/html/entities: 1.0 MiB oga/xpath/lexer: 0.8906 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-sts: 0.8125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-sts/client: 0.4844 MiB seahorse: 0.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-ssooidc: 0.5938 MiB aws-partitions: 0.4375 MiB jmespath: 0.3594 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/pager) /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-sso: 0.3438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchevents-1.67.0/lib/aws-sdk-cloudwatchevents/client_api: 0.8438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchevents-1.67.0/lib/aws-sdk-cloudwatchevents/types: 0.7344 MiB aws-sdk-s3: 8.0469 MiB (Also required by: uppy/s3_multipart/client) /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/client_api: 2.375 MiB aws-sdk-kms: 1.625 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-kms-1.72.0/lib/aws-sdk-kms/types: 1.0625 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-kms-1.72.0/lib/aws-sdk-kms/client_api: 0.4219 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/types: 1.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/customizations: 1.0625 MiB aws-sdk-s3/encryption_v2: 0.3438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/client: 0.6875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/endpoint_provider: 0.3438 MiB irb: 7.0625 MiB reline: 6.4844 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.9.0/lib/irb/input-method, /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.9.0/lib/irb/color, and 2 others) reline/line_editor: 0.8906 MiB reline/unicode: 0.4219 MiB traject: 6.0156 MiB traject/indexer: 4.5781 MiB traject/solr_json_writer: 2.9688 MiB httpclient: 2.9063 MiB httpclient/cookie: 1.625 MiB http-cookie: 1.5938 MiB http/cookie: 1.5938 MiB (Also required by: http/cookie_jar) domain_name: 1.5781 MiB domain_name/etld_data: 1.5 MiB httpclient/util: 0.4688 MiB (Also required by: httpclient/http, httpclient/cookie) addressable/uri: 0.4688 MiB (Also required by: http/uri, addressable, and 3 others) httpclient/session: 0.3438 MiB (Also required by: httpclient/auth) traject/marc_reader: 1.1563 MiB marc: 1.1406 MiB (Also required by: traject/ndj_reader, marc/fastxmlwriter) /Users/jrochkind/.gem/ruby/3.2.2/gems/marc-1.2.0/lib/marc/xmlwriter: 0.8438 MiB rexml/document: 0.8281 MiB (Also required by: prawn-svg, oai/provider) /Users/jrochkind/.gem/ruby/3.2.2/gems/rexml-3.2.6/lib/rexml/element: 0.5938 MiB traject/indexer/nokogiri_indexer: 1.3906 MiB traject/oai_pmh_nokogiri_reader: 1.3125 MiB http: 1.2813 MiB http/client: 0.5469 MiB (Also required by: http/response/body) qa: 5.8125 MiB activerecord-import: 5.7656 MiB activerecord-import/base: 5.7656 MiB activerecord-import/active_record/adapters/abstract_adapter: 4.4063 MiB active_record/connection_adapters/abstract_adapter: 4.3906 MiB (Also required by: active_record/connection_adapters/postgresql_adapter) active_record/connection_adapters/abstract/quoting: 2.6406 MiB active_support/multibyte/chars: 2.6094 MiB active_support/core_ext/string/behavior: 2.5313 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string.rb) active_record/connection_adapters/abstract/schema_statements: 0.4063 MiB active_record/connection_adapters/abstract/database_statements: 0.3281 MiB activerecord-import/import: 1.3125 MiB active_record/associations/collection_proxy: 0.6406 MiB bootstrap: 5.7969 MiB bootstrap/engine: 5.7813 MiB sassc-rails: 4.2031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc/rails: 4.2031 MiB sassc: 3.6094 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-2.4.0/lib/sassc/native: 2.8906 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc/rails/importer: 0.4375 MiB autoprefixer-rails: 1.5625 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/autoprefixer-rails-10.4.15.0/lib/autoprefixer-rails/railtie: 1.5469 MiB sprockets/railtie: 1.5469 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc/rails/railtie, sprockets/rails) sprockets/rails/context: 1.125 MiB action_view/helpers: 1.125 MiB action_view/helpers/sanitize_helper: 0.3594 MiB (Also required by: action_view/helpers/text_helper) rails-html-sanitizer: 0.3594 MiB loofah: 0.3125 MiB pg: 4.4219 MiB (Also required by: active_record/connection_adapters/postgresql_adapter) pg_ext: 4.25 MiB aws-sdk-mediaconvert: 4.0 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/client_api: 1.875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/client: 1.2031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/types: 0.75 MiB kithe: 3.6563 MiB kithe/engine: 3.5938 MiB active_record/base: 3.375 MiB active_record/querying: 0.3281 MiB active_record/relation: 0.3281 MiB scout_apm: 3.6406 MiB net/http: 1.0313 MiB (Also required by: scout_apm/error_service, /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/seahorse/client/net_http/connection_pool, and 9 others) resolv: 0.5781 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/ecs_credentials) oai: 2.7344 MiB oai/client: 1.6406 MiB faraday: 1.3906 MiB (Also required by: TOP, faraday/retry) oai/provider: 1.0938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/oai-1.2.1/lib/oai/provider/model/activerecord_caching_wrapper: 0.4531 MiB prawn: 2.3594 MiB (Also required by: prawn-svg, prawn-html) /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/transformation_stack: 0.3906 MiB matrix: 0.3906 MiB (Also required by: TOP) /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/document: 0.375 MiB ttfunk: 0.375 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/fonts/ttf, pdf/reader/font_descriptor) /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/font: 0.3438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/text: 0.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/text/formatted: 0.3125 MiB aws-sdk-cloudwatchlogs: 2.2969 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchlogs-1.71.0/lib/aws-sdk-cloudwatchlogs/client_api: 1.0156 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchlogs-1.71.0/lib/aws-sdk-cloudwatchlogs/types: 0.5625 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchlogs-1.71.0/lib/aws-sdk-cloudwatchlogs/client: 0.4688 MiB sprockets: 2.25 MiB (Also required by: sprockets/railtie, sprockets/rails/context, and 2 others) sprockets/environment: 1.2969 MiB sprockets/base: 1.2813 MiB (Also required by: sprockets/cached_environment) sprockets/configuration: 0.6875 MiB (Also required by: sprockets) browse-everything: 1.9844 MiB browse_everything: 1.9844 MiB browse_everything/retriever: 1.875 MiB typhoeus: 1.6406 MiB ethon: 1.0313 MiB ethon/curl: 0.4219 MiB ethon/easy: 0.3438 MiB honeybadger: 1.75 MiB honeybadger/init/rails: 1.7188 MiB honeybadger/ruby: 1.7188 MiB honeybadger/singleton: 1.7188 MiB honeybadger/agent: 1.5469 MiB honeybadger/config: 0.625 MiB resque: 1.5 MiB (Also required by: resque/pool, resque-heroku-signals) redis/namespace: 0.7656 MiB redis: 0.7031 MiB redis/client: 0.3594 MiB redis-client: 0.3594 MiB redis_client: 0.3594 MiB resque/worker: 0.4063 MiB (Also required by: resque/pool, resque/pool/pooled_worker) redis/distributed: 0.3594 MiB dalli: 1.4063 MiB activerecord/postgres_enum: 1.3906 MiB active_record/postgres_enum: 1.3906 MiB active_record/connection_adapters/postgresql_adapter: 1.0156 MiB active_record/connection_adapters/postgresql/schema_definitions: 0.4219 MiB ransack: 0.9531 MiB uppy-s3_multipart: 0.8906 MiB uppy/s3_multipart: 0.8906 MiB uppy/s3_multipart/app: 0.8906 MiB roda: 0.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/roda-3.73.0/lib/roda/request: 0.4375 MiB prawn-svg: 0.8438 MiB css_parser: 0.375 MiB blacklight_range_limit: 0.8438 MiB (Also required by: blacklight_range_limit/engine) blacklight_range_limit/controller_override: 0.7813 MiB browser: 0.7813 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser/browser: 0.75 MiB blacklight: 0.7813 MiB (Also required by: blacklight_range_limit/engine) jbuilder: 0.5156 MiB (Also required by: TOP) jbuilder/railtie: 0.4531 MiB jbuilder/jbuilder_template: 0.4219 MiB pdf-reader: 0.75 MiB pdf/reader: 0.75 MiB sitemap_generator: 0.7344 MiB sitemap_generator/builder: 0.4063 MiB attr_json: 0.5313 MiB devise: 0.5313 MiB device_detector: 0.3281 MiB ```

Rails 7.0 (cc8e840)

bundle exec derailed bundle:mem

``` TOP: 154.2031 MiB rails/all: 56.1094 MiB action_mailbox/engine: 31.2813 MiB action_mailbox: 31.25 MiB action_mailbox/mail_ext: 31.25 MiB action_mailbox/mail_ext/address_equality.rb: 29.5313 MiB mail/elements/address: 29.5313 MiB mail/parsers/address_lists_parser: 29.5 MiB mail: 1.7188 MiB (Also required by: TOP) net/smtp: 0.6094 MiB mail/field: 0.3125 MiB rails: 14.8438 MiB (Also required by: active_record/railtie, active_model/railtie, and 21 others) rails/application: 7.6406 MiB active_support/key_generator: 4.75 MiB openssl: 4.75 MiB (Also required by: active_support/message_verifier, active_support/message_encryptor, and 27 others) openssl.so: 2.5469 MiB /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/openssl/ssl: 2.1406 MiB ipaddr: 0.3438 MiB (Also required by: active_support/core_ext/object/json, /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/endpoints/url, and 4 others) yaml: 1.1719 MiB (Also required by: active_support/encrypted_configuration, rails/secrets, and 14 others) psych: 1.1719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/psych-5.1.0/lib/psych/visitors: 0.4688 MiB rails/engine: 0.8125 MiB rails/railtie: 0.4375 MiB (Also required by: global_id/railtie, vite_rails/engine, and 10 others) active_support/encrypted_configuration: 0.6719 MiB active_support/encrypted_file: 0.6406 MiB tempfile: 0.5938 MiB (Also required by: rails/secrets, rack/utils, and 15 others) tmpdir: 0.5313 MiB (Also required by: aws-sdk-s3/file_downloader, execjs/external_runtime) fileutils: 0.4531 MiB (Also required by: rack/utils, /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-test-2.1.0/lib/rack/test/uploaded_file, and 5 others) active_support: 4.2813 MiB (Also required by: active_support/railtie, active_support/i18n_railtie, and 20 others) active_support/logger: 3.5313 MiB active_support/logger_silence: 3.4063 MiB active_support/logger_thread_safe_level: 3.3594 MiB (Also required by: active_support/logger) concurrent: 3.3438 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/tzinfo-2.0.6/lib/tzinfo/string_deduper, /Users/jrochkind/.gem/ruby/3.2.2/gems/tzinfo-2.0.6/lib/tzinfo/data_source, and 5 others) concurrent/configuration: 1.5625 MiB (Also required by: concurrent/scheduled_task, concurrent/options, and 3 others) concurrent/delay: 1.1875 MiB (Also required by: concurrent/utility/processor_counter, concurrent) concurrent/concern/obligation: 0.6406 MiB (Also required by: concurrent/ivar) concurrent/atomic/event: 0.5781 MiB (Also required by: concurrent/executor/immediate_executor, concurrent/executor/ruby_thread_pool_executor, and 4 others) concurrent/synchronization/lockable_object: 0.5625 MiB (Also required by: concurrent/executor/abstract_executor_service, concurrent/delay, and 16 others) concurrent/executor/immediate_executor: 0.4688 MiB (Also required by: concurrent/configuration, concurrent/executors) concurrent/executor/abstract_executor_service: 0.3906 MiB (Also required by: concurrent/executor/ruby_executor_service, concurrent/executors) concurrent/atomics: 0.7031 MiB concurrent/atomic/reentrant_read_write_lock: 0.3438 MiB concurrent/promises: 0.3594 MiB active_support/dependencies/autoload: 0.6719 MiB (Also required by: view_component) active_support/inflector/methods: 0.6719 MiB (Also required by: active_support/inflector, active_support/core_ext/string/inflections, and 4 others) active_support/inflections: 0.6406 MiB (Also required by: active_support/inflector) active_support/inflector/inflections: 0.5781 MiB (Also required by: active_support/inflector) active_support/i18n: 0.5313 MiB (Also required by: active_support/inflector/transliterate, abstract_controller) i18n: 0.3594 MiB (Also required by: ransack/translate) active_support/railtie: 2.3906 MiB active_support/i18n_railtie: 2.3594 MiB rails/railtie/configuration: 2.3438 MiB (Also required by: rails/engine/configuration) rails/configuration: 2.3438 MiB active_support/core_ext/object: 2.25 MiB active_support/core_ext/object/json: 2.1875 MiB (Also required by: active_support/json/encoding) active_support/core_ext/time/conversions: 1.6563 MiB (Also required by: active_support/core_ext/date_time/conversions, active_support/core_ext/time/calculations) active_support/values/time_zone: 1.6563 MiB (Also required by: active_support/core_ext/date_time/conversions, active_support/time_with_zone) tzinfo: 1.4688 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/tzinfo-2.0.6/lib/tzinfo/with_offset: 0.7813 MiB action_dispatch/railtie: 0.3438 MiB (Also required by: action_controller/railtie) action_dispatch: 0.3438 MiB (Also required by: action_controller, active_storage/service) active_storage/engine: 3.8281 MiB (Also required by: action_mailbox/engine, action_text/engine) active_storage: 1.8281 MiB marcel: 1.7188 MiB marcel/magic: 1.7031 MiB marcel/tables: 1.6563 MiB action_dispatch/routing/route_set: 0.9844 MiB action_dispatch/journey: 0.7969 MiB action_dispatch/journey/router: 0.5313 MiB active_job/railtie: 0.4375 MiB (Also required by: action_mailer/railtie, rails/all) global_id/railtie: 0.4219 MiB (Also required by: globalid) active_support/core_ext/integer/time: 0.375 MiB (Also required by: active_encode/polling, active_support/time) active_storage/service/registry: 0.4063 MiB active_storage/service: 0.4063 MiB active_storage/log_subscriber: 0.3906 MiB active_record/railtie: 3.4844 MiB (Also required by: active_storage/engine, action_mailbox/engine) active_record: 2.4219 MiB (Also required by: active_storage, attr_json, and 6 others) arel: 1.1406 MiB arel/nodes: 0.6563 MiB arel/visitors: 0.375 MiB active_record/errors: 0.3594 MiB active_model/errors: 0.3125 MiB active_record/attribute_methods: 0.3281 MiB action_controller/railtie: 0.9844 MiB (Also required by: active_storage/engine, rails/all, and 4 others) action_controller: 0.6563 MiB (Also required by: responders) action_controller/metal/strong_parameters: 0.5313 MiB rack/test: 0.375 MiB action_text/engine: 1.7188 MiB action_text: 1.7188 MiB nokogiri: 1.7188 MiB (Also required by: loofah, traject/marc_reader, and 3 others) /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/extension: 0.5156 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/version/info) /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/3.2/nokogiri: 0.5156 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4: 0.4688 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html) /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4/element_description_defaults: 0.3125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml: 0.3906 MiB rails/test_unit/railtie: 0.5938 MiB rails/test_unit/line_filtering: 0.5938 MiB rails/test_unit/runner: 0.5938 MiB rake/file_list: 0.4688 MiB citeproc/ruby: 11.2344 MiB csl: 7.0625 MiB (Also required by: csl/styles) csl/schema: 6.375 MiB citeproc: 3.7969 MiB citeproc/variable: 1.1094 MiB citeproc/item: 0.9219 MiB citeproc/names: 0.875 MiB citeproc/date: 0.6875 MiB aws-sdk-cloudwatchevents: 9.4688 MiB aws-sdk-core: 7.7969 MiB (Also required by: aws-sdk-cloudwatchlogs, aws-sdk-mediaconvert, and 2 others) /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/xml: 2.8438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/xml/parser: 2.5938 MiB aws-sdk-core/xml/parser/engines/oga: 2.5781 MiB oga: 2.5781 MiB (Also required by: prawn_html/html_parser) oga/xpath/lexer: 0.6406 MiB oga/html/entities: 0.5938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-sts: 0.9531 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-sts/client: 0.6563 MiB seahorse: 0.9375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-ssooidc: 0.3906 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/rest: 0.3906 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchevents-1.67.0/lib/aws-sdk-cloudwatchevents/client_api: 0.7656 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchevents-1.67.0/lib/aws-sdk-cloudwatchevents/types: 0.625 MiB aws-sdk-mediaconvert: 8.2031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/client_api: 5.9844 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/client: 1.1875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/types: 0.875 MiB aws-sdk-s3: 7.6875 MiB (Also required by: uppy/s3_multipart/client) /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/client_api: 2.2188 MiB aws-sdk-kms: 1.4688 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-kms-1.72.0/lib/aws-sdk-kms/client_api: 0.7813 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-kms-1.72.0/lib/aws-sdk-kms/client: 0.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/types: 1.1406 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/client: 0.7969 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/customizations: 0.7656 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/endpoints: 0.4375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/endpoint_provider: 0.3125 MiB prawn: 5.6875 MiB (Also required by: prawn-svg, prawn-html) ttfunk: 1.2344 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/fonts/ttf, pdf/reader/font_descriptor) /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/transformation_stack: 0.8438 MiB matrix: 0.8438 MiB (Also required by: TOP) /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/text: 0.7344 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/text/formatted: 0.5313 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/font: 0.7031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/fonts/ttf: 0.3594 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/fonts/dfont, /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/fonts/otf) pdf/core: 0.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/document: 0.4844 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn/graphics: 0.4531 MiB traject: 5.6094 MiB traject/indexer: 4.3438 MiB traject/solr_json_writer: 3.0156 MiB httpclient: 2.9844 MiB httpclient/cookie: 1.7031 MiB http-cookie: 1.6875 MiB http/cookie: 1.6875 MiB (Also required by: http/cookie_jar) domain_name: 1.6094 MiB domain_name/etld_data: 1.5156 MiB httpclient/session: 0.5156 MiB (Also required by: httpclient/auth) httpclient/util: 0.4375 MiB (Also required by: httpclient/http, httpclient/cookie) addressable/uri: 0.4375 MiB (Also required by: http/uri, addressable, and 3 others) traject/marc_reader: 0.875 MiB marc: 0.8594 MiB (Also required by: traject/ndj_reader, marc/fastxmlwriter) /Users/jrochkind/.gem/ruby/3.2.2/gems/marc-1.2.0/lib/marc/xmlwriter: 0.8125 MiB rexml/document: 0.7813 MiB (Also required by: prawn-svg, oai/provider) /Users/jrochkind/.gem/ruby/3.2.2/gems/rexml-3.2.6/lib/rexml/element: 0.5156 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/rexml-3.2.6/lib/rexml/attribute: 0.3281 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/rexml-3.2.6/lib/rexml/xpath_parser) traject/macros/marc21: 0.3594 MiB (Also required by: traject) traject/indexer/nokogiri_indexer: 1.2344 MiB traject/oai_pmh_nokogiri_reader: 1.2344 MiB http: 1.1563 MiB http/client: 0.4844 MiB (Also required by: http/response/body) bootstrap: 4.9531 MiB bootstrap/engine: 4.9063 MiB sassc-rails: 2.8438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc/rails: 2.8438 MiB sassc: 2.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-2.4.0/lib/sassc/native: 2.5625 MiB autoprefixer-rails: 2.0469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/autoprefixer-rails-10.4.15.0/lib/autoprefixer-rails/railtie: 1.9844 MiB sprockets/railtie: 1.9375 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc/rails/railtie, sprockets/rails) sprockets/rails/context: 1.5938 MiB action_view/helpers: 1.5625 MiB action_view/helpers/sanitize_helper: 0.7344 MiB (Also required by: action_view/helpers/text_helper) rails-html-sanitizer: 0.7344 MiB loofah: 0.6875 MiB pg: 4.4531 MiB (Also required by: active_record/connection_adapters/postgresql_adapter) pg_ext: 4.2031 MiB scout_apm: 4.3594 MiB net/http: 1.2969 MiB (Also required by: scout_apm/error_service, /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/seahorse/client/net_http/connection_pool, and 9 others) resolv: 0.4531 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/ecs_credentials) pp: 0.3594 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/rexml-3.2.6/lib/rexml/xpath_parser, faraday/logging/formatter) resque: 3.8906 MiB (Also required by: resque/pool, resque-heroku-signals) redis/namespace: 2.5 MiB redis: 2.2031 MiB redis/commands: 1.375 MiB redis/commands/sorted_sets: 0.9688 MiB redis/client: 0.7188 MiB redis-client: 0.7188 MiB redis_client: 0.7188 MiB resque/worker: 0.5625 MiB (Also required by: resque/pool, resque/pool/pooled_worker) redis/distributed: 0.3281 MiB irb: 3.6875 MiB reline: 2.2188 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.8.1/lib/irb/input-method, /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.8.1/lib/irb/color, and 2 others) reline/line_editor: 1.0781 MiB reline/unicode: 0.3594 MiB ripper: 0.8438 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.8.1/lib/irb/ruby-lex, /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.8.1/lib/irb/color) ripper/sexp: 0.4688 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.8.1/lib/irb/context: 0.3281 MiB kithe: 3.375 MiB kithe/engine: 3.3438 MiB active_record/base: 3.1875 MiB active_record/relation: 0.5 MiB browse-everything: 2.625 MiB browse_everything: 2.625 MiB browse_everything/retriever: 2.5781 MiB typhoeus: 2.25 MiB ethon: 1.4531 MiB ethon/easy: 0.6563 MiB ethon/curl: 0.5313 MiB sprockets: 2.4688 MiB (Also required by: sprockets/railtie, sprockets/rails/context, and 2 others) sprockets/environment: 0.7969 MiB sprockets/base: 0.7969 MiB (Also required by: sprockets/cached_environment) sprockets/configuration: 0.3281 MiB (Also required by: sprockets) prawn-svg: 2.3594 MiB css_parser: 0.7656 MiB prawn/svg/elements: 0.6406 MiB ransack: 2.0469 MiB active_support/core_ext: 1.5938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.0.8/lib/active_support/core_ext/kernel.rb: 1.5156 MiB active_support/core_ext/kernel/concern: 1.5156 MiB qa: 1.9531 MiB activerecord-import: 1.9219 MiB activerecord-import/base: 1.9219 MiB activerecord-import/import: 1.0 MiB active_record/associations/collection_proxy: 0.5625 MiB activerecord-import/active_record/adapters/abstract_adapter: 0.9219 MiB active_record/connection_adapters/abstract_adapter: 0.9063 MiB (Also required by: active_record/connection_adapters/postgresql_adapter) aws-sdk-cloudwatchlogs: 1.5781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchlogs-1.71.0/lib/aws-sdk-cloudwatchlogs/client_api: 0.7656 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchlogs-1.71.0/lib/aws-sdk-cloudwatchlogs/types: 0.6563 MiB activerecord/postgres_enum: 1.5313 MiB active_record/postgres_enum: 1.5313 MiB active_record/connection_adapters/postgresql_adapter: 1.1563 MiB active_record/connection_adapters/postgresql/schema_definitions: 0.375 MiB active_record/connection_adapters/postgresql/oid: 0.3125 MiB attr_json: 1.5156 MiB attr_json/record: 0.9531 MiB attr_json/attribute_definition: 0.5 MiB (Also required by: attr_json/attribute_definition/registry, attr_json/model) attr_json/type/array: 0.5 MiB active_model/type: 0.5 MiB (Also required by: active_record/type, attr_json/model) attr_json/type/container_attribute: 0.3438 MiB active_record/type: 0.3125 MiB attr_json/model: 0.5313 MiB active_model/model: 0.3594 MiB active_model/api: 0.3594 MiB oai: 1.0 MiB oai/client: 0.6094 MiB faraday: 0.5469 MiB (Also required by: TOP, faraday/retry) oai/provider: 0.3906 MiB pdf-reader: 0.8906 MiB pdf/reader: 0.875 MiB devise: 0.8125 MiB blacklight: 0.8125 MiB (Also required by: blacklight_range_limit/engine) jbuilder: 0.5 MiB (Also required by: TOP) jbuilder/railtie: 0.4375 MiB jbuilder/jbuilder_template: 0.4375 MiB jbuilder/collection_renderer: 0.3281 MiB browser: 0.5781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser/browser: 0.5781 MiB honeybadger: 0.5313 MiB honeybadger/init/rails: 0.5156 MiB honeybadger/ruby: 0.5156 MiB honeybadger/singleton: 0.5 MiB honeybadger/agent: 0.3906 MiB terser: 0.5156 MiB execjs: 0.3438 MiB (Also required by: /Users/jrochkind/.gem/ruby/3.2.2/gems/autoprefixer-rails-10.4.15.0/lib/autoprefixer-rails/processor) execjs/runtimes: 0.3281 MiB device_detector: 0.4063 MiB dalli: 0.4063 MiB uppy-s3_multipart: 0.3594 MiB uppy/s3_multipart: 0.3594 MiB uppy/s3_multipart/app: 0.3594 MiB lockbox: 0.3594 MiB ruby-progressbar: 0.3438 MiB ruby-progressbar/base: 0.3281 MiB ```

Comparison

This particular report only shows 3 meg difference, which is no big deal. The extra 3MB does seem to be between Rails 7.1 and 7.0 specifically.... but this isn't that much, and doesn't explain our observed results yet.

There are some clues in there as to what gems are taking up a lot of RAM... but mostly just taht we have a lot of dependencies, some of which are bigger than one would like, but no single star of badness I think.

[jrochkind]

OK... have env set up, hacikly....

When trying to run bundle exec derailed exec perf:mem, getting an error that I do not get when actually running my app in production (including locally)

/Users/jrochkind/.gem/ruby/3.2.2/gems/activestorage-7.0.8/lib/active_storage/engine.rb:147:in `block (2 levels) in ': Couldn't find Active Storage configuration in /Users/jrochkind/code/scihist_digicoll/config/storage.yml (RuntimeError)

This app doesn't actually use activestorage, although I guess does include the dependency (as default rails)... indeed it does not have a config/storage.yml, but ordinarily it does not need one as it's not actually using ActiveStorage... not sure why it's complaining about this only under bundle exec derailed exec perf:mem or what to do about it...

I'll think on it some more, or perhaps it's time to take Ben up on pairing offer.

This sort of thing is why I haven't previously gotten further with this sort of tool, it often turns into some confusing yak shaving.

[jrochkind]

OK, our app actually doesn't intentionally load activestorage at all.... Which is making me wonder why derailed is, and how/if derailed perf:mem differs from an actually running app. Here's our config/application.rb

require "rails"
# Pick the frameworks you want:
require "active_model/railtie"
require "active_job/railtie"
require "active_record/railtie"
# require "active_storage/engine"
require "action_controller/railtie"
require "action_mailer/railtie"
require "action_view/railtie"
require "action_cable/engine"
require "sprockets/railtie"
require "rails/test_unit/railtie"

Note how we pick and choose parts of rails, and do not include activestorage. However, perhaps some dependency we have requires activestorage, not sure.

Nevertheless, simply adding a default template config/storage.yml is sufficient to get derailed perf:mem to run. Output forthcoming.

[jrochkind]

OK, with config/storage.yml added.

Rails 7.1 (3afd9c5 )

CUT_OFF=1 PATH_TO_HIT=/catalog bundle exec derailed exec perf:mem

``` Booting: production Database 'scihist_digicoll_development' already exists Endpoint: "/catalog" ## Impact of `require ` on RAM Showing all `require ` calls that consume 1 MiB or more of RSS Configure with `CUT_OFF=0` for all entries or `CUT_OFF=5` for few entries Note: Files only count against RAM on their first load. If multiple libraries require the same file, then the 'cost' only shows up under the first library I, [2023-11-29T16:09:48.366346 #91076] INFO -- : [fc95ee9f-2d4a-4954-8de2-f998ad79e03f] method=GET path=/catalog format=html controller=CatalogController action=index status=200 allocations=424981 duration=617.10 view=116.72 db=51.54 ua= ip= TOP: 208.8281 MiB application: 127.875 MiB /Users/jrochkind/code/scihist_digicoll/config/boot: 47.0781 MiB bootsnap/setup: 47.0781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchevents-1.67.0/lib/aws-sdk-cloudwatchevents.rb: 7.7031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core.rb: 6.6563 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/xml: 1.9063 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/xml/parser: 1.7656 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/aws-sdk-core/xml/parser/engines/oga.rb: 1.6875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/oga-3.4/lib/oga.rb: 1.6406 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.186.0/lib/seahorse.rb: 1.2344 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3.rb: 7.3594 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/client_api: 2.6563 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-kms-1.72.0/lib/aws-sdk-kms.rb: 1.4375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/types: 1.3438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject.rb: 6.1719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/indexer.rb: 4.2344 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/solr_json_writer.rb: 2.5625 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/httpclient-2.8.3/lib/httpclient.rb: 2.5 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/marc_reader.rb: 1.0469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/marc-1.2.0/lib/marc.rb: 1.0313 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/indexer/nokogiri_indexer.rb: 1.9219 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/oai_pmh_nokogiri_reader.rb: 1.875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/http-5.1.1/lib/http.rb: 1.8594 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/railtie.rb: 5.8281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/context.rb: 2.7344 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers.rb: 2.7188 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/sanitize_helper.rb: 1.3438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/rails-html-sanitizer-1.6.0/lib/rails-html-sanitizer.rb: 1.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah.rb: 1.1563 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-4.2.1/lib/sprockets.rb: 2.6406 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-4.2.1/lib/sprockets/environment.rb: 1.0469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby.rb: 5.375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc.rb: 3.5156 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/csl-2.0.0/lib/csl.rb: 1.4688 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/csl-2.0.0/lib/csl/schema.rb: 1.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/pg-1.5.4/lib/pg.rb: 4.2813 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/pg-1.5.4/lib/pg_ext.bundle: 4.2188 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/scout_apm-5.3.5/lib/scout_apm.rb: 3.9531 MiB /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/net/http.rb: 1.25 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browse-everything-1.3.0/lib/browse-everything.rb: 3.5781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browse-everything-1.3.0/lib/browse_everything.rb: 3.5781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browse-everything-1.3.0/lib/browse_everything/retriever.rb: 3.5469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/typhoeus-1.4.0/lib/typhoeus.rb: 3.3594 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/ethon-0.16.0/lib/ethon.rb: 2.6563 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert.rb: 3.4688 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/client_api: 2.4531 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.9.0/lib/irb.rb: 3.4063 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/reline-0.4.0/lib/reline.rb: 2.7031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/reline-0.4.0/lib/reline/line_editor.rb: 1.0469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/bootstrap-4.6.2/lib/bootstrap.rb: 3.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/bootstrap-4.6.2/lib/bootstrap/engine.rb: 3.2031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc-rails.rb: 3.2031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc/rails: 3.2031 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-2.4.0/lib/sassc.rb: 2.8906 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-2.4.0/lib/sassc/native: 2.5469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/resque-2.6.0/lib/resque.rb: 2.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/redis-namespace-1.11.0/lib/redis/namespace.rb: 1.0938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/redis-5.0.8/lib/redis.rb: 1.0 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/qa-5.11.0/lib/qa.rb: 2.2813 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-import-1.5.0/lib/activerecord-import.rb: 2.1875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-import-1.5.0/lib/activerecord-import/base.rb: 2.1875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-import-1.5.0/lib/activerecord-import/active_record/adapters/abstract_adapter.rb: 1.1719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract_adapter.rb: 1.1719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/kithe-2.13.0/lib/kithe.rb: 2.125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/kithe-2.13.0/lib/kithe/engine.rb: 1.9844 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/base.rb: 1.75 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/devise-4.9.3/lib/devise.rb: 1.875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-postgres_enum-2.0.1/lib/activerecord/postgres_enum.rb: 1.7188 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-postgres_enum-2.0.1/lib/active_record/postgres_enum.rb: 1.7188 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql_adapter.rb: 1.375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/prawn-2.4.0/lib/prawn.rb: 1.6875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/ransack-4.1.1/lib/ransack.rb: 1.6406 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/lib/blacklight.rb: 1.4688 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json.rb: 1.4531 MiB /Users/jrochkind/code/scihist_digicoll/app/controllers/oai_pmh_controller.rb: 3.7969 MiB /Users/jrochkind/code/scihist_digicoll/app/controllers/catalog_controller.rb: 2.25 MiB /Users/jrochkind/code/scihist_digicoll/app/components/thumb_component.rb: 1.7969 MiB /Users/jrochkind/code/scihist_digicoll/app/models/asset.rb: 1.7813 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/qa-5.11.0/app/models/qa/linked_data/config/context_property_map.rb: 1.3125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/ldpath-1.2.0/lib/ldpath.rb: 1.2969 MiB ```

Rails 7.0 (cc8e84042)

CUT_OFF=1 PATH_TO_HIT=/catalog bundle exec derailed exec perf:mem

``` Booting: production Database 'scihist_digicoll_development' already exists Endpoint: "/catalog" ## Impact of `require ` on RAM Showing all `require ` calls that consume 1 MiB or more of RSS Configure with `CUT_OFF=0` for all entries or `CUT_OFF=5` for few entries Note: Files only count against RAM on their first load. If multiple libraries require the same file, then the 'cost' only shows up under the first library TOP: 198.1406 MiB application: 134.25 MiB /Users/jrochkind/code/scihist_digicoll/config/boot: 55.8906 MiB bootsnap/setup: 55.875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject.rb: 9.5469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/indexer.rb: 6.8594 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/solr_json_writer.rb: 3.5781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/httpclient-2.8.3/lib/httpclient.rb: 3.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/httpclient-2.8.3/lib/httpclient/util.rb: 1.3125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/addressable-2.8.5/lib/addressable/uri.rb: 1.2656 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/marc_reader.rb: 2.2813 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/marc-1.2.0/lib/marc.rb: 2.2656 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/marc-1.2.0/lib/marc/xmlwriter: 1.8281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/rexml-3.2.6/lib/rexml/document.rb: 1.8125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/rexml-3.2.6/lib/rexml/element: 1.4531 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/indexer/nokogiri_indexer.rb: 2.5469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/traject-3.8.1/lib/traject/oai_pmh_nokogiri_reader.rb: 2.4375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/http-5.1.1/lib/http.rb: 2.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/http-5.1.1/lib/http/client.rb: 1.0313 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/kithe-2.11.0/lib/kithe.rb: 6.2969 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/kithe-2.11.0/lib/kithe/engine.rb: 6.0 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/base.rb: 5.6094 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby.rb: 5.875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc.rb: 3.7344 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc/names.rb: 1.3906 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/csl-2.0.0/lib/csl.rb: 1.5938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/csl-2.0.0/lib/csl/schema.rb: 1.3438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/railtie.rb: 5.6406 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-4.2.1/lib/sprockets.rb: 2.9844 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-4.2.1/lib/sprockets/environment.rb: 1.1563 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-4.2.1/lib/sprockets/base.rb: 1.1406 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/context.rb: 2.2188 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.0.8/lib/action_view/helpers.rb: 2.1875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchevents-1.67.0/lib/aws-sdk-cloudwatchevents.rb: 5.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core.rb: 4.6563 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/xml: 1.7969 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/xml/parser: 1.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/xml/parser/engines/oga.rb: 1.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/oga-3.4/lib/oga.rb: 1.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/pg-1.5.4/lib/pg.rb: 4.3594 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/pg-1.5.4/lib/pg_ext.bundle: 4.2969 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/scout_apm-5.3.5/lib/scout_apm.rb: 4.3281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3.rb: 3.9688 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-kms-1.72.0/lib/aws-sdk-kms.rb: 1.8281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/types: 1.125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert.rb: 2.9844 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/client_api: 1.5 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-mediaconvert-1.119.0/lib/aws-sdk-mediaconvert/types: 1.2969 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/bootstrap-4.6.2/lib/bootstrap.rb: 2.9219 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/bootstrap-4.6.2/lib/bootstrap/engine.rb: 2.9063 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc-rails.rb: 2.8281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-rails-2.1.2/lib/sassc/rails: 2.8281 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-2.4.0/lib/sassc.rb: 2.6719 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/sassc-2.4.0/lib/sassc/native: 2.5938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browse-everything-1.2.0/lib/browse-everything.rb: 2.6875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browse-everything-1.2.0/lib/browse_everything.rb: 2.6875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browse-everything-1.2.0/lib/browse_everything/retriever.rb: 2.5938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/typhoeus-1.4.0/lib/typhoeus.rb: 2.375 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/ethon-0.16.0/lib/ethon.rb: 1.5313 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/irb-1.8.1/lib/irb.rb: 2.2344 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/reline-0.3.9/lib/reline.rb: 1.4063 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/pdf-reader-2.11.0/lib/pdf-reader.rb: 2.1875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/pdf-reader-2.11.0/lib/pdf/reader.rb: 2.1875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/oai-1.2.1/lib/oai.rb: 2.0469 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/oai-1.2.1/lib/oai/client.rb: 1.3125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/faraday-2.7.11/lib/faraday.rb: 1.0313 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/resque-2.6.0/lib/resque.rb: 2.0313 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-postgres_enum-2.0.1/lib/activerecord/postgres_enum.rb: 1.6875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-postgres_enum-2.0.1/lib/active_record/postgres_enum.rb: 1.6875 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql_adapter.rb: 1.2969 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/aws-sdk-cloudwatchlogs-1.71.0/lib/aws-sdk-cloudwatchlogs.rb: 1.4063 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/qa-5.10.0/lib/qa.rb: 1.3125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-import-1.5.0/lib/activerecord-import.rb: 1.3125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-import-1.5.0/lib/activerecord-import/base.rb: 1.3125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser.rb: 1.125 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser/browser: 1.0938 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.34.0/lib/blacklight.rb: 1.0781 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/qa-5.10.0/app/services/qa/linked_data/deep_sort_service.rb: 8.25 MiB /Users/jrochkind/code/scihist_digicoll/app/controllers/oai_pmh_controller.rb: 2.4531 MiB /Users/jrochkind/code/scihist_digicoll/app/controllers/catalog_controller.rb: 2.375 MiB /Users/jrochkind/code/scihist_digicoll/app/components/thumb_component.rb: 1.6406 MiB /Users/jrochkind/code/scihist_digicoll/app/models/asset.rb: 1.6406 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/qa-5.10.0/app/models/qa/linked_data/config/context_property_map.rb: 1.3438 MiB /Users/jrochkind/.gem/ruby/3.2.2/gems/ldpath-1.2.0/lib/ldpath.rb: 1.3438 MiB /Users/jrochkind/code/scihist_digicoll/app/controllers/collection_show_controller.rb: 1.0156 MiB ```

Conclusion

Still nothing obvious to me.

Rails 7.1 is about 10M bigger... haven't totally analyzed where that is coming from, but it does not seem enough to explain the observed discrepency (or is it?)

In general, the weight seems just like a whole bunch of dependencies, a handful of of which are bigger than they probably should be to some extent, but no specific egregious ones, just an accumulation?

Still doesn't totally explain why 3 puma workers with 3 threads are touching our memory limits in a 2.5 G heroku dyno.... that'd be like 800 megs per dyno, when this analysis is only showing a couple hundred -- maybe it really isn't memory just at "boot time" after all, but memory that gets taken under actual use very soon after boot?

We tested with search results page /catalog instead of just the home page, but we could also try testing with a large item-detail page maybe?

@BenSheldon I made an appointment with you tomorrow at 2pm, if that time remains good, I figure you'll reach out to me somehow to tell me how we will connect? I really appreciate it, I'm looking forward to seeing what else we can figure out or if you have any advice! So far I'm still fairly stumped about even if I've yet found anything significant or actionable, or what next steps might be.

[bensheldon]

@jrochkind looking forward to chatting today 👍🏻 I'll add a Zoom link to our calendar invite.

That's a bummer we haven't found any smoke yet! When we meet we can go over it together and see if there's anything being missed. We might want to also experiment with perf:mem_over_time

And if none of that bears fruit, we might want to move on to trying to get a memory dump of the running process. I'll write down some options for that below:

• For all of these we'll need to use heroku ps:exec to ssh into one of your existing web dynos that is using lots of memory. That on its own might give us a little insight into double-checking that it is the workers using that memory, and not the Puma cluster controller process (though now I'm wondering whether we'll even have ps available in a dyno, let alone other typical linux tools)
• The batteries-included option is to use sigdump to dump out some memory summaries. That should be safe to run in production.
• If we need more, we can use rbtrace. On its own, rbtrace is safe to leave enabled in production, but it's likely we'll want to subsequently use something like heap-profiler and that might require restarting the process with a new, local ENV variable to enable tracing... waiting for the memory to grow, and then dumping the data.

[jrochkind]

@bensheldon you do I think have ps avail in a dyno, but you can't actually ssh into an existing dyno.

I am pretty sure: The way heroku works, there actually isn't a way to get into a running web dyno, only into new one-off created-just-for-you-just-now dynos (that do not handle web traffic).

[jrochkind]

@bensheldon oh wait, I didn't know about ps:exec, what, is this new? Anyway, clearly things I don't know about heroku.

[bensheldon]

heroku run will spin up a new dyno, but ps:exec allows ssh'ing into an existing dyno eg heroku ps:exec --dyno=web.2.

It's new as of 2017: https://devcenter.heroku.com/changelog-items/1264

[jrochkind]

So unfortunately a pairing session with @bensheldon revealed no smoking guns, and only enduring mystery:

The derailed tools suggest that an individual process should be 200-300M (which is fairly reasonable -- especially with anticicpated "copy on write" memory savings), but we're seeing more like 400-500 meg, and are not sure if we're seeing the "copy on write" memory savings we expect (this can be confusing to understand from OS RAM reports, and I am not confident I know what I'm looking at).

Ben suggests using the rbtrace gem in production to see if we can more insight into where the RAM is going. (rbtrace alas does not work on Mac at present, so I wasn't able to play with it in dev to get a sense of what it might tell us).

Ben also suggests using heroku's puma stats, which we already had as a recent issue at #2446. We should get on that, and decide at meeting today who will do it. (It does require a new gem and deploy).

Meanwhile however, derailed perf:objects command is showing me a lot of object allocations from our code. While theoretically these should be temporary and GC'able and not impact long-term RAM consumption... they don't help, and practice does not always match this theory, it strikes me as worth it to spend a bit of time optimizing memory allocations, at least the low-hanging fruit -- which is often string literals that should be "frozen" with frozen string pragma or otherwise. See https://medium.com/@seanrobenalt/what-are-ruby-frozen-string-literals-ccf97907623a

The perf:objects report reveals lots of memory allocations in: kithe, attr-json, faster-s3-url and local code -- all code I'm maintainer on, and which includes a lot of low-hanging fruit strings we can optimize.

It also reveals a lot of memory allocations from blacklight, which we aren't going to try to touch right now -- especially since we are currently stuck on blacklight 7 due to some blockers in BL 8. (See #2164). It also reveals the deprecation gem as a culprit, but looking at the stack trace it is probably covering up for something else not truly the gem itself -- the "something else" would still be Blacklight, as nothing else in our stack uses this gem (which actually comes from BL community). BL8 does not use deprecation gem.

BL is quite difficult to PR to, so if BL is a culprit, we may decide the cost/benefit does not look good for spending time on it.

I am also a bit worried that our own attr_json might be a culprit in hard to fix ways, the ways it implements JSON embedding for ActiveRecord might just use a lot of RAM. ActiveRecord in general uses a lot of RAM, but it's code has been subject to many hours of developer optimizations in somewhat sophisticated ways recently, which attr_json doesn't have the resources for.

[jrochkind]

Comparing derailed perf:objects reports between our app on 7.0 and 7.1..... for a works detail page it definitely does not get worse in 7.1, allocations are actually fewer in 7.1 dependencies.

Nevertheless, we'll work to improve them while we're looking at them.

[jrochkind]

OK, after doing more work... I think my efforts to reduce object allocations are actually an irrelevant red herring.

But I noticed a pretty important thing in the derailed perf:objects reports between the 7.0 and 7.1 versions... huge differences in retained objects.

@bensheldon I'm curious what you make of this, or how to investigate it further... I'm wondering if Rails 7.1 could actually have some kind of memory leak in AR... which we are maybe exascerbating by our use of shrine and attr_json, both of which hook into AR? And by sometimes fetching hundreds of AR objects (using both) per page?

But also I"m not sure how derailed counts "retained"; it can very from run to run... but not nearly as much variance as this 7.0 run vs 7.1 run shows....

TEST_COUNT=10 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:objects

my app under Rails 7.0

retained memory by gem
-----------------------------------
     12728  nokogiri-1.15.4-arm64-darwin
      3576  rack-2.2.8
      1376  loofah-2.21.3
       640  racc-1.7.1
       424  activesupport-7.0.8
       392  other
       324  actionview-7.0.8
       160  random
       152  timeout-0.4.0
       112  actionpack-7.0.8
        40  honeybadger-5.2.1
        40  request_store-1.5.1

Hardly any, trivial.

My app under Rails 7.1

retained memory by gem
-----------------------------------
  91668560  activerecord-7.1.2
  81789016  activemodel-7.1.2
  21417316  activesupport-7.1.2
  17258880  json
   3023648  shrine-3.5.0
    178560  attr_json-2.2.0
    152224  scihist_digicoll/app
     69120  access-granted-1.3.3
     26096  rack-2.2.8
     24008  actionpack-7.1.2
     18080  actionview-7.1.2
     13987  nokogiri-1.15.4-arm64-darwin
      6560  warden-1.2.9
      2464  set
      2440  citeproc-1.0.10
      2242  citeproc-ruby-2.0.0
      1624  other
      1504  loofah-2.21.4
      1096  observer
       960  monitor
       800  blacklight-7.35.0
       640  devise-4.9.3
       640  racc-1.7.3
       640  random
       320  sprockets-rails-3.4.2
       296  csl-2.0.0
       160  browser-5.3.1
       160  view_component-3.7.0
       152  timeout-0.4.1
        40  honeybadger-5.3.0
        40  request_store-1.5.1

Pretty huge!

retained memory by location
-----------------------------------
 110795500  /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72
 100111752  /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/type/string.rb:35
  83327867  /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:72
  21575360  /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216
   1967360  /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:590
   1953440  /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:169
   1058800  /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute.rb:9
    964200  /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine.rb:122

OK, that huge amount of retained memory from active_record/connection_adapters/postgresql/database_statements.rb:72 does not seem good.

https://github.com/rails/rails/blob/v7.1.2/activerecord/lib/active_record/connection_adapters/postgresql/database_statements.rb#L72

Why would this be "retained"... or is this an instrumentation error/inexactness?

I think JSON in there because we make heavy use of postgres jsonb (and things like shrine and attr_json that de/serialize json from AR).

what?

What's going on with this? The version of ruby did not change btw.

[bensheldon]

I think that retained memory report is exactly what we're looking for 🎯 That looks pretty damning to me: I don't think anything should be retaining 100MB+. And that's nice that you can generate it locally 👍🏻

Can you try running perf:heap_diff? https://github.com/zombocom/derailed_benchmarks/blob/main/README.md#i-want-more-heap-dumps

One thing about those "retained memory by location": I believe that is where the retained object originates, but it doesn't say exactly who/where is continuing to hold a reference to the object. So that would require tracing through the application. I am dubious that the problem is in Rails, but that something outside of Rails is holding onto them (e.g. putting them into some global array somewhere, or something of that flavor).

[bensheldon]

Totally aside, please don't be distracted but... Is this line loading up everything, multiple times to load WorksController#show? It seems like at a minimum it's missing ||= memoization, but I also get the sense it is loading a lot:

https://github.com/sciencehistory/scihist_digicoll/blob/ca8a86e7524a3a69b8f044382ab6bdfa6043cc6c/app/components/work_file_list_show_component.rb#L81-L85

[jrochkind]

@bensheldon Hmm, good find that I think it clearly does require memoization!

it's loading one AR object for every "page" of our digitized works -- with maybe 1-3 associated records for each page. We might have anywhere from 10-700 pages -- median probably around 100. So that is kind of a lot, but the current design of our app (internal and intended UX) kind of requires it.

But unless there's something I'm missing, it clearly should be memoized, not sure how we missed that, thank you! Assuming tests pass and it works, that could be a huge performance gain as well as RAM, and explain some mysteries!

Still doesn't explain why it would differ between 7.0 and 7.1 though.

[jrochkind]

Thanks @bensheldon! ... the thing is, almost nothing changes between these two branches except the Rails version (including updating to new Rails config defaults), so I can't figure out what my app could possibly be doing differently to retain 100MB more in the diff.... but I guess that's the task before us....

Note also my "retained" memory counts are not exactly the same from run to run, although they are in the same ballpark. Whereas hte "allocated" memory counts actually are exactly the same from run to run I think? Is this all expected? It makes me wonder how accurate this "retained" instrumentation really is and what it's measuring.

ok, heap_diff. Try with TEST_COUNT of 100 (rather than default 1000), which already requires me to wait a while, maybe could have been even fewer.

I'm not really sure what to do with this.... I am glad I ran it against the Rails 7.0 version too, because it didn't actually differ in quite the ways I expected... Curious what you think of these... if these lines mentioned aren't what is actually retaining, not really sure how to go about figuring out what is retaining, especially when these seem kind of all over the place...

my app on Rails 7.1

jrochkind-shi scihist_digicoll (derailed_rails71_3afd9c514 ?) $ TEST_COUNT=100 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:heap_diff
Booting: production
Database 'scihist_digicoll_development' already exists
Endpoint: "/works/dvoykwl"
Running 100 times
Heap file generated: "tmp/2023-12-04T16:58:16-05:00-heap-0.ndjson"
Running 100 times
Heap file generated: "tmp/2023-12-04T16:58:16-05:00-heap-1.ndjson"
Running 100 times
Heap file generated: "tmp/2023-12-04T16:58:16-05:00-heap-2.ndjson"

Diff

``` Diff ==== Retained STRING 106872 objects of size 9795320/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained HASH 45362 objects of size 9190688/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained STRING 23036 objects of size 149478193/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained OBJECT 18972 objects of size 758880/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine.rb:122 Retained OBJECT 15225 objects of size 1218000/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute.rb:9 Retained STRING 14816 objects of size 113364404/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/type/string.rb:35 Retained DATA 8564 objects of size 736504/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained OBJECT 4042 objects of size 646720/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:320 Retained HASH 2823 objects of size 2619744/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:169 Retained OBJECT 2820 objects of size 902400/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/persistence.rb:640 Retained ARRAY 2536 objects of size 102520/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:26 Retained ARRAY 2481 objects of size 100320/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:30 Retained OBJECT 2462 objects of size 98480/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/dirty.rb:377 Retained OBJECT 2445 objects of size 195600/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:16 Retained HASH 2437 objects of size 409416/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:28 Retained HASH 2422 objects of size 96880/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_mutation_tracker.rb:71 Retained HASH 2409 objects of size 404200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:339 Retained HASH 2369 objects of size 420792/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:22 Retained OBJECT 2116 objects of size 338560/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/entity.rb:71 Retained STRING 1970 objects of size 78800/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained OBJECT 1951 objects of size 78040/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained DATA 1705 objects of size 122760/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:112 Retained HASH 1702 objects of size 1579456/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:590 Retained HASH 1675 objects of size 281400/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/attacher.rb:45 Retained ARRAY 324 objects of size 87784/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained HASH 79 objects of size 13272/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:300 Retained OBJECT 78 objects of size 3120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:151 Retained HASH 70 objects of size 11760/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:382 Retained HASH 55 objects of size 7960/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:28 Retained OBJECT 51 objects of size 8160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/delegation.rb:132 Retained STRUCT 36 objects of size 1440/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/table.rb:89 Retained OBJECT 29 objects of size 2320/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:59 Retained STRING 28 objects of size 14624/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained HASH 25 objects of size 1000/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:22 Retained HASH 23 objects of size 21344/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:177 Retained OBJECT 23 objects of size 920/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/predications.rb:18 Retained STRING 23 objects of size 1080/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:476 Retained OBJECT 22 objects of size 880/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained OBJECT 22 objects of size 1760/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/predicate_builder.rb:68 Retained ARRAY 21 objects of size 840/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained ARRAY 20 objects of size 800/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1367 Retained ARRAY 19 objects of size 760/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:73 Retained HASH 19 objects of size 33440/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:69 Retained DATA 15 objects of size 1200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained ARRAY 14 objects of size 560/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained HASH 14 objects of size 2080/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/set.rb:246 Retained ARRAY 13 objects of size 520/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:10 Retained ARRAY 12 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work.rb:231 Retained STRING 12 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/buffers.rb:23 Retained IMEMO 12 objects of size 10368/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained ARRAY 12 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work/text_page.rb:30 Retained OBJECT 11 objects of size 880/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained OBJECT 11 objects of size 440/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained OBJECT 11 objects of size 440/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/association_scope.rb:55 Retained DATA 11 objects of size 880/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained STRING 11 objects of size 19313644/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:72 Retained IMEMO 10 objects of size 440/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained ARRAY 9 objects of size 360/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:87 Retained HASH 9 objects of size 43536/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:31 Retained IMEMO 9 objects of size 360/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:20 Retained ARRAY 8 objects of size 320/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:10 Retained STRING 8 objects of size 704/312718305 (in bytes) at: :9 Retained DATA 7 objects of size 560/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained ARRAY 7 objects of size 280/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:252 Retained DATA 7 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:238 Retained HASH 7 objects of size 1176/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained HASH 7 objects of size 664/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:29 Retained HASH 7 objects of size 280/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:106 Retained HASH 7 objects of size 792/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:108 Retained STRING 6 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained OBJECT 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:88 Retained HASH 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/utils.rb:58 Retained IMEMO 6 objects of size 432/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained HASH 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:309 Retained IMEMO 6 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained ARRAY 6 objects of size 1200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:8 Retained HASH 6 objects of size 1008/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/hash_with_indifferent_access.rb:265 Retained HASH 6 objects of size 1008/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:126 Retained HASH 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/strategies/base.rb:46 Retained OBJECT 6 objects of size 960/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:381 Retained HASH 6 objects of size 1008/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:443 Retained IMEMO 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:48 Retained IMEMO 6 objects of size 432/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained OBJECT 6 objects of size 960/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:405 Retained OBJECT 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:20 Retained IMEMO 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:52 Retained ARRAY 6 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/merger.rb:123 Retained OBJECT 5 objects of size 800/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:90 Retained OBJECT 5 objects of size 400/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/rails/controller_methods.rb:5 Retained STRING 5 objects of size 400/312718305 (in bytes) at: :30 Retained ARRAY 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:109 Retained OBJECT 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:143 Retained HASH 5 objects of size 840/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/abstract_controller/rendering.rb:64 Retained DATA 5 objects of size 400/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained STRING 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained IMEMO 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:13 Retained STRING 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:20 Retained DATA 5 objects of size 400/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:151 Retained HASH 5 objects of size 840/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:238 Retained STRING 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:117 Retained STRING 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:148 Retained STRING 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:173 Retained ARRAY 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:44 Retained OBJECT 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:51 Retained HASH 5 objects of size 840/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/url_for.rb:41 Retained ARRAY 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:107 Retained HASH 5 objects of size 960/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/has_many_through_association.rb:11 Retained HASH 5 objects of size 840/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:9 Retained ARRAY 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:773 Retained OBJECT 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:253 Retained ARRAY 5 objects of size 17640/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:81 Retained OBJECT 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1289 Retained ARRAY 5 objects of size 200/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained CLASS 4 objects of size 3368/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained ARRAY 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:97 Retained IMEMO 4 objects of size 352/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained IMEMO 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:369 Retained ARRAY 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1524 Retained OBJECT 4 objects of size 1280/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/rendering.rb:110 Retained ARRAY 4 objects of size 13664/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:36 Retained STRING 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/url.rb:227 Retained OBJECT 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:130 Retained HASH 4 objects of size 672/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained DATA 4 objects of size 320/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:20 Retained HASH 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/renderer/renderer.rb:59 Retained ARRAY 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained OBJECT 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:91 Retained OBJECT 4 objects of size 640/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained OBJECT 4 objects of size 640/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/abstract_store.rb:39 Retained STRING 4 objects of size 2208/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/asset_tag_helper.rb:652 Retained HASH 4 objects of size 672/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained HASH 4 objects of size 672/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:308 Retained ARRAY 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained HASH 4 objects of size 672/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:311 Retained HASH 4 objects of size 3712/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:393 Retained STRING 4 objects of size 288/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:446 Retained IMEMO 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:35 Retained CLASS 4 objects of size 2624/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained ARRAY 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:265 Retained OBJECT 4 objects of size 640/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/manager.rb:33 Retained STRING 4 objects of size 224/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:20 Retained HASH 4 objects of size 672/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:30 Retained ARRAY 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:14 Retained STRING 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:114 Retained IMEMO 4 objects of size 288/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained STRING 4 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:116 Retained ICLASS 4 objects of size 640/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained OBJECT 4 objects of size 640/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained HASH 4 objects of size 672/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1283 Retained HASH 3 objects of size 504/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router.rb:43 Retained HASH 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/cache.rb:145 Retained OBJECT 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:259 Retained STRING 3 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/random/formatter.rb:174 Retained HASH 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:258 Retained HASH 3 objects of size 1536/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:56 Retained ARRAY 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained OBJECT 3 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:159 Retained OBJECT 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/tag_helper.rb:424 Retained STRING 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:512 Retained DATA 3 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained STRING 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:113 Retained HASH 3 objects of size 10272/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:109 Retained DATA 3 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained ARRAY 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:338 Retained OBJECT 3 objects of size 480/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained HASH 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/hash/keys.rb:11 Retained STRING 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:111 Retained ARRAY 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:259 Retained OBJECT 3 objects of size 960/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/controllers/works_controller.rb:72 Retained OBJECT 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser/browser.rb:105 Retained OBJECT 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:23 Retained OBJECT 3 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:489 Retained OBJECT 3 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:470 Retained OBJECT 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:215 Retained OBJECT 3 objects of size 960/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:309 Retained ARRAY 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:51 Retained DATA 3 objects of size 192/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:149 Retained DATA 3 objects of size 240/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained STRING 3 objects of size 120/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:65 Retained HASH 3 objects of size 552/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:172 Retained OBJECT 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:226 Retained OBJECT 2 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/filter_parameters.rb:64 Retained STRING 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel.rb:50 Retained IMEMO 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1562 Retained STRING 2 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/visitors.rb:184 Retained OBJECT 2 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:19 Retained OBJECT 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:80 Retained DATA 2 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained ARRAY 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:90 Retained HASH 2 objects of size 336/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained STRING 2 objects of size 372/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:150 Retained HASH 2 objects of size 336/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:99 Retained OBJECT 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/view_component-3.7.0/lib/view_component/base.rb:75 Retained ARRAY 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:113 Retained STRING 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:112 Retained IMEMO 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1373 Retained IMEMO 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1295 Retained ARRAY 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained OBJECT 2 objects of size 320/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained ARRAY 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained HASH 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/query_parser.rb:166 Retained IMEMO 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained ARRAY 2 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:67 Retained OBJECT 2 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:300 Retained IMEMO 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/scrubber.rb:132 Retained OBJECT 2 objects of size 320/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/devise-4.9.3/lib/devise/rails/warden_compat.rb:5 Retained OBJECT 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/base.rb:233 Retained OBJECT 2 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:57 Retained HASH 2 objects of size 336/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:77 Retained IMEMO 2 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rails-html-sanitizer-1.6.0/lib/rails/html/scrubbers.rb:88 Retained STRING 2 objects of size 320/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:265 Retained IMEMO 2 objects of size 88/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:53 Retained STRING 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/content_security_policy.rb:293 Retained STRING 1 objects of size 104/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/etag.rb:37 Retained OBJECT 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:16 Retained DATA 1 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:24 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/objspace.rb:103 Retained HASH 1 objects of size 168/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained OBJECT 1 objects of size 160/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/railties-7.1.2/lib/rails/rack/logger.rb:21 Retained STRING 1 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:240 Retained HASH 1 objects of size 168/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained DATA 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1562 Retained OBJECT 1 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/view_paths.rb:90 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:149 Retained STRING 1 objects of size 80/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:177 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1546 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node_set.rb:352 Retained OBJECT 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:241 Retained ARRAY 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:211 Retained STRING 1 objects of size 56/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:233 Retained STRING 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:159 Retained STRING 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/formats/html.rb:154 Retained OBJECT 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:19 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/sanitizers/description_sanitizer.rb:34 Retained OBJECT 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/presenters/citation_display.rb:23 Retained HASH 1 objects of size 168/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc/attributes.rb:11 Retained HASH 1 objects of size 168/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc/names.rb:116 Retained STRING 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc/attributes.rb:48 Retained HASH 1 objects of size 512/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/formats/html.rb:35 Retained HASH 1 objects of size 168/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained IMEMO 1 objects of size 88/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained HASH 1 objects of size 928/312718305 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/observer.rb:154 Retained ARRAY 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/response.rb:275 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4/document_fragment.rb:48 Retained STRING 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:260 Retained ARRAY 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/runtime.rb:31 Retained STRING 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/presenters/citable_attributes.rb:177 Retained DATA 1 objects of size 396/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:172 Retained DATA 1 objects of size 72/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:177 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rails-html-sanitizer-1.6.0/lib/rails/html/sanitizer.rb:61 Retained DATA 1 objects of size 56/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:43 Retained IMEMO 1 objects of size 48/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:43 Retained DATA 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/scrubber.rb:132 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1156 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rails-html-sanitizer-1.6.0/lib/rails/html/scrubbers.rb:195 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1147 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1522 Retained IMEMO 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1524 Retained DATA 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1530 Retained IMEMO 1 objects of size 112/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained OBJECT 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/request_store-1.5.1/lib/request_store/middleware.rb:21 Retained OBJECT 1 objects of size 40/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/lograge-0.14.0/lib/lograge/rails_ext/rack/logger.rb:21 Retained DATA 1 objects of size 136/312718305 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/csl-2.0.0/lib/csl/treelike.rb:25 Run `$ heapy --help` for more options Also read https://medium.com/klaxit-techblog/tracking-a-ruby-memory-leak-in-2021-9eb56575f731#875b to understand better what you are reading. ```

My app on Rails 7.0

jrochkind-shi scihist_digicoll (derailed_rails70_cc8e84042 ?) $ $ TEST_COUNT=100 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:heap_diff
Booting: production
Database 'scihist_digicoll_development' already exists
Endpoint: "/works/dvoykwl"
Running 100 times
Heap file generated: "tmp/2023-12-04T17:18:52-05:00-heap-0.ndjson"
Running 100 times
Heap file generated: "tmp/2023-12-04T17:18:52-05:00-heap-1.ndjson"
Running 100 times
Heap file generated: "tmp/2023-12-04T17:18:52-05:00-heap-2.ndjson"

diff

``` Diff ==== Retained STRING 3947 objects of size 364560/6785655 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained HASH 1200 objects of size 246112/6785655 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained STRING 600 objects of size 2384118/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:62 Retained OBJECT 509 objects of size 20360/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine.rb:122 Retained OBJECT 460 objects of size 36800/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.0.8/lib/active_model/attribute.rb:9 Retained STRING 436 objects of size 3356105/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.0.8/lib/active_model/type/string.rb:27 Retained HASH 229 objects of size 212512/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/result.rb:161 Retained DATA 140 objects of size 12040/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:62 Retained OBJECT 125 objects of size 10080/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/associations.rb:304 Retained ARRAY 85 objects of size 3520/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.1.0/lib/attr_json/type/array.rb:30 Retained OBJECT 84 objects of size 3360/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.0.8/lib/active_model/dirty.rb:256 Retained HASH 81 objects of size 13608/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.0.8/lib/active_model/attribute_set/builder.rb:28 Retained HASH 81 objects of size 13608/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/associations.rb:322 Retained OBJECT 80 objects of size 6400/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.0.8/lib/active_model/attribute_set/builder.rb:16 Retained HASH 76 objects of size 12768/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.0.8/lib/active_model/attribute_set/builder.rb:22 Retained ARRAY 72 objects of size 3000/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.1.0/lib/attr_json/type/array.rb:26 Retained OBJECT 60 objects of size 9600/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/entity.rb:71 Retained HASH 58 objects of size 2320/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.0.8/lib/active_model/attribute_mutation_tracker.rb:71 Retained STRING 53 objects of size 2120/6785655 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained OBJECT 53 objects of size 2120/6785655 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained OBJECT 52 objects of size 8320/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/persistence.rb:537 Retained DATA 50 objects of size 3600/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:112 Retained HASH 48 objects of size 44544/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:590 Retained HASH 45 objects of size 7560/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/attacher.rb:45 Retained ARRAY 9 objects of size 2480/6785655 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained OBJECT 4 objects of size 640/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/relation/delegation.rb:118 Retained OBJECT 2 objects of size 320/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:381 Retained ICLASS 2 objects of size 320/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained DATA 2 objects of size 160/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.0.8/lib/active_support/ordered_options.rb:83 Retained IMEMO 2 objects of size 80/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:369 Retained IMEMO 2 objects of size 88/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.3/lib/loofah/concerns.rb:43 Retained STRING 2 objects of size 80/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:476 Retained ARRAY 1 objects of size 640/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/racc-1.7.1/lib/racc/parser.rb:265 Retained ARRAY 1 objects of size 160/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.0.8/lib/action_view/path_set.rb:75 Retained OBJECT 1 objects of size 160/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.0.8/lib/action_dispatch/middleware/cookies.rb:654 Retained STRING 1 objects of size 80/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/objspace.rb:103 Retained STRING 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:512 Retained IMEMO 1 objects of size 128/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained STRING 1 objects of size 104/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/etag.rb:37 Retained STRING 1 objects of size 72/6785655 (in bytes) at: :9 Retained HASH 1 objects of size 168/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.1.0/lib/attr_json/model.rb:382 Retained OBJECT 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/arel/predications.rb:18 Retained ARRAY 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/relation/where_clause.rb:15 Retained OBJECT 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/relation/where_clause.rb:15 Retained HASH 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.0.8/lib/active_support/ordered_options.rb:92 Retained IMEMO 1 objects of size 80/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.0.8/lib/active_support/ordered_options.rb:83 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4/document_fragment.rb:51 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node_set.rb:234 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1156 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:273 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1530 Retained DATA 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1530 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1378 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1546 Retained STRING 1 objects of size 40/6785655 (in bytes) at: parser.y:285 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/relation.rb:909 Retained OBJECT 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.0.8/lib/active_record/associations/association_scope.rb:55 Retained IMEMO 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:153 Retained OBJECT 1 objects of size 40/6785655 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/etag.rb:34 Run `$ heapy --help` for more options Also read https://medium.com/klaxit-techblog/tracking-a-ruby-memory-leak-in-2021-9eb56575f731#875b to understand better what you are reading. ```

[jrochkind]

@bensheldon ah and actually that particular WorkFileListShowComponent is actually fairly rarely used though, and is not rendered in the path I have been testing. That may be partially how we didn't notice it, it's used mostly in some edge cases. AND in those edge cases does NOT have 100-800 constituent members, but only 10-30. So not as big a deal and probably not relevant, but still will fix it it should be memoized!

[bensheldon]

So not as big a deal and probably not relevant, but still will fix it it should be memoized!

Agreed! I didn't want to distract, but saw it.

Thank you for generating that heap dump! That's really helpful.

It looks to me like there is maybe some really-big JSON files in the database that are being parsed? Does that ring a bell?

The lines seem to correspond to:

• https://github.com/ruby/ruby/blob/5a2d70ef73d4d17f5999f5087499a5eda1126567/ext/json/lib/json/common.rb#L216
• https://github.com/shrinerb/shrine/blob/bf43c6de28f4b6d58fa8c2d4ec22747db4e44e60/lib/shrine.rb#L122
• The Active Record issues look a lot like the problems described here. This is (1) covers that maybe a bunch of (really big?) SQL queries are being stored in the Statement Cache and (2) doing includes(:relation).find which loads up all the records I think.
• I don't know for sure whether these are all related to each other, or separate problems, but I could hypothesize that a big chunk of JSON is being interpolated (not parameterized) into a sql query that's getting stuck in the statement cache. You might add prepared_statements: false to your database.yml and try re-running the heap dump (this wouldn't be a recommended fix, but rather to quickly diagnose that this is the source of some of the retained objects and then figure out what big chunk of data is getting interpolated into the SQL)

[jrochkind]

Thank you, this is super helpful, I really appreciate it.

OK, there is definitely some big json in the database, yeah.

These come from two places:

• We use shrine, which stores file attachment pointers (including variants/derivatives) and metadata in json in the db, and it can get pretty large
• Possibly even bigger json, we use the attr_json gem that I wrote to embed lots of complex attributes into a jsonb column. With lots of stuff. We have made the design choice to store a lot of stuff in json in the db... but it's always worked out well until now.

So yeah, there is some big json -- which will be in insert/update queries and in column returned by select, there's no reason it should end up in anything interpolated into an SQL query other than insert or update, not sure if Rails uses prepared staetment cache for those? Either way it's actually worked out quite well for us for some years now, the app may have began as Rails 5.0 even.

So... why would upgrade to Rails 7.1 suddenly start making this a bigger problem? I know the problem is almost never in Rails it's in your code -- and possibly attr_json is to blame, or more likely doing something that wasn't a perf problem before Rails 7.1 but is now, which would be really hard to solve if it's not considered a bug...

But part of me is still wondering if something in Rails 7.1 is holding on to things in a statement cache that it shouldn't be or longer than it should be, or at any rate differently than Rails 7.0 regardless of "should". I definitely agree it looks like something related to our json in ActiveRecord.

If I put prepared_statements: false in the database.yml, do we expect the retained objects to be better or or worse?

Not sure how to verfy prepared_statements:false is really taking effect... if I'm setting my database.yml from a url: ENV['DATABASE_URL'], can I still add prepared_statements:false as a separate line, and have it merged in? Not sure, but that's what I'm trying....

update, no, prob need to merge it into my DATABASE_URL myself, so ignore below

Not sure how to understand this compared to above or if it's different.... looks pretty similar?

Rails 7.1, prepared_statements:false (I think)

``` jrochkind-shi scihist_digicoll (derailed_rails71_3afd9c514 !?) $ TEST_COUNT=100 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:heap_diff Booting: production Database 'scihist_digicoll_development' already exists Endpoint: "/works/dvoykwl" Running 100 times Heap file generated: "tmp/2023-12-04T18:01:31-05:00-heap-0.ndjson" Running 100 times Heap file generated: "tmp/2023-12-04T18:01:31-05:00-heap-1.ndjson" Running 100 times Heap file generated: "tmp/2023-12-04T18:01:31-05:00-heap-2.ndjson" Diff ==== Retained STRING 129423 objects of size 11852480/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained HASH 52333 objects of size 10629248/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained STRING 26571 objects of size 208346301/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained OBJECT 22287 objects of size 891480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine.rb:122 Retained STRING 18239 objects of size 140983518/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/type/string.rb:35 Retained OBJECT 15387 objects of size 1230960/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute.rb:9 Retained DATA 11666 objects of size 1003276/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained HASH 4496 objects of size 4172288/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:169 Retained OBJECT 3867 objects of size 618720/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:320 Retained ARRAY 3177 objects of size 127800/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:30 Retained ARRAY 3089 objects of size 124280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:26 Retained HASH 3068 objects of size 515040/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:339 Retained OBJECT 3050 objects of size 122000/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/dirty.rb:377 Retained HASH 3031 objects of size 509208/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:28 Retained HASH 3022 objects of size 538856/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:22 Retained HASH 3000 objects of size 120000/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_mutation_tracker.rb:71 Retained OBJECT 2532 objects of size 405120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/entity.rb:71 Retained OBJECT 2502 objects of size 200160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:16 Retained OBJECT 2492 objects of size 797440/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/persistence.rb:640 Retained HASH 1995 objects of size 335160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/attacher.rb:45 Retained DATA 1982 objects of size 142704/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:112 Retained HASH 1956 objects of size 1815168/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:590 Retained OBJECT 1750 objects of size 70000/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained STRING 1728 objects of size 69120/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained ARRAY 423 objects of size 112072/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained OBJECT 84 objects of size 3360/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:151 Retained HASH 84 objects of size 14112/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:382 Retained HASH 81 objects of size 13608/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:300 Retained HASH 67 objects of size 8696/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:28 Retained STRING 43 objects of size 16232/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained OBJECT 41 objects of size 6560/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/delegation.rb:132 Retained OBJECT 34 objects of size 2720/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:59 Retained OBJECT 33 objects of size 2640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/predicate_builder.rb:68 Retained ARRAY 32 objects of size 1280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained OBJECT 31 objects of size 1240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained OBJECT 30 objects of size 1200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/predications.rb:18 Retained STRUCT 30 objects of size 1200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/table.rb:89 Retained ARRAY 29 objects of size 1160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:73 Retained STRING 27 objects of size 1480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:476 Retained HASH 27 objects of size 1080/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:22 Retained ARRAY 25 objects of size 1000/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1367 Retained HASH 25 objects of size 23200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:177 Retained HASH 25 objects of size 44000/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:69 Retained ARRAY 24 objects of size 960/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:10 Retained DATA 23 objects of size 1840/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained STRING 15 objects of size 25824678/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:72 Retained OBJECT 15 objects of size 1200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained DATA 15 objects of size 1032/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:238 Retained ARRAY 15 objects of size 600/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:10 Retained HASH 15 objects of size 2272/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/set.rb:246 Retained STRING 14 objects of size 560/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/buffers.rb:23 Retained OBJECT 13 objects of size 520/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:88 Retained ARRAY 12 objects of size 480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:87 Retained IMEMO 12 objects of size 480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:13 Retained ARRAY 12 objects of size 480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained DATA 11 objects of size 880/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained HASH 11 objects of size 29224/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:31 Retained ARRAY 10 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work.rb:231 Retained ARRAY 10 objects of size 35280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:81 Retained OBJECT 10 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained ARRAY 10 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work/text_page.rb:30 Retained HASH 10 objects of size 1680/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained HASH 10 objects of size 1168/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:108 Retained IMEMO 9 objects of size 648/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained DATA 9 objects of size 720/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained IMEMO 9 objects of size 392/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained HASH 9 objects of size 360/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/strategies/base.rb:46 Retained HASH 9 objects of size 1512/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/hash_with_indifferent_access.rb:265 Retained STRING 8 objects of size 640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained HASH 8 objects of size 1472/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:172 Retained STRING 8 objects of size 768/411878691 (in bytes) at: :9 Retained HASH 8 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/utils.rb:58 Retained HASH 8 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:106 Retained OBJECT 8 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:226 Retained IMEMO 8 objects of size 640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained ARRAY 7 objects of size 280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained HASH 7 objects of size 1176/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router.rb:43 Retained OBJECT 7 objects of size 1120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:381 Retained HASH 7 objects of size 1176/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained ARRAY 7 objects of size 280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:211 Retained HASH 7 objects of size 1176/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained ARRAY 7 objects of size 280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:338 Retained HASH 7 objects of size 280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/hash/keys.rb:11 Retained ARRAY 6 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:773 Retained OBJECT 6 objects of size 480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:300 Retained ARRAY 6 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:252 Retained OBJECT 6 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/tag_helper.rb:424 Retained HASH 6 objects of size 624/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:29 Retained STRING 6 objects of size 3312/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/asset_tag_helper.rb:652 Retained HASH 6 objects of size 1008/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:443 Retained IMEMO 6 objects of size 3536/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained OBJECT 6 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:19 Retained ARRAY 6 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:97 Retained ARRAY 6 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:259 Retained OBJECT 5 objects of size 1600/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/controllers/works_controller.rb:72 Retained OBJECT 5 objects of size 1600/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/rendering.rb:110 Retained OBJECT 5 objects of size 1600/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:309 Retained OBJECT 5 objects of size 800/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/devise-4.9.3/lib/devise/rails/warden_compat.rb:5 Retained OBJECT 5 objects of size 800/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained OBJECT 5 objects of size 800/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/abstract_store.rb:39 Retained OBJECT 5 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:19 Retained STRING 5 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/visitors.rb:184 Retained ARRAY 5 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:67 Retained IMEMO 5 objects of size 360/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained STRING 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:113 Retained OBJECT 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:215 Retained HASH 5 objects of size 840/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:238 Retained ARRAY 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained STRING 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:117 Retained DATA 5 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:151 Retained DATA 5 objects of size 400/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained STRING 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:65 Retained HASH 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/cache.rb:145 Retained ARRAY 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:51 Retained HASH 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:309 Retained ARRAY 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained ARRAY 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained OBJECT 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:20 Retained STRING 5 objects of size 360/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:446 Retained OBJECT 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/base.rb:233 Retained STRING 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:116 Retained IMEMO 5 objects of size 440/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained STRING 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:20 Retained STRING 5 objects of size 280/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:20 Retained OBJECT 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:130 Retained OBJECT 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:56 Retained HASH 5 objects of size 840/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:311 Retained OBJECT 5 objects of size 200/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:259 Retained STRING 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:111 Retained DATA 4 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:24 Retained ARRAY 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:90 Retained ARRAY 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:265 Retained OBJECT 4 objects of size 640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:159 Retained OBJECT 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1289 Retained IMEMO 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:20 Retained ARRAY 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:107 Retained OBJECT 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/association_scope.rb:55 Retained HASH 4 objects of size 672/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:99 Retained ARRAY 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:109 Retained HASH 4 objects of size 672/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:308 Retained HASH 4 objects of size 672/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained IMEMO 4 objects of size 288/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained OBJECT 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:253 Retained HASH 4 objects of size 672/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:126 Retained DATA 4 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained OBJECT 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:80 Retained HASH 4 objects of size 13696/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:109 Retained OBJECT 4 objects of size 640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:405 Retained OBJECT 4 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/view_paths.rb:90 Retained OBJECT 4 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/filter_parameters.rb:64 Retained OBJECT 4 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:470 Retained HASH 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/query_parser.rb:166 Retained OBJECT 4 objects of size 640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/manager.rb:33 Retained STRING 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:148 Retained OBJECT 4 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/rails/controller_methods.rb:5 Retained STRING 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained HASH 4 objects of size 672/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:9 Retained DATA 4 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:20 Retained HASH 4 objects of size 768/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/has_many_through_association.rb:11 Retained STRING 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:114 Retained STRING 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:512 Retained CLASS 4 objects of size 2624/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained STRING 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:173 Retained ARRAY 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained DATA 4 objects of size 256/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:149 Retained OBJECT 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser/browser.rb:105 Retained OBJECT 4 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:241 Retained OBJECT 4 objects of size 640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained HASH 3 objects of size 504/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:30 Retained ARRAY 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:14 Retained DATA 3 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained OBJECT 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/view_component-3.7.0/lib/view_component/base.rb:75 Retained STRING 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:159 Retained HASH 3 objects of size 504/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:77 Retained OBJECT 3 objects of size 480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained STRING 3 objects of size 168/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:233 Retained OBJECT 3 objects of size 480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:90 Retained ARRAY 3 objects of size 600/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:8 Retained ARRAY 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:44 Retained HASH 3 objects of size 504/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained HASH 3 objects of size 1536/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:56 Retained OBJECT 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:23 Retained STRING 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel.rb:50 Retained DATA 3 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained OBJECT 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:143 Retained STRING 3 objects of size 1920/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:337 Retained STRING 3 objects of size 424/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:265 Retained STRING 3 objects of size 240/411878691 (in bytes) at: :30 Retained OBJECT 3 objects of size 240/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:489 Retained STRING 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/url.rb:227 Retained IMEMO 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:35 Retained ARRAY 3 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained OBJECT 3 objects of size 480/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained ARRAY 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:113 Retained OBJECT 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:91 Retained IMEMO 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:52 Retained HASH 2 objects of size 336/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/abstract_controller/rendering.rb:64 Retained HASH 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:258 Retained ARRAY 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/merger.rb:123 Retained STRING 2 objects of size 320/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/random/formatter.rb:174 Retained IMEMO 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:48 Retained OBJECT 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:51 Retained HASH 2 objects of size 336/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained HASH 2 objects of size 336/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1283 Retained STRING 2 objects of size 120/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained HASH 2 objects of size 336/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/url_for.rb:41 Retained HASH 2 objects of size 1856/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:393 Retained ARRAY 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1524 Retained OBJECT 2 objects of size 160/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:57 Retained HASH 2 objects of size 384/411878691 (in bytes) at: :18 Retained HASH 2 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/renderer/renderer.rb:59 Retained HASH 1 objects of size 168/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained DATA 1 objects of size 40/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html5/document_fragment.rb:48 Retained IMEMO 1 objects of size 40/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/objspace.rb:103 Retained STRING 1 objects of size 40/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:112 Retained DATA 1 objects of size 72/411878691 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/objspace.rb:102 Retained FILE 1 objects of size 8440/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained IMEMO 1 objects of size 40/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/choose.rb:14 Retained ARRAY 1 objects of size 40/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:326 Retained STRING 1 objects of size 70/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:150 Retained ARRAY 1 objects of size 3416/411878691 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:36 Retained IMEMO 1 objects of size 48/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:53 Retained STRING 1 objects of size 640/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:256 Retained IMEMO 1 objects of size 40/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/query_cache.rb:111 Retained DATA 1 objects of size 80/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained DATA 1 objects of size 192/411878691 (in bytes) at: parser.y:187 Retained IMEMO 1 objects of size 40/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1546 Retained IMEMO 1 objects of size 88/411878691 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Run `$ heapy --help` for more options Also read https://medium.com/klaxit-techblog/tracking-a-ruby-memory-leak-in-2021-9eb56575f731#875b to understand better what you are reading. jrochkind-shi scihist_digicoll (derailed_rails71_3afd9c514 !?) $ ```

[jrochkind]

OK, I'm also realizing that before the Rails 7.1 update, our config.load_defaults was still on 6.1.... it's now on 7.0 with additional defaults opted in in new_framework_defaults_7.1. So it's possible something in 7.0 framework defaults could be relevant too, will try to take a look there... or try profiling under 6.1 defaults?

update: okay, ran perf:objects with Rails framework defaults 6.1, "retained memory by gem" report largely the same, so okay prob not that.

[bensheldon]

If I put prepared_statements: false in the database.yml, do we expect the retained objects to be better or or worse?

Better, if the hypothesis is that there are really big chunks of SQL or something else in the statement catch

if I'm setting my database.yml from a url: ENV['DATABASE_URL'], can I still add prepared_statements:false as a separate line, and have it merged in?

I think you want to add it as a query param on the url e.g. postgresql://....?prepared_statements=false. I think you may also be able to verify with ActiveRecord.disable_prepared_statements == true

[jrochkind]

OK modified DATABASE_URL but ActiveRecord.disable_prepared_statements is still false... so currently stuck on shaving the yak of figuring out how to actually set this, looking into it.

DATABASE_URL is "postgresql:///scihist_digicoll_development?port=5432&prepared_statements=false"

[jrochkind]

OK I dunno, even though ActiveRecord.disable_prepared_statements isn't reflecting it, I think it is disabled?

DATABASE_URL is postgresql:///scihist_digicoll_development?port=5432&prepared_statements=false&statement_limit=1

in console, ActiveRecord::Base.configurations

includes:

   #<ActiveRecord::DatabaseConfigurations::UrlConfig:0x0000000130012da0
    @configuration_hash=
     {:adapter=>"postgresql", :encoding=>"unicode", :pool=>5, :prepared_statements=>"false", :port=>"5432", :database=>"scihist_digicoll_development", :statement_limit=>"1"},
    @env_name="production",
    @name="primary",
    @url="postgresql:///scihist_digicoll_development?port=5432&prepared_statements=false&statement_limit=1">]>

Even though ActiveRecord.disable_prepared_statements is still false.

Running heap_diff again.... I am getting confused about whether it looks better or not, or as good as Rails 7.0 version I think not?

I agree something AR must be hanging on to these values... note discussion about inserts/updates and expected large interpolation above... of course no inserts/updates are actually happening in this profile run!

Rails 7.1 heap_diff

``` jrochkind-shi scihist_digicoll (derailed_rails71_3afd9c514 !?) $ TEST_COUNT=100 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:heap_diff Booting: production DEPRECATION WARNING: Support for `config.active_support.cache_format_version = 6.1` has been deprecated and will be removed in Rails 7.2. Check the Rails upgrade guide at https://guides.rubyonrails.org/upgrading_ruby_on_rails.html#new-activesupport-cache-serialization-format for more information on how to upgrade. (called from load at /Users/jrochkind/.gem/ruby/3.2.2/bin/derailed:25) Database 'scihist_digicoll_development' already exists Endpoint: "/works/dvoykwl" Running 100 times Heap file generated: "tmp/2023-12-04T18:32:12-05:00-heap-0.ndjson" Running 100 times Heap file generated: "tmp/2023-12-04T18:32:12-05:00-heap-1.ndjson" Running 100 times Heap file generated: "tmp/2023-12-04T18:32:12-05:00-heap-2.ndjson" Diff ==== Retained STRING 69003 objects of size 6303200/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained HASH 27733 objects of size 5620672/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained STRING 20092 objects of size 139153615/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained OBJECT 13435 objects of size 1074800/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute.rb:9 Retained STRING 11345 objects of size 89563601/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/type/string.rb:35 Retained OBJECT 11325 objects of size 453000/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine.rb:122 Retained DATA 7803 objects of size 671058/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained HASH 3727 objects of size 3458656/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:169 Retained OBJECT 2874 objects of size 459840/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:320 Retained OBJECT 2182 objects of size 174560/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:16 Retained ARRAY 1982 objects of size 79640/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:30 Retained ARRAY 1949 objects of size 78320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:26 Retained HASH 1916 objects of size 76640/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_mutation_tracker.rb:71 Retained OBJECT 1882 objects of size 75280/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/dirty.rb:377 Retained HASH 1879 objects of size 315288/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:339 Retained HASH 1857 objects of size 311976/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:28 Retained HASH 1825 objects of size 331680/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:22 Retained OBJECT 1357 objects of size 217120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/entity.rb:71 Retained OBJECT 1268 objects of size 405760/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/persistence.rb:640 Retained OBJECT 1029 objects of size 41160/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained HASH 1017 objects of size 170856/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/attacher.rb:45 Retained DATA 1011 objects of size 72792/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:112 Retained STRING 1005 objects of size 40200/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained HASH 991 objects of size 919648/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:590 Retained ARRAY 270 objects of size 72632/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained OBJECT 49 objects of size 1960/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:151 Retained HASH 41 objects of size 6888/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:382 Retained HASH 40 objects of size 6720/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:300 Retained OBJECT 39 objects of size 3120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:59 Retained STRING 37 objects of size 2560/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:476 Retained OBJECT 31 objects of size 4960/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/delegation.rb:132 Retained OBJECT 24 objects of size 1920/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/predicate_builder.rb:68 Retained HASH 22 objects of size 20416/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:177 Retained STRUCT 19 objects of size 760/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/table.rb:89 Retained OBJECT 19 objects of size 1520/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained HASH 17 objects of size 2472/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:28 Retained STRING 16 objects of size 7552/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained HASH 15 objects of size 26400/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:69 Retained OBJECT 13 objects of size 520/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/predications.rb:18 Retained ARRAY 13 objects of size 520/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1367 Retained OBJECT 12 objects of size 480/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained ARRAY 12 objects of size 480/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:73 Retained ARRAY 12 objects of size 480/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained ARRAY 10 objects of size 400/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained ARRAY 9 objects of size 31752/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:81 Retained DATA 9 objects of size 720/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained HASH 8 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:22 Retained HASH 8 objects of size 1080/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/set.rb:246 Retained OBJECT 8 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained STRING 7 objects of size 560/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained HASH 7 objects of size 920/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:29 Retained STRING 7 objects of size 13741477/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:72 Retained ARRAY 6 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:10 Retained IMEMO 6 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:20 Retained OBJECT 6 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:88 Retained ARRAY 6 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:87 Retained OBJECT 6 objects of size 480/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:19 Retained OBJECT 6 objects of size 480/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/filter_parameters.rb:64 Retained DATA 6 objects of size 408/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:238 Retained HASH 6 objects of size 1008/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained OBJECT 6 objects of size 480/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/view_paths.rb:90 Retained HASH 5 objects of size 840/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained STRING 5 objects of size 400/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/visitors.rb:184 Retained IMEMO 5 objects of size 216/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained IMEMO 5 objects of size 200/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:13 Retained ARRAY 5 objects of size 200/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work.rb:231 Retained ARRAY 5 objects of size 200/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:252 Retained STRING 5 objects of size 200/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained HASH 5 objects of size 17120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:109 Retained STRING 5 objects of size 200/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:512 Retained HASH 5 objects of size 840/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:99 Retained IMEMO 5 objects of size 400/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained ARRAY 5 objects of size 400/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:67 Retained DATA 5 objects of size 400/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:151 Retained HASH 5 objects of size 456/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:108 Retained STRING 5 objects of size 200/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:20 Retained OBJECT 5 objects of size 800/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:381 Retained HASH 4 objects of size 672/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained OBJECT 4 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/rails/controller_methods.rb:5 Retained OBJECT 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/base.rb:233 Retained HASH 4 objects of size 672/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router.rb:43 Retained HASH 4 objects of size 672/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/abstract_controller/rendering.rb:64 Retained HASH 4 objects of size 672/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/url_for.rb:41 Retained OBJECT 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/association_scope.rb:55 Retained ARRAY 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:10 Retained STRING 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:114 Retained HASH 4 objects of size 14552/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:31 Retained DATA 4 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained ARRAY 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained STRING 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:65 Retained HASH 4 objects of size 672/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:443 Retained STRING 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:116 Retained OBJECT 4 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:300 Retained IMEMO 4 objects of size 1808/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained ARRAY 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:259 Retained STRING 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:159 Retained ARRAY 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:14 Retained OBJECT 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:130 Retained ICLASS 4 objects of size 640/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained HASH 4 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/utils.rb:58 Retained DATA 4 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:91 Retained HASH 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:309 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:109 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:97 Retained OBJECT 3 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:470 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:241 Retained HASH 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:258 Retained HASH 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/strategies/base.rb:46 Retained HASH 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/renderer/renderer.rb:59 Retained STRING 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:117 Retained DATA 3 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained STRING 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:148 Retained STRING 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:173 Retained STRING 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:111 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:773 Retained STRING 3 objects of size 280/264097875 (in bytes) at: :9 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained STRING 3 objects of size 1656/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/asset_tag_helper.rb:652 Retained HASH 3 objects of size 504/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1283 Retained IMEMO 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:35 Retained OBJECT 3 objects of size 480/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:44 Retained CLASS 3 objects of size 3400/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained OBJECT 3 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:489 Retained HASH 3 objects of size 552/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:172 Retained HASH 3 objects of size 504/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:308 Retained DATA 3 objects of size 240/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:24 Retained IMEMO 3 objects of size 216/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained STRING 3 objects of size 216/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:446 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:143 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser/browser.rb:105 Retained HASH 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/hash/keys.rb:11 Retained STRING 3 objects of size 168/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:233 Retained STRING 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/url.rb:227 Retained HASH 3 objects of size 504/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:253 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:51 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:259 Retained HASH 3 objects of size 504/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:20 Retained ARRAY 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:211 Retained OBJECT 3 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:215 Retained OBJECT 2 objects of size 640/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/controllers/works_controller.rb:72 Retained OBJECT 2 objects of size 640/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/rendering.rb:110 Retained OBJECT 2 objects of size 640/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:309 Retained STRING 2 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:265 Retained OBJECT 2 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:405 Retained OBJECT 2 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/abstract_store.rb:39 Retained OBJECT 2 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/manager.rb:33 Retained OBJECT 2 objects of size 320/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained STRING 2 objects of size 160/264097875 (in bytes) at: :30 Retained OBJECT 2 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:57 Retained STRING 2 objects of size 120/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained HASH 2 objects of size 336/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained HASH 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/cache.rb:145 Retained DATA 2 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained DATA 2 objects of size 128/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:149 Retained STRING 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:113 Retained IMEMO 2 objects of size 144/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained ARRAY 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained HASH 2 objects of size 1024/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:56 Retained REGEXP 2 objects of size 1608/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:121 Retained HASH 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/query_parser.rb:166 Retained ARRAY 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:90 Retained IMEMO 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:52 Retained HASH 2 objects of size 336/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:30 Retained IMEMO 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:48 Retained OBJECT 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/view_component-3.7.0/lib/view_component/base.rb:75 Retained HASH 2 objects of size 336/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:311 Retained STRING 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:112 Retained ARRAY 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:265 Retained STRING 2 objects of size 112/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:20 Retained REGEXP 2 objects of size 1100/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:122 Retained ARRAY 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained ARRAY 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work/text_page.rb:30 Retained DATA 2 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained OBJECT 2 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:80 Retained DATA 2 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained STRING 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel.rb:50 Retained OBJECT 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:226 Retained OBJECT 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:23 Retained HASH 1 objects of size 168/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:77 Retained HASH 1 objects of size 928/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:393 Retained HASH 1 objects of size 168/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:126 Retained CLASS 1 objects of size 656/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained DATA 1 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:20 Retained HASH 1 objects of size 168/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/hash_with_indifferent_access.rb:265 Retained ARRAY 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:51 Retained HASH 1 objects of size 168/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:238 Retained OBJECT 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1289 Retained HASH 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:106 Retained IMEMO 1 objects of size 72/264097875 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained HASH 1 objects of size 168/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:9 Retained ARRAY 1 objects of size 200/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:8 Retained ARRAY 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:122 Retained STRING 1 objects of size 640/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:337 Retained OBJECT 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/tag_helper.rb:424 Retained OBJECT 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:19 Retained IMEMO 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:50 Retained STRING 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/formats/html.rb:154 Retained STRING 1 objects of size 72/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:259 Retained IMEMO 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/objspace.rb:103 Retained STRING 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/response.rb:292 Retained DATA 1 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/response.rb:46 Retained HASH 1 objects of size 168/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:233 Retained OBJECT 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/executor.rb:15 Retained IMEMO 1 objects of size 128/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained DATA 1 objects of size 80/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained DATA 1 objects of size 72/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/objspace.rb:102 Retained FILE 1 objects of size 8440/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained STRING 1 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/random/formatter.rb:174 Retained OBJECT 1 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained OBJECT 1 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:159 Retained OBJECT 1 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/devise-4.9.3/lib/devise/rails/warden_compat.rb:5 Retained OBJECT 1 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained OBJECT 1 objects of size 160/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:90 Retained STRING 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/buffers.rb:23 Retained HASH 1 objects of size 192/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/has_many_through_association.rb:11 Retained ARRAY 1 objects of size 40/264097875 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:113 Run `$ heapy --help` for more options Also read https://medium.com/klaxit-techblog/tracking-a-ruby-memory-leak-in-2021-9eb56575f731#875b to understand better what you are reading. jrochkind-shi scihist_digicoll (derailed_rails71_3afd9c514 !?) $ ```

I will leave for the day shortly, I keep meaning to, but then you leave me a tantalizing next step. :)

[jrochkind]

I wonder if it could be the query cache? Apparently Rails 7.1 does make changes to query cache, making it LRU... but query cache should be cleared at end of every request and not result in retained memory.... looking for how to disable query cache to test that, although apparently there is no simple way....

What other places could AR be caching returned json values?

[jrochkind]

OK, Rails 7.1 lets you limit or disable query cache from connection properties too.

Trying to disable both query cache and prepared statement cache with: postgresql:///scihist_digicoll_development?port=5432&prepared_statements=false&statement_limit=1&query_cache=false

Does not seem to me to have much effect on retained objects.

I am not sure where to go next, how to figure out what is hanging on to these objects or why it differs from Rails 7.0 (hypothesis remains, they are ActiveRecord objects with a lot of parsed JSON).

From TEST_COUNT=10 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:objects, retained objects seem about the same order of magnitude as previous (they always differ from run to run, which I don't understand the significance of).

retained memory by gem
-----------------------------------
 122128976  activemodel-7.1.2
 114586820  activerecord-7.1.2
  83349519  activesupport-7.1.2
  21575160  json
   3779560  shrine-3.5.0
   1658925  net-protocol-0.2.2
    223200  attr_json-2.2.0
    191314  scihist_digicoll/app
    136000  faster_s3_url-1.0.0

retained objects by gem
-----------------------------------
    171901  json
     38327  activemodel-7.1.2
     34489  activerecord-7.1.2
     32585  shrine-3.5.0
     26434  activesupport-7.1.2
      4880  attr_json-2.2.0
      4277  scihist_digicoll/app
       850  faster_s3_url-1.0.0
       273  actionpack-7.1.2

And TEST_COUNT=100 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:heap_diff

perf:heap_diff

``` jrochkind-shi scihist_digicoll (derailed_rails71_3afd9c514 !?) $ TEST_COUNT=100 PATH_TO_HIT=/works/dvoykwl bundle exec derailed exec perf:heap_diff Booting: production DEPRECATION WARNING: Support for `config.active_support.cache_format_version = 6.1` has been deprecated and will be removed in Rails 7.2. Check the Rails upgrade guide at https://guides.rubyonrails.org/upgrading_ruby_on_rails.html#new-activesupport-cache-serialization-format for more information on how to upgrade. (called from load at /Users/jrochkind/.gem/ruby/3.2.2/bin/derailed:25) Database 'scihist_digicoll_development' already exists Endpoint: "/works/dvoykwl" Running 100 times Heap file generated: "tmp/2023-12-05T11:22:45-05:00-heap-0.ndjson" Running 100 times Heap file generated: "tmp/2023-12-05T11:22:45-05:00-heap-1.ndjson" Running 100 times Heap file generated: "tmp/2023-12-05T11:22:45-05:00-heap-2.ndjson" Diff ==== Retained STRING 133006 objects of size 11788280/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained HASH 61724 objects of size 12434576/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained OBJECT 28167 objects of size 1126680/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine.rb:122 Retained OBJECT 15626 objects of size 1250080/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute.rb:9 Retained STRING 14776 objects of size 83421500/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained STRING 11039 objects of size 67942684/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/type/string.rb:35 Retained OBJECT 6415 objects of size 1026400/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:320 Retained DATA 4964 objects of size 426904/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained OBJECT 2511 objects of size 200880/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:16 Retained DATA 2489 objects of size 179208/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:112 Retained HASH 2477 objects of size 2298656/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/derivatives.rb:590 Retained HASH 2420 objects of size 406560/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/attacher.rb:45 Retained OBJECT 2348 objects of size 375680/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/shrine-3.5.0/lib/shrine/plugins/entity.rb:71 Retained OBJECT 2222 objects of size 711040/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/persistence.rb:640 Retained ARRAY 1632 objects of size 66600/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:26 Retained ARRAY 1632 objects of size 66720/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:30 Retained STRING 1614 objects of size 64560/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained OBJECT 1614 objects of size 64560/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:37 Retained HASH 1460 objects of size 245280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:28 Retained OBJECT 1444 objects of size 57760/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/dirty.rb:377 Retained HASH 1409 objects of size 251912/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:22 Retained HASH 1398 objects of size 234224/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations.rb:339 Retained HASH 1394 objects of size 55760/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activemodel-7.1.2/lib/active_model/attribute_mutation_tracker.rb:71 Retained HASH 1106 objects of size 1026368/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:169 Retained ARRAY 226 objects of size 60840/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained OBJECT 137 objects of size 5480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:151 Retained HASH 123 objects of size 20664/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:300 Retained HASH 113 objects of size 18984/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/attr_json-2.2.0/lib/attr_json/model.rb:382 Retained OBJECT 71 objects of size 11360/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/delegation.rb:132 Retained HASH 64 objects of size 7936/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:28 Retained HASH 49 objects of size 1960/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:22 Retained STRING 44 objects of size 2600/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:476 Retained ARRAY 44 objects of size 1760/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:73 Retained STRUCT 43 objects of size 1720/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/table.rb:89 Retained OBJECT 40 objects of size 1600/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel/predications.rb:18 Retained OBJECT 38 objects of size 1520/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained ARRAY 35 objects of size 1400/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:10 Retained STRING 35 objects of size 15784/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained ARRAY 34 objects of size 1360/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1367 Retained OBJECT 34 objects of size 2720/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/predicate_builder.rb:68 Retained OBJECT 33 objects of size 2640/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:59 Retained ARRAY 32 objects of size 1280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained ARRAY 26 objects of size 1040/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:10 Retained DATA 25 objects of size 2000/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained HASH 23 objects of size 40480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:69 Retained HASH 20 objects of size 115936/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:31 Retained HASH 20 objects of size 18560/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/result.rb:177 Retained HASH 20 objects of size 3384/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/set.rb:246 Retained OBJECT 17 objects of size 680/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:88 Retained ARRAY 17 objects of size 680/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:87 Retained STRING 16 objects of size 1280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained HASH 16 objects of size 2688/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/hash_with_indifferent_access.rb:265 Retained DATA 16 objects of size 1280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained IMEMO 15 objects of size 1080/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained ARRAY 15 objects of size 600/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained OBJECT 15 objects of size 600/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained IMEMO 14 objects of size 600/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained OBJECT 14 objects of size 1120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained IMEMO 14 objects of size 1120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained IMEMO 14 objects of size 560/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:13 Retained IMEMO 14 objects of size 10448/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained OBJECT 14 objects of size 2240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:381 Retained HASH 14 objects of size 560/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/utils.rb:58 Retained HASH 13 objects of size 520/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/strategies/base.rb:46 Retained STRING 13 objects of size 520/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:512 Retained IMEMO 12 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:20 Retained HASH 12 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:106 Retained DATA 12 objects of size 960/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained STRING 12 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/buffers.rb:23 Retained HASH 11 objects of size 1080/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:29 Retained ARRAY 11 objects of size 440/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work.rb:231 Retained HASH 11 objects of size 1848/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1283 Retained DATA 11 objects of size 768/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:238 Retained STRING 10 objects of size 816/205345354 (in bytes) at: :9 Retained ICLASS 10 objects of size 1600/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained DATA 10 objects of size 800/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:20 Retained ARRAY 10 objects of size 400/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/models/work/text_page.rb:30 Retained STRING 10 objects of size 19094186/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:72 Retained HASH 10 objects of size 1680/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained HASH 10 objects of size 400/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:309 Retained CLASS 10 objects of size 10640/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:371 Retained OBJECT 9 objects of size 360/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:56 Retained STRING 9 objects of size 360/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:116 Retained DATA 9 objects of size 720/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained OBJECT 9 objects of size 360/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1289 Retained DATA 9 objects of size 720/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained STRING 9 objects of size 360/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained HASH 9 objects of size 872/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:108 Retained CLASS 9 objects of size 5904/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained STRING 8 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:114 Retained OBJECT 8 objects of size 640/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:57 Retained OBJECT 8 objects of size 640/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:489 Retained IMEMO 8 objects of size 576/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained OBJECT 8 objects of size 2560/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/rendering.rb:110 Retained OBJECT 8 objects of size 2560/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/controllers/works_controller.rb:72 Retained STRING 8 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/url.rb:227 Retained OBJECT 8 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:19 Retained ARRAY 8 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:122 Retained HASH 8 objects of size 1344/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained ARRAY 8 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained OBJECT 8 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/view_component-3.7.0/lib/view_component/base.rb:75 Retained OBJECT 8 objects of size 1280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained HASH 8 objects of size 1344/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained OBJECT 8 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:91 Retained HASH 8 objects of size 1344/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal/url_for.rb:41 Retained HASH 7 objects of size 1288/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:172 Retained ARRAY 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:90 Retained HASH 7 objects of size 23968/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:109 Retained OBJECT 7 objects of size 560/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:470 Retained HASH 7 objects of size 1176/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/utils.rb:443 Retained OBJECT 7 objects of size 2240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:309 Retained ARRAY 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_controller/metal.rb:211 Retained OBJECT 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/association_scope.rb:55 Retained OBJECT 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:215 Retained OBJECT 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:253 Retained STRING 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:113 Retained HASH 7 objects of size 1176/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:311 Retained ARRAY 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained OBJECT 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:80 Retained ARRAY 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:14 Retained IMEMO 7 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:35 Retained OBJECT 6 objects of size 960/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained OBJECT 6 objects of size 960/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained OBJECT 6 objects of size 960/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/devise-4.9.3/lib/devise/rails/warden_compat.rb:5 Retained OBJECT 6 objects of size 960/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:90 Retained OBJECT 6 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:300 Retained STRING 6 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/visitors.rb:184 Retained ARRAY 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained OBJECT 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/base.rb:233 Retained ARRAY 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained HASH 6 objects of size 1008/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained IMEMO 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:48 Retained STRING 6 objects of size 3312/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/asset_tag_helper.rb:652 Retained HASH 6 objects of size 3072/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:56 Retained HASH 6 objects of size 1008/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained STRING 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:112 Retained DATA 6 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:151 Retained ARRAY 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:265 Retained REGEXP 6 objects of size 4824/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:121 Retained OBJECT 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:23 Retained ARRAY 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:51 Retained IMEMO 6 objects of size 432/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:15 Retained DATA 6 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/policies/access_policy.rb:24 Retained STRING 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:111 Retained STRING 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:148 Retained HASH 6 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/request.rb:258 Retained DATA 6 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained OBJECT 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/browser-5.3.1/lib/browser/browser.rb:105 Retained IMEMO 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:52 Retained STRING 5 objects of size 360/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:446 Retained OBJECT 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/helpers/tag_helper.rb:424 Retained HASH 5 objects of size 840/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router.rb:43 Retained HASH 5 objects of size 960/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/associations/has_many_through_association.rb:11 Retained OBJECT 5 objects of size 800/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained STRING 5 objects of size 280/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:20 Retained ARRAY 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:338 Retained ARRAY 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained OBJECT 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:241 Retained REGEXP 5 objects of size 2750/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:122 Retained OBJECT 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/context.rb:20 Retained HASH 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/renderer/renderer.rb:59 Retained HASH 5 objects of size 840/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:99 Retained HASH 5 objects of size 840/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:77 Retained HASH 5 objects of size 840/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:238 Retained ARRAY 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:252 Retained OBJECT 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:130 Retained OBJECT 5 objects of size 800/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/manager.rb:33 Retained STRING 5 objects of size 400/205345354 (in bytes) at: :30 Retained STRING 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:159 Retained HASH 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/cache.rb:145 Retained STRING 5 objects of size 800/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/random/formatter.rb:174 Retained STRING 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:117 Retained OBJECT 5 objects of size 400/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/view_paths.rb:90 Retained OBJECT 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:51 Retained OBJECT 5 objects of size 800/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:159 Retained ARRAY 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/config.rb:109 Retained HASH 5 objects of size 840/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained ARRAY 5 objects of size 1000/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:8 Retained HASH 5 objects of size 840/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:9 Retained ARRAY 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1524 Retained HASH 5 objects of size 840/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:126 Retained ARRAY 5 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:113 Retained STRING 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:173 Retained OBJECT 4 objects of size 640/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/abstract_store.rb:39 Retained IMEMO 4 objects of size 352/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained OBJECT 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:259 Retained HASH 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/query_parser.rb:166 Retained STRING 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/arel.rb:50 Retained HASH 4 objects of size 3712/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:393 Retained ARRAY 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:773 Retained ARRAY 4 objects of size 13664/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/components/work_image_show_component.rb:36 Retained ARRAY 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained DATA 4 objects of size 256/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/monitor.rb:149 Retained OBJECT 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:226 Retained ARRAY 4 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:44 Retained ARRAY 4 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/policy.rb:67 Retained OBJECT 4 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:19 Retained OBJECT 4 objects of size 320/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/access-granted-1.3.3/lib/access-granted/rails/controller_methods.rb:5 Retained STRING 4 objects of size 224/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:233 Retained HASH 3 objects of size 120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/core_ext/hash/keys.rb:11 Retained IMEMO 3 objects of size 128/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:53 Retained STRING 3 objects of size 226/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:150 Retained ARRAY 3 objects of size 120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation.rb:259 Retained OBJECT 3 objects of size 120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:143 Retained STRING 3 objects of size 120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:20 Retained STRING 3 objects of size 424/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:265 Retained OBJECT 3 objects of size 480/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:405 Retained OBJECT 3 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/http/filter_parameters.rb:64 Retained DATA 3 objects of size 240/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained STRING 3 objects of size 120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:65 Retained HASH 3 objects of size 504/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/abstract_controller/rendering.rb:64 Retained ARRAY 2 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:326 Retained IMEMO 2 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:369 Retained OBJECT 2 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/presenters/citation_display.rb:26 Retained STRING 2 objects of size 120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc/attributes.rb:48 Retained ARRAY 2 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:97 Retained HASH 2 objects of size 336/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/warden-1.2.9/lib/warden/proxy.rb:30 Retained IMEMO 2 objects of size 224/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained ARRAY 2 objects of size 7056/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/querying.rb:81 Retained STRING 2 objects of size 120/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained HASH 2 objects of size 336/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:308 Retained IMEMO 2 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:149 Retained ARRAY 2 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activerecord-7.1.2/lib/active_record/relation/merger.rb:123 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:43 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node_set.rb:232 Retained STRING 1 objects of size 72/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:259 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/objspace.rb:103 Retained HASH 1 objects of size 168/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-1.0.10/lib/citeproc/attributes.rb:11 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node_set.rb:234 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/csl-2.0.0/lib/csl/node.rb:211 Retained OBJECT 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/timeout-0.4.1/lib/timeout.rb:180 Retained OBJECT 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer.rb:11 Retained OBJECT 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/presenters/citation_display.rb:24 Retained STRING 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/presenters/citable_attributes.rb:347 Retained STRING 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/presenters/citable_attributes.rb:304 Retained STRING 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/code/scihist_digicoll/app/presenters/citable_attributes.rb:417 Retained STRING 1 objects of size 80/205345354 (in bytes) at: parser.y:195 Retained STRING 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:240 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/scrubber.rb:132 Retained DATA 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/scrubber.rb:132 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/html5/scrub.rb:68 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1308 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1295 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4/document_fragment.rb:51 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4/document_fragment.rb:48 Retained STRING 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/etag.rb:74 Retained OBJECT 1 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/railties-7.1.2/lib/rails/rack/logger.rb:21 Retained DATA 1 objects of size 192/205345354 (in bytes) at: parser.y:187 Retained OBJECT 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/actionpack-7.1.2/lib/action_dispatch/middleware/executor.rb:15 Retained ARRAY 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/runtime.rb:31 Retained HASH 1 objects of size 168/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/state.rb:98 Retained DATA 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained OBJECT 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/request_store-1.5.1/lib/request_store/middleware.rb:21 Retained HASH 1 objects of size 168/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/honeybadger-5.3.0/lib/honeybadger/breadcrumbs/logging.rb:15 Retained HASH 1 objects of size 168/205345354 (in bytes) at: /Users/jrochkind/.rubies/ruby-3.2.2/lib/ruby/3.2.0/observer.rb:154 Retained ARRAY 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/state.rb:58 Retained HASH 1 objects of size 168/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained OBJECT 1 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/mock.rb:85 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/choose.rb:14 Retained STRING 1 objects of size 160/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rack-2.2.8/lib/rack/runtime.rb:28 Retained ARRAY 1 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:61 Retained HASH 1 objects of size 168/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/document.rb:325 Retained ARRAY 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/html4/document.rb:209 Retained IMEMO 1 objects of size 88/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained STRING 1 objects of size 640/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:337 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:153 Retained DATA 1 objects of size 80/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/nokogiri-1.15.4-arm64-darwin/lib/nokogiri/xml/node.rb:1562 Retained IMEMO 1 objects of size 40/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/rails-html-sanitizer-1.6.0/lib/rails/html/sanitizer.rb:61 Retained ARRAY 1 objects of size 200/205345354 (in bytes) at: /Users/jrochkind/.gem/ruby/3.2.2/gems/loofah-2.21.4/lib/loofah/concerns.rb:172 Run `$ heapy --help` for more options Also read https://medium.com/klaxit-techblog/tracking-a-ruby-memory-leak-in-2021-9eb56575f731#875b to understand better what you are reading. jrochkind-shi scihist_digicoll (derailed_rails71_3afd9c514 !?) $ ```

[bensheldon]

I asked my colleague @jhawthorn for thoughts on this. Together we came up with a rough test plan using his vernier gem. I tried it on my own GoodJob and it had some intriguing results. The process would be:

1. bundle add vernier
2. Add this to an initializer, which will generate (and overwrite, which could be improved) a profile for every request.

# config/initializers/vernier.rb

# VERNIER_MODE=wall VERNIER_FILE=tmp/vernier.json rails s

if ENV['VERNIER_MODE']
  require 'vernier'

  Rails.application.executor.to_run do
    3.times { GC.start } if ENV['VERNIER_MODE'] == 'retained'

    @vernier = Vernier::Collector.new(ENV['VERNIER_MODE'].to_sym)
    @vernier.start
  end

  Rails.application.executor.to_complete do
    next unless @vernier

    result = @vernier.stop
    filepath = ENV.fetch('VERNIER_FILE', "tmp/vernier_#{ENV['VERNIER_MODE']}.json")
    if ENV['VERNIER_MODE'] == 'retained'
      File.write(filepath, result.to_gecko)
    else
      File.write(filepath, Vernier::Output::Firefox.new(result).output)
    end
  ensure
    @vernier = nil
  end
end

3. Run derailed in production mode with VERNIER_MODE=retained. I did this trying it on GoodJob, with derailed perf:mem_over_time and hitting the path 100 times to try to isolate the memory being retained:

VERNIER_MODE=retained TEST_COUNT=100 PATH_TO_HIT=/good_job/jobs bin/profile derailed exec perf:mem_over_time

^^ This was the result, albeit my application isn't leaking memory (😊). For Vernier's memory profiling, the left-to-right width is the percentage of retained memory (not time). In my results, it was basically the CookieJar and that really wide string, which we think is the HTML response. This is helpful because the flamegraph shows the stacktrace (unlike that heapy dump). Unfortunately it doesn't tell us who is retaining a reference to those objects, but it hopefully should help narrow down their source.

[jrochkind]

OK, I did some more data collection before getting that recent advice for another metric from @bensheldon (thank you very much for that @bensheldon , I will try it next).

It turns out it is very time-consuming to take all these measurements while keeping straight what I'm measuring and make sure I have the numbers right. But here is what I think I have discovered:

• We are assuming that derailed perf:objects retained counts AND derailed heap:diff counts are accurate representations of "what is taking so much memory" --- I am not 100% sure of this, but seems safe... I think it's less safe to be sure they are really "retained" (permanently) rather than "GC just hasn't gotten to them yet, for some reason they are harder to GC"

• I think it is safe to say the "problem" areas do involve fetching lots of our ActiveRecord objects, where our AR objects include lots of serialized JSON that gets de-serialized. It is our pages involving lots of AR objects fetched that have a problem

• For some reason this gets MUCH WORSE in Rails 7.1 , like orders of magnitude. It isn't really a problem in Rails 7.0.

  • Since it doesn't seem to exhibit in search results, which also fetches in a few objects, but does exhibit (a little bit!) in "small work" that should only fetch in a two... it is possible we have another lack-of-memoizatino or other problem going on that is fetching more than we think? Still, these should all be GC'able even if not memoized...

• I do not necessarily think we have a "memory leak" where anything is actually retaining these objects -- if we didn't have it in Rails 7.0, it seems unlikely that the few changes we made to introduce to Rails 7.1 introduced it.

• So I still at least am willing to consider the hypothesis that this really is due to some change in Rails that is making it much harder for GC to happen for AR fetches -- or specifically our fetches using jsonb and some custom ActiveModel types to handle json serialization (attr_json) .

  • Because ruby memory always increases and never decreases once allocated to VM, if it just takes a longer time for a large chunk GC-able things to get GC'd, this could increase total memory size. Something in rails 7.1 is just making these objects stick around longer before getting GC'd (even with query cache and statement cache disabled!)

• It remains a mystery of why this happens, or why it's much much worse in Rails 7.1 than Rails 7.0

• BUT.... hypothesis: rearchitecting our app to not fetching 422 pages in at once in a single web request would likely improve the situation significantly.

  • We have an issue at #905 about re-architecting to not fetch so many objects at once/in one backend request. Even with similar UX where we want to show all thumbs on page, we could fetch them in batches with client-side JS so fewer are fetched in a single backend request, presumably improving GC and overall memory consumption -- since our work endpoint with fewer pages does better!

Summary of statistics in our three actions, between Rails 7.0 and 7.1

• "large work" -- 422 pages, meaning O(422) AR objects (might be 2-3 per page, not sure)
• "small work", 2 pages, O(2) AR objects fetched
• "search results", 16 results, O(16) AR objects fetched, still might be 2-3 AR objects per result

"small work" gets substantially bigger in 7.1, "large work" gets unreasonably bigger in 7.1.

I can not explain why "search results" gets SMALLER retained memory in Rails 7.1, but it is same order of magnitude.

These numbers are not identical from run to run, but are close, same order of magnitude.

	Perf:objects retained biggest gem name	Perf:objects retained biggest gem size	Heap:diff retained
Rails 7.1 large work	activerecord	91 meg	312 meg
Rails 7.1 small work	activesupport	1.4 meg	1.6 meg
Rails 7.1 search results	ostruct	402K	532 K
Rails 7.0 large work	nokogiri	12K	8 meg
Rails 7.0 small work	net	708K	160 BYTES
Rails 7.0 search results	net	933K	71K

[jrochkind]

@bensheldon OK, here's the vernier output json files. i used Firefox Profiler online to look at them.

I think it's consistent with our model so far, without unfortunately telling us anything new.

The memory allocated comes from the AR fetch at:

https://github.com/sciencehistory/scihist_digicoll/blob/70f83852a5b097e8ccb908acbc25ca3ef635aa59/app/components/work_image_show_component.rb#L46-L48

Which is expected to be the main AR fetch for this page (the one that fetches in the ~422 AR objects one for each digitized page).

So that confirms the model we had already developed of wha'ts going on I think... the question is why those end up getting observed as "retained" -- and especially why it differs between the same app with Rails 7.0 and 7.1!

~Comparing the Rails 7.0 vs 7.1 vernier doc.... they actually look pretty similar...~ (wrong had wrong doc) I'm not sure if there any units on these flamegraphs to know if they are the same or not!

Curious if you see anything interesting I'm not seeing in either of these, @bensheldon !

I think I'm leaning towards declaring defeat on really solving the mystery.... I think the answer is

a) to improve RAM performance we probably have to stop fetching 422 AR objects (with large embedded JSON) in a single Rails request, and it will remain a mystery why this became more of a problem in Rails 7.1, but also

b) after looking at it and thinking about it (including looking at our puma metrics now being captured on heroku dashboard), I think I can actually tweak our puma worker/thread figures to probably use a lot less memory without actually losing much capacity, and thus fit comfortably within same dyno again.

vernier_retained_rails71_3afd9c514.json

~vernier_retained_rails70_cc8e84042.json~ (wrong)

[jrochkind]

I guess there's not yet the technology to give us an easy report of what is actually referencing the retained objects!

[jrochkind]

OK, here is actual vernier graphs produced by above script, for both the Rails 7.0 and 7.1 version to compare.

I'm still not sure the comparison tells us anything useful -- they are still both pretty similar, both showing a lot of allocations from the AR fetch in work.ordered_viewable_members(current_user: current_user) as expected -- I don't know if there are units on these to compare?

Confirming that, yeah, fetching 422 AR objects does allocate a lot of RAM -- still unclear why it would be retained instead of GC'd, or why the effect on app RAM consumtion or retention would differ from Rails 7.0 to 7.1.

vernier_retained_rails70_cc8e84042.json vernier_retained_rails71_3afd9c514.json

In somewhat better news... yesterday we deployed at 70f83852 a version that reduces object allocations, largely by making sure a lot of string literals are frozen, plus a couple other allocation reductions in ActiveRecord-related hotspots.

One day is really too little to see a trend when as we can see it's so variable... but maybe this may have reduced overall RAM consumption in heroku after all, maybe back to the levels we had before before Rails 7.1 deploy, where RAM mostly stays under 2GB on our single 2.5GB performance-m dyno? (But you can see in the last week there was a 12-20 hour period before where it stayed this low too, so).

i also noticed that in my Rails 7.1 branch I had updated view_component from 3.6 to 3.7, so I did some derailed profiling with Rails 7.1 and view_component back at 3.6 just to make sure that didn't make a difference -- it didn't.

@bensheldon I really appreciate your help, I understand the tools available and how to use them much better now. I think I may be close to declaring defeat, for now, on understanding why derailed "retained memory" metrics go up an order of magnitude or two when I upgraded to Rails 7.1. If you have any other insights I'd love to hear them, but don't want to take any significant more of your time when I don't have the time to really delve into your suggestions more at present!

Plan for adjusting puma worker/thread counts in same dyno formation

Right now we run a heroku performance-m dyno, with 2.5 GB of RAM. It also appears to have two "virtual CPUs", although this is actually maybe only one real cpu and one "hyperthread". From https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#process-count-value , which also has other heroku puma sizing advice we are looking at.

We currently run 3 puma workers, with 3 threads each, so 3x3 is 9 total threads.

I'm not sure how we decided on this -- the heroku docs above actually recommend 2 workers with 5 threads each (2x5 is 10 total threads).

One reason to run multiple worker processes is because ruby can't do true multi-core parallelism, you want at least as many ruby processes as cpus. But we only have 1-2, so three worker processes is probably not necessary, and possibly is even counter-productive?

Changing to 2 workers with 4 threads each should be more RAM efficient, while providing about the same number of total threads (2x4 == 8), which should mean about the same actual capacity. We could also try heroku's recommendation of 2x5==10.

I will try that next week, don't want to do it at the end of the week.

Our current 7-day metrics, show that dyno load mostly stays under 2 (where we want it), but occasionally has spikes up to 3 or 4 showing in those spikes we don't actually have enough CPU resources to handle capacity.

We've only been keeping heroku puma stats for 2 days, but in those two days it also shows mostly only two thread utilization (20% of capacity), up to 5 thread (55% capacity), but occasional spikes to 8-9 threads (90-100%) capacity, suggesting similar, we mostly do fine, but do have spikes that top off our capacity. In the last 72 hours, we didn't actually have any timeout alerts, which we do get occasionally (a minute or two ever few weeks), so these spikes are probably enough to slow down response times somewhere under the timeout values when queues are filled up.

The 100% puma capacity does line up with our dyno load max -- with no apparent RAM spike there though. There is a dyno restart there too, I think maybe the dyno actually got totally locked up and heroku restarted it? (if so not sure why we didn't get any error alerts)... Other spikes smaller than 100% in the past two days don't seem to have load bumps corresponding or other unusual activity.

So I think we are basically handling our usual traffic, with occasional brief spikes our current resources have trouble handling (we think from bots, mostly) -- that my guess is we'd need more dynos power to handle, more workers wouldn't help even if we could fit them in RAM? I'd hope the changed worker/thread config would leave that about the same, with less RAM!

[bensheldon]

After a substantial pairing session with @jhawthorn, we're pretty sure we found a problem introduced into Rails 7.1. In brief, this change to RoutesProxy causes a new anonymous class to be created for every RoutesProxy instance which is every request; this anonymous class references the controller, which references the models, which is pretty big. This anonymous class is fairly durable (maybe ~2k of them will be retained by the VM) so we believe it's the main source of the memory growth you saw when upgrading 🎉

(It might be particular to your app because you use main_app.root_path, which isn't common, but shouldn't be a problem)

Rails PR: https://github.com/rails/rails/pull/50298

Before patching (looks like what you're familiar with)

``` $ PATH_TO_HIT=/works/76zn9cs foreman run bundle exec derailed exec perf:heap_diff .... Diff ==== Retained STRING 10545 objects of size 437640/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activemodel-7.1.2/lib/active_model/type/string.rb:35 Retained OBJECT 10314 objects of size 825120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activemodel-7.1.2/lib/active_model/attribute.rb:9 Retained ARRAY 5404 objects of size 216160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:30 Retained ARRAY 5268 objects of size 210720/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/attr_json-2.2.0/lib/attr_json/type/array.rb:26 Retained STRING 1642 objects of size 248428/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained OBJECT 939 objects of size 75120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/role.rb:59 Retained DATA 668 objects of size 57448/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:72 Retained OBJECT 643 objects of size 25720/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/attr_json-2.2.0/lib/attr_json/model.rb:151 Retained HASH 628 objects of size 105504/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/attr_json-2.2.0/lib/attr_json/model.rb:382 Retained STRING 616 objects of size 36520/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/utils.rb:476 Retained HASH 609 objects of size 102312/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/attr_json-2.2.0/lib/attr_json/model.rb:300 Retained STRING 555 objects of size 195496/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained OBJECT 550 objects of size 88000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/associations.rb:320 Retained HASH 550 objects of size 74224/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation.rb:28 Retained ARRAY 527 objects of size 21080/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/role.rb:73 Retained HASH 522 objects of size 20880/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/role.rb:22 Retained OBJECT 499 objects of size 79840/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/delegation.rb:132 Retained STRUCT 457 objects of size 18280/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/arel/table.rb:89 Retained OBJECT 433 objects of size 34640/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained OBJECT 405 objects of size 32400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:16 Retained OBJECT 401 objects of size 32080/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/predicate_builder.rb:68 Retained OBJECT 373 objects of size 14920/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/arel/predications.rb:18 Retained OBJECT 343 objects of size 13720/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained HASH 336 objects of size 311808/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/result.rb:177 Retained ARRAY 321 objects of size 12840/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/role.rb:10 Retained ARRAY 317 objects of size 12680/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1367 Retained HASH 317 objects of size 53256/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:28 Retained ARRAY 315 objects of size 12600/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:10 Retained OBJECT 310 objects of size 12400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activemodel-7.1.2/lib/active_model/dirty.rb:377 Retained HASH 310 objects of size 545600/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/querying.rb:69 Retained DATA 308 objects of size 24640/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/policy.rb:24 Retained HASH 305 objects of size 283040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/json/common.rb:216 Retained HASH 305 objects of size 12200/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activemodel-7.1.2/lib/active_model/attribute_mutation_tracker.rb:71 Retained ARRAY 301 objects of size 12040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:15 Retained HASH 295 objects of size 273760/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activemodel-7.1.2/lib/active_model/attribute_set/builder.rb:22 Retained HASH 286 objects of size 39216/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/associations.rb:339 Retained OBJECT 276 objects of size 88320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/persistence.rb:640 Retained HASH 240 objects of size 33768/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/set.rb:246 Retained HASH 230 objects of size 17904/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/policy.rb:31 Retained STRING 224 objects of size 6938848/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:72 Retained STRING 206 objects of size 16480/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained IMEMO 179 objects of size 145592/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained HASH 171 objects of size 28728/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained IMEMO 170 objects of size 6800/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:20 Retained DATA 165 objects of size 11176/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/monitor.rb:238 Retained STRING 162 objects of size 6480/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/buffers.rb:23 Retained HASH 162 objects of size 16848/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/config.rb:108 Retained IMEMO 160 objects of size 12800/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained HASH 159 objects of size 16216/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/proxy.rb:29 Retained DATA 158 objects of size 12640/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:97 Retained HASH 158 objects of size 6320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/strategies/base.rb:46 Retained OBJECT 158 objects of size 6320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:88 Retained IMEMO 157 objects of size 6928/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained STRING 154 objects of size 13488/13433828 (in bytes) at: :9 Retained STRING 153 objects of size 6120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/request.rb:512 Retained ARRAY 153 objects of size 6120/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/models/work/text_page.rb:30 Retained DATA 152 objects of size 12160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained IMEMO 152 objects of size 6080/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/role.rb:13 Retained IMEMO 150 objects of size 10800/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/role.rb:23 Retained HASH 148 objects of size 5920/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/ordered_options.rb:106 Retained HASH 146 objects of size 5840/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/request/utils.rb:58 Retained OBJECT 137 objects of size 5480/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/associations/association_scope.rb:55 Retained HASH 135 objects of size 22680/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/hash_with_indifferent_access.rb:265 Retained OBJECT 132 objects of size 21120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/proxy.rb:381 Retained STRING 109 objects of size 8720/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/journey/visitors.rb:184 Retained ARRAY 108 objects of size 8640/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/policy.rb:67 Retained OBJECT 107 objects of size 8560/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/rails/controller_methods.rb:5 Retained OBJECT 102 objects of size 8160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:489 Retained OBJECT 99 objects of size 7920/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:19 Retained OBJECT 99 objects of size 7920/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:300 Retained OBJECT 97 objects of size 7760/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:470 Retained OBJECT 97 objects of size 7760/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/view_paths.rb:90 Retained OBJECT 94 objects of size 3760/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/view_component-3.7.0/lib/view_component/base.rb:75 Retained ARRAY 92 objects of size 3680/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/components/work_image_show_component.rb:36 Retained OBJECT 92 objects of size 7360/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/filter_parameters.rb:64 Retained ARRAY 90 objects of size 3600/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:51 Retained IMEMO 89 objects of size 3560/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:48 Retained OBJECT 89 objects of size 7120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:57 Retained ARRAY 88 objects of size 3520/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained IMEMO 88 objects of size 6336/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained HASH 88 objects of size 14784/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/policy.rb:9 Retained HASH 88 objects of size 14784/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:126 Retained DATA 87 objects of size 6960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained STRING 87 objects of size 3480/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/url.rb:227 Retained HASH 87 objects of size 14616/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained HASH 86 objects of size 14448/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1283 Retained DATA 86 objects of size 6880/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:151 Retained ARRAY 86 objects of size 3440/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/models/work.rb:229 Retained STRING 86 objects of size 3440/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained IMEMO 85 objects of size 7480/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained STRING 85 objects of size 3400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:114 Retained OBJECT 85 objects of size 3400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_controller/metal/strong_parameters.rb:1289 Retained IMEMO 85 objects of size 3400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:35 Retained ARRAY 85 objects of size 3400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:107 Retained ARRAY 85 objects of size 3400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained STRING 84 objects of size 3360/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:116 Retained OBJECT 84 objects of size 3360/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/context.rb:19 Retained ARRAY 84 objects of size 3360/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained STRING 84 objects of size 3360/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:148 Retained STRING 84 objects of size 6720/13433828 (in bytes) at: :30 Retained ARRAY 84 objects of size 3360/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/associations/collection_association.rb:87 Retained HASH 84 objects of size 14112/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:77 Retained OBJECT 83 objects of size 3320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:23 Retained ARRAY 83 objects of size 3320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/config.rb:109 Retained OBJECT 83 objects of size 3320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:253 Retained DATA 83 objects of size 6640/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:124 Retained STRING 83 objects of size 4648/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/flows.rb:20 Retained HASH 83 objects of size 3320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/query_parser.rb:166 Retained OBJECT 83 objects of size 3320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:226 Retained HASH 82 objects of size 13776/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained HASH 82 objects of size 13776/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_controller/metal/url_for.rb:41 Retained STRING 81 objects of size 3240/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:111 Retained ARRAY 81 objects of size 3240/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:32 Retained HASH 81 objects of size 13608/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/proxy.rb:30 Retained HASH 81 objects of size 13608/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:238 Retained OBJECT 81 objects of size 25920/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/controllers/works_controller.rb:72 Retained HASH 80 objects of size 13440/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/flows.rb:10 Retained ARRAY 80 objects of size 3200/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:773 Retained HASH 80 objects of size 273920/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:109 Retained ARRAY 80 objects of size 3200/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:113 Retained STRING 80 objects of size 3200/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:113 Retained OBJECT 79 objects of size 3160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/browser-5.3.1/lib/browser/browser.rb:105 Retained OBJECT 79 objects of size 25280/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/rendering.rb:110 Retained OBJECT 79 objects of size 3160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:80 Retained HASH 79 objects of size 13272/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/journey/router.rb:43 Retained HASH 79 objects of size 3160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/request.rb:258 Retained HASH 79 objects of size 15168/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/associations/has_many_through_association.rb:11 Retained HASH 79 objects of size 13272/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained STRING 79 objects of size 3160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:65 Retained ARRAY 78 objects of size 3120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_controller/metal.rb:211 Retained DATA 78 objects of size 6240/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/policies/access_policy.rb:15 Retained ARRAY 78 objects of size 3120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/query_methods.rb:1524 Retained HASH 78 objects of size 39936/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/request/session.rb:56 Retained ARRAY 78 objects of size 15600/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/access-granted-1.3.3/lib/access-granted/policy.rb:8 Retained OBJECT 78 objects of size 3120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/monitor.rb:215 Retained DATA 77 objects of size 6160/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/policies/access_policy.rb:24 Retained STRING 77 objects of size 4312/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/core_ext/string/output_safety.rb:233 Retained ARRAY 77 objects of size 3080/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation.rb:259 Retained DATA 77 objects of size 6160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/proxy.rb:31 Retained OBJECT 76 objects of size 3040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained ARRAY 76 objects of size 3040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/parameter_filter.rb:97 Retained ARRAY 76 objects of size 3040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained STRING 76 objects of size 3040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:159 Retained OBJECT 76 objects of size 3040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/base.rb:233 Retained OBJECT 76 objects of size 3040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:91 Retained OBJECT 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/table_metadata.rb:56 Retained STRING 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/arel.rb:50 Retained ARRAY 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/querying.rb:81 Retained STRING 75 objects of size 41400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/helpers/asset_tag_helper.rb:652 Retained STRING 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:117 Retained STRING 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/request.rb:173 Retained IMEMO 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:52 Retained ARRAY 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/helper.rb:252 Retained ARRAY 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/relation/where_clause.rb:33 Retained HASH 75 objects of size 13800/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:172 Retained DATA 75 objects of size 4800/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/monitor.rb:149 Retained OBJECT 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:259 Retained ARRAY 75 objects of size 3000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:265 Retained STRING 75 objects of size 48000/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:337 Retained OBJECT 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/context.rb:20 Retained ARRAY 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:14 Retained OBJECT 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:130 Retained ARRAY 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/lookup_context.rb:44 Retained ARRAY 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/blacklight-7.35.0/app/helpers/blacklight/blacklight_helper_behavior.rb:75 Retained HASH 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/cache.rb:145 Retained HASH 74 objects of size 12432/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/utils.rb:443 Retained HASH 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/renderer/renderer.rb:59 Retained OBJECT 74 objects of size 11840/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/messages/rotator.rb:45 Retained OBJECT 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/proxy.rb:51 Retained HASH 74 objects of size 2960/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/core_ext/hash/keys.rb:11 Retained ARRAY 73 objects of size 2920/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_negotiation.rb:90 Retained STRING 72 objects of size 2880/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:112 Retained HASH 72 objects of size 12096/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:308 Retained HASH 72 objects of size 2880/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:309 Retained OBJECT 72 objects of size 2880/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:241 Retained HASH 71 objects of size 11928/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained STRING 71 objects of size 2840/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/journey/router/utils.rb:20 Retained OBJECT 71 objects of size 11360/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/devise-4.9.3/lib/devise/rails/warden_compat.rb:5 Retained HASH 71 objects of size 11928/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/abstract_controller/rendering.rb:64 Retained HASH 70 objects of size 11760/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:311 Retained STRING 70 objects of size 5040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:446 Retained HASH 69 objects of size 64032/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:393 Retained OBJECT 69 objects of size 2760/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/flash.rb:143 Retained OBJECT 69 objects of size 11040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/warden-1.2.9/lib/warden/manager.rb:33 Retained IMEMO 69 objects of size 4968/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/policies/access_policy.rb:15 Retained OBJECT 69 objects of size 11040/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:609 Retained HASH 69 objects of size 11592/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/cookie_store.rb:99 Retained DATA 67 objects of size 5360/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/policies/access_policy.rb:20 Retained OBJECT 67 objects of size 10720/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/route_set.rb:405 Retained OBJECT 66 objects of size 10560/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/session/abstract_store.rb:39 Retained OBJECT 66 objects of size 10560/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/remote_ip.rb:90 Retained OBJECT 66 objects of size 10560/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/middleware/cookies.rb:640 Retained OBJECT 66 objects of size 10560/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/response.rb:159 Retained STRING 65 objects of size 10400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/random/formatter.rb:174 Retained OBJECT 65 objects of size 10400/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.2/lib/active_support/message_encryptor.rb:189 Retained CLASS 65 objects of size 42640/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb:33 Retained OBJECT 61 objects of size 2440/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionview-7.1.2/lib/action_view/helpers/tag_helper.rb:424 Retained OBJECT 60 objects of size 19200/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_controller/metal.rb:309 Retained STRING 6 objects of size 280/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:80 Retained OBJECT 3 objects of size 120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-1.0.10/lib/citeproc/variable.rb:125 Retained IMEMO 2 objects of size 88/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:51 Retained HASH 2 objects of size 1096/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-1.0.10/lib/citeproc/attributes.rb:11 Retained IMEMO 2 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/timeout-0.4.1/lib/timeout.rb:105 Retained STRING 2 objects of size 120/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:305 Retained IMEMO 2 objects of size 240/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained OBJECT 2 objects of size 160/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/presenters/citation_display.rb:26 Retained STRING 2 objects of size 320/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:265 Retained IMEMO 2 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:80 Retained ARRAY 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/choose.rb:25 Retained STRING 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/mock.rb:260 Retained ARRAY 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/runtime.rb:31 Retained IMEMO 1 objects of size 48/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:48 Retained OBJECT 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/request_store-1.5.1/lib/request_store/middleware.rb:21 Retained DATA 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/body_proxy.rb:12 Retained OBJECT 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/lograge-0.14.0/lib/lograge/rails_ext/rack/logger.rb:21 Retained STRING 1 objects of size 104/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/etag.rb:37 Retained ARRAY 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/honeybadger-5.3.0/lib/honeybadger/util/sanitizer.rb:59 Retained DATA 1 objects of size 72/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/objspace.rb:102 Retained STRING 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/http/mime_type.rb:293 Retained DATA 1 objects of size 72/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/timeout-0.4.1/lib/timeout.rb:66 Retained STRING 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/etag.rb:74 Retained STRING 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:82 Retained STRING 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/presenters/citable_attributes.rb:341 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/query_cache.rb:111 Retained OBJECT 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer.rb:11 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:443 Retained STRING 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-1.0.10/lib/citeproc/attributes.rb:48 Retained STRING 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/presenters/citable_attributes.rb:347 Retained STRING 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/app/presenters/citable_attributes.rb:304 Retained STRING 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/derailed_benchmarks-2.1.2/lib/derailed_benchmarks/tasks.rb:266 Retained HASH 1 objects of size 168/13433828 (in bytes) at: :35 Retained ARRAY 1 objects of size 40/13433828 (in bytes) at: :35 Retained HASH 1 objects of size 512/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/formats/html.rb:35 Retained DATA 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/history.rb:12 Retained ARRAY 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/renderer/state.rb:58 Retained OBJECT 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/asynchronous_queries_tracker.rb:51 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/objspace.rb:103 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/weakref.rb:56 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/weakref.rb:45 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:86 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:125 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:93 Retained IMEMO 1 objects of size 48/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:55 Retained DATA 1 objects of size 1464/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/monitor.rb:202 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:438 Retained IMEMO 1 objects of size 40/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:50 Retained ARRAY 1 objects of size 200/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/csl-2.0.0/lib/csl/node.rb:428 Retained DATA 1 objects of size 136/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:80 Retained STRING 1 objects of size 160/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/citeproc-ruby-2.0.0/lib/citeproc/ruby/format.rb:150 Retained DATA 1 objects of size 80/13433828 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/rack-2.2.8/lib/rack/response.rb:46 Run `$ heapy --help` for more options Also read https://medium.com/klaxit-techblog/tracking-a-ruby-memory-leak-in-2021-9eb56575f731#875b to understand better what you are reading. ```

After patching 🎉

``` $ PATH_TO_HIT=/works/76zn9cs foreman run bundle exec derailed exec perf:heap_diff .... Diff ==== Retained IMEMO 2 objects of size 88/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:51 Retained IMEMO 2 objects of size 88/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:48 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/weakref.rb:56 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:84 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/weakref.rb:45 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:86 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:125 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/delegate.rb:93 Retained IMEMO 1 objects of size 48/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:55 Retained DATA 1 objects of size 1464/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/monitor.rb:202 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:438 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:443 Retained IMEMO 1 objects of size 40/2048 (in bytes) at: /Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/3.2.0/objspace.rb:103 Run `$ heapy --help` for more options Also read https://medium.com/klaxit-techblog/tracking-a-ruby-memory-leak-in-2021-9eb56575f731#875b to understand better what you are reading. ```

We were able to debug this thanks to @jhawthorn's sheap gem (under development) which makes it really easy to explore a heap dump and answer the question you posed "what object is referencing this and how is it being retained?"

# $ irb
require './lib/sheap.rb

diff = Sheap::Diff.new("/Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/tmp/2023-12-07T13:24:15-08:00-heap-1.ndjson"
, "/Users/bensheldon/Repositories/sciencehistory/scihist_digicoll/tmp/2023-12-07T13:24:15-08:00-heap-2.ndjson")

# Find one of the Work records that's been retained
model = diff.after.class_named("Work").first.instances[200]
=> <OBJECT 0x117cf5c98 Work (4 refs)>

# Find the path to the (default) root
diff.after.find_path(model)
=>
[<ROOT vm (2984 refs)>,
 <IMEMO 0x126c9ab68 callcache (1 refs)>,
 <IMEMO 0x126c9acf8 ment (4 refs)>,
 <CLASS 0x12197c080 (anonymous) (15 refs)>,
 <OBJECT 0x122ddba08 (0x12197c080) (3 refs)>,
 <OBJECT 0x117cfc458 WorksController (13 refs)>,
 <OBJECT 0x117cf7318 WorkImageShowComponent (15 refs)>,
 <OBJECT 0x117cf5c98 Work (4 refs)>]

diff.after.at("0x126c9ab68").data
=>
{"address"=>"0x126c9ab68",
 "type"=>"IMEMO",
 "shape_id"=>0,
 "slot_size"=>40,
 "imemo_type"=>"callcache",
 "references"=>["0x126c9acf8"],
 "file"=>"/Users/bensheldon/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/actionpack-7.1.2/lib/action_dispatch/routing/routes_proxy.rb",
 "line"=>48,
 "method"=>"public_send",
 "generation"=>288,
 "memsize"=>40,
 "flags"=>{"wb_protected"=>true, "old"=>true, "uncollectible"=>true, "marked"=>true}}

[jrochkind]

@bensheldon

First, in a possibly deleted comment you asked:

What is Kith::Indexable do? I saw it far down in the stacktrace and it looked interesting because, if I understand it, it collects records when they're updated in order to do a batch update in Solr

You got the right idea of what it does basically -- it shows up in stack traces because it wraps every request in an around_filter, which is really annoying for stack traces -- but for most requests it does nothing but "yield", it's just sort of like "middleware". I am confident that in the requests we are profiling it is doing nothing (as the requests we are profiling don't do any writes).

But good call, as it does collect things to batch (but should only do so for the duration of a request, and then release them) -- if it had a bug, perhaps it could have a leak or extra retention (although again it didn't differ between Rails 7.0 and 7.1)

But then the meat!

OK! Wow! Thank you for finding that!

I know usually "it's not you it's Rails" , but I had my suspicions anyway, since so little had changed in my app between 7.0 and 7.1.

MOST apps don't fetch hundreds of AR objects or otherwise use as much RAM to process a single request as mine does, so the bug as you describe it wouldn't be as significant to most apps. So it is true that to some extent this app's "bad behavior" does make the bug worse. (And in general does lead to more memory consumption for this app, since it does need enough RAM to cover the RAM-hungry requests, as many as may be running concurrently).

So that probably explains why nobody else noticed the bug, it took a somewhat poorly behaved app like this one to notice it.

BUT it is a bug! Wow, thank you! Or... at least a RAM regression if not a bug. The patch is pretty confusing to me! And probably would be non-trivial, even if not as massive, total memory increase for most apps, I'm guessing.

(It might be particular to your app because you use main_app.root_path, which isn't common, but shouldn't be a problem)

I don't recall using main.root_path or have any idea what we use it for, I wonder if we can stop using it... haven't grepped yet (I'm actually off today mostly)...

Since you're giving me profiles of my own app, apparently you got my app to run, and in production mode? I am gratified you found it not too hard to set it up to do that, and apologize for how hard it was (I could have given you some ENV vars instead of making you figure it out yourself!), and grateful you spent the time!

I may try patching the patch in locally instead of waiting for the Rails version... but probably don't feel safe doing that until it's merged into Rails.

Looking at it... wow how funky, who would have predicted that the use of instance_eval would retain memory where class_eval would not... i still don't really understand why.

[jrochkind]

@bensheldon in understanding the tooling and metrics, one mystery is why the vernier profiles looked so similar between the two cases, even though the derailed heap_diff and objects reports differed, and indeed there was retention happening in 7.1 that was not in 7.0. I guess the vernier profile was in fact more measuring allocations (that were the same in both cases), rather than retentions (which did differ, as the derailed metrics it turns out correctly indicated).

[jrochkind]

Oh this ALSO explains why the deploy we did a couple days ago that reduced allocations (but wasn't logically expected to reduce retention) may indeed have had a measurable impact on max RAM use -- the Rails regression means that anything referenced in a controller instance variable was retained past action complete, so turning non-trivial number of things in the models ( de-serialized json via attr_json gem) from new allocations to frozen strings would plausibly have an impact. (and maybe strings referenced in ViewComponents used in rendering wound up referenced too? Froze a bunch of those too)

It all kind of fits together into a theory, which is a good sign.

[jrochkind]

(And wait, I assumed you had run our app since you showed us an "after patch" profile with PATH_TO_HIT=/works/76zn9cs which is my app... but I actually don't understand how you would possibly have gotten the app set up with data in database to represent /works/76zn9cs! Just curious to know if you actually got our app running and if so how!)

[bensheldon]

in a possibly deleted comment

Oops, I had too many tabs open and had started writing a response, then realized I was wrong and abandoned it, and then accidentally submitted it later. You can ignore that 😊

Looking at it... wow how funky, who would have predicted that the use of instance_eval would retain memory where class_eval would not... i still don't really understand why.

I wouldn't have caught it without @jhawthorn's help, but we were able to isolate the heap dump down to the RoutesProxy, and then looked up the blame on that and saw it changed between Rails versions. Then @jhawthorn explained:

It's instance_evaling a def method. That means it's defining a new method on just that instance (not on the class definition). To do that, the Ruby VM creates a new anonymous class just for that instance and puts the method in it. That anonymous class, because its a class is held onto a little more tightly by the VM/heap. Because a new RoutesProxy is create on every request, the Ruby VM is creating a new anonymous class for every request, and those build up over time in the heap (and because they reference the controller and everything it references, it can be quite a lot). At a certain point the older ones do seem to get cleaned up, but happens separately than a normal GC sweep of less-tightly held objects.

I assumed you had run our app since you showed us an "after patch"

I did! It really wasn't too bad to get running, set up an admin account, and create some "asdfadsfadsf" records. I dunno why that URL is the same, unless maybe 76zn9cs is just id: 1 passed through a hashing function.

This was my eventual .env to boot the app (the XXXX are my actual values, so they must not be too important... though I never did get Solr connected correctly, so every update required reloading the page past a connection error, but it didn't seem to matter):

AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
APP_URL_BASE=http://localhost:3000
SOLR_URL=http://localhost:8983/solr
LOCKBOX_MASTER_KEY=XXXXXXXX
S3_BUCKET_UPLOADS=XXXXXXXX

S3_BUCKET_ORIGINALS=XXXXXXXX
S3_BUCKET_ORIGINALS_VIDEO=XXXXXXXX
S3_BUCKET_DERIVATIVES_VIDEO=XXXXXXXX
S3_BUCKET_DERIVATIVES=XXXXXXXX
S3_BUCKET_ON_DEMAND_DERIVATIVES=XXXXXXXX
S3_BUCKET_DZI=XXXXXXXX
SECRET_KEY_BASE=XXXXXXXX
RAILS_DATABASE_URL=postgres://localhost:5432/scihist_digicoll_development
FORCE_SSL=false

[jrochkind]

I'm honestly kind of proud that you were able to get the app running without too much trouble, thank you!

And interested in your unintentional feedback on pain points in doing so -- you can ignore the rest of this if you want, but I'm kind of talking to myself.

That particular action we were profiling doesn't use solr at all -- although I'm to sure about the error that happened anyway (why!) or why you were able to get past it if an error did happen! Also solr ought to be easy to get going with an automated thing that even downloads solr for you, but oh well.

AWS is only needed for storing images (we have a LOT of images of digitized historical document pages), and presumably your test data didn't include actual images, which is fine for this case, although I wonder how representative your test data was of what we were testing, or how you managed to create it! -- it maybe didn't include hundreds of page objects.

I actually COULD have sent you a JSON file that you could have used to load in our actual (non-sensitive in this case) production data, we have a function for that we use for copying data from staging to dev to reproduce problems -- but that would have required you to set up some of the AWS stuff.

All in all I'm pleased you were able to get it to work though, it's a good test of our bootstrap instructions and methods, which I know aren't as perfect as they could be or as easy as a docker container, but apparently are not ridiculous! Thank you!

[bensheldon]

I'm honestly kind of proud that you were able to get the app running without too much trouble

You should be proud! It's a good app; I've seen way weirder.

That particular action we were profiling doesn't use solr at all

It didn't, but my manually creating the records through the admin panel did require solr. I just had to sort of limp through that activity, but it took 5 minutes (and another 5 minutes to find the "publish" button in the UI).

My recommendation there would be:

• create some seeds.rb for both a default admin user and some content
• try to get a local storage option for dev instead of AWS
• make solr fully optional

(maybe the latter 2 already exist and I didn't figure out how to configure that; or just rescue errors in seeds.rb so that they mostly work ok regardless of whether they actually are set up)

(I'm actually off today mostly)

Get offline and enjoy your day! You deserve it ❤️

[jrochkind]

I kind of sort of understand how the instance_eval ended up preserving memory (basically in a closure)... although I'm not sure why the anonymous (singleton?) classes it created were themselves retained...

But also what this makes me think is -- if this hadn't been a regression between 7.0 and 7.1, and if I hadn't noticed, because my poorly behaved app made it more noticeable, and if you hadn't spent the time with me and then brought in @jhawthorn.... I wonder if there could be other retaining-too-much bugs/optimizations-in-waiting in Rails, maybe that have been there for a long time, ... and how feasible it would be for them ever to be found, or what tooling might (sheap!) might make it feasible to find them.

try to get a local storage option for dev instead of AWS

There is one, you just didn't discover it! And perhaps it could have been documented better. STORAGE_MODE=dev_file would have given you the local storage option that should have Just Worked. Although it's possible it wouldn't, we mostly don't use it anymore, instead using STORAGE_MODE=dev_s3 that's a single-bucket approach to S3 -- we wound up finding too many bugs that weren't triggered in dev_file mode, so stopped loving using it in dev.

Also, right, trying to write to solr in admin when adding records... you could have avoided that with SOLR_INDEXING=false. Which would make solr fully optional for everythign but public searching -- there's no public search without solr, although there is a more limited admin dashboard search.

I guess ideally we'd doc all the env's better, although they are all centralized in the env.rb, it's messy there.

[jrochkind]

OK some updates and plans

R14 out of memory is back

I had hoped that the deploy on Wednesday Dec 6th (70f8385) which reduced allocations might return our app to it's historical RAM patterns of staying under 2GB.

While it initially looked like it did as a trend, last night we did get some R14's again. While not an emergency to get an R14 now and then, we like to eliminate them because: app perf is degraded while they happen; they are errors in our logs/monitoring services distracting us from "real" errors; and solve the problem now before it does become an emergency, if we keep our app from having R14's we'll never get to R15.

So we do want to consider doing something.

Let's patch in the fix

@bensheldon's PR to Rails at https://github.com/rails/rails/pull/50298 is now merged into Rails main. This makes me feel secure about "monkey-patching" it into our app, until a 7.1.x Rails release happens including it.

The change is to one method, so is pretty monkey-patch-able.

This should hopefully return our app more fully to it's previous memory profie... we will see?

Change worker/thread counts: in the future

I am still optimistic about changing our worker/thread counts on our single performance-m dyno.

from currently 3 workers x 3 threads to 2 workers x 4 or 5 threads -- closer to heroku standard recommendations

I think this should give us lower RAM usage with pretty similar performance profile, not actually giving maybe anything up.

But in order to only change one thing at a time to have a better idea of what had an effect, I'm going to hold off on this, and start with patching the memory fix in. Filed as separate ticket at #2465

Extra ruby language heroku stats may be busting our logs

Our logs go into papertrail, and our plan has a certain capacity.

Lately we have been exceeding our papertrail capacity on many/most days. This approximately coincides with when we turned on ruby language metrics at #2446.

I'm not sure why -- haven't figured it out yet. But we probably need to reverse it -- or identify what is causing the logging increase, and have papertrail exclude those lines? Changing from our current Papertrail 250 MB/month plan to the next larger one at 550 MB/month would be an extra $35/month, which we don't really want to pay.

I'm actually having a lot of trouble figuring out what extra log lines (that would make it to papertrai) were created by the settings to turn on ruby language and puma metrics... the docs aren't clear, and I don't see anything obvious in visual inspection of logs.

I may start just by turnign this stuff off again, to see if our log volume goes down again.

[jrochkind]

RAM usage trends already looked like they got closer to with Rails 7.0 after we significantly reduced allocations.

After "monkey-patching" in https://github.com/rails/rails/pull/50298 (only deploy showing on graph below), RAM usage looks to have returned to Rails 7.0 trends, staying reliably below 2G on our 2.5G-limit dyno.