Shopify / bootsnap

Boot large Ruby/Rails apps faster
MIT License
2.68k stars 183 forks source link

`bin/bootsnap precompile` not compiling everything #465

Closed n-rodriguez closed 10 months ago

n-rodriguez commented 10 months ago

Hi there!

I've enabled bootsnap logs in my Rails docker containers (1 container with puma / 3 containers with sidekiq / 1 container with crono) and found that some paths were not pre-compiled by bootsnap.

All are Rails engines files (under <gem>/app or <gem>/config :

(I've sorted logs output for better readability, the logs are the same for the 5 containers)

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/crono-ab140b6600af/app/controllers/crono/jobs_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/crono-ab140b6600af/app/controllers/crono/application_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/crono-ab140b6600af/config/routes.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/crono-ab140b6600af/app/models/crono/application_record.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/crono-ab140b6600af/app/models/crono/crono_job.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/crono-ab140b6600af/config/routes.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/controllers/redis_web_manager/application_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/helpers/redis_web_manager/application_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/helpers/redis_web_manager/clients_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/helpers/redis_web_manager/dashboard_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/helpers/redis_web_manager/keys_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/config/routes.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/controllers/redis_web_manager/actions_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/controllers/redis_web_manager/clients_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/controllers/redis_web_manager/configuration_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/controllers/redis_web_manager/dashboard_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/controllers/redis_web_manager/information_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/app/controllers/redis_web_manager/keys_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/redis_web_manager-3d6d8d5d2e4b/config/routes.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/health_monitor-0ff0129dc182/config/routes.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/health_monitor-0ff0129dc182/config/routes.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/smart_listing-7278a01b581c/config/routes.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/smart_listing-7278a01b581c/app/helpers/smart_listing/helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/smart_listing-7278a01b581c/app/helpers/smart_listing/application_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/bundler/gems/smart_listing-7278a01b581c/config/routes.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/helpers/cable_ready/view_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/channels/cable_ready/stream.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/jobs/cable_ready/broadcast_job.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/models/concerns/extend_has_many.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/models/concerns/cable_ready/updatable.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/models/concerns/cable_ready/updatable/collection_updatable_callbacks.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/models/concerns/cable_ready/updatable/collections_registry.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/models/concerns/cable_ready/updatable/memory_cache_debounce_adapter.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/cable_ready-5.0.3/app/models/concerns/cable_ready/updatable/model_updatable_callbacks.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/controllers/devise/sessions_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/controllers/devise_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/helpers/devise_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/controllers/devise/passwords_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/controllers/devise/registrations_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/mailers/devise/mailer.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/controllers/devise/confirmations_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/controllers/devise/omniauth_callbacks_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-4.9.3/app/controllers/devise/unlocks_controller.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/models/concerns/turbo/broadcastable.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/controllers/turbo/streams/turbo_streams_tag_builder.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/controllers/turbo/frames/frame_request.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/controllers/turbo/native/navigation.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/helpers/turbo/drive_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/helpers/turbo/frames_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/helpers/turbo/includes_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/helpers/turbo/streams_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/helpers/turbo/streams/action_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/config/routes.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/channels/turbo/streams_channel.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/channels/turbo/streams/broadcasts.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/channels/turbo/streams/stream_name.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/controllers/turbo/native/navigation_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/models/turbo/streams/tag_builder.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/jobs/turbo/streams/action_broadcast_job.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/app/jobs/turbo/streams/broadcast_job.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/turbo-rails-1.5.0/config/routes.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test/cookie_jar.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test/utils.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test/methods.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test/uploaded_file.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rack-test-2.1.0/lib/rack/test/version.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-otp-0.6.0/app/controllers/devise_otp/devise/otp_credentials_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise-otp-0.6.0/app/controllers/devise_otp/devise/otp_tokens_controller.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/pghero-3.4.0/app/controllers/pg_hero/home_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/pghero-3.4.0/app/helpers/pg_hero/home_helper.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/pghero-3.4.0/config/routes.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/pghero-3.4.0/config/routes.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rails-pg-extras-5.3.1/app/controllers/rails_pg_extras/web/application_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rails-pg-extras-5.3.1/config/routes.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rails-pg-extras-5.3.1/app/controllers/rails_pg_extras/web/actions_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rails-pg-extras-5.3.1/app/controllers/rails_pg_extras/web/queries_controller.rb
[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/rails-pg-extras-5.3.1/config/routes.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/devise_masquerade-2.1.3/app/controllers/devise/masquerades_controller.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/sentry-rails-5.16.1/app/jobs/sentry/send_event_job.rb

[Bootsnap] miss /app/vendor/bundle/ruby/3.2.0/gems/punching_bag-0.7.0/app/models/punch.rb

Bootsnap cache is created when building Docker images in CI :

#################
# STAGE 8: test #
#################

# Fetch image
FROM release AS test

# Copy final build (app + gems + assets)
COPY --from=secure /app /app

# Setup user environment
ENV USER nonroot
ENV HOME /home/nonroot
ENV PATH /home/nonroot/.asdf/shims:/home/nonroot/.asdf/bin:$PATH
ENV BOOTSNAP_CACHE_DIR /tmp/cache

# Set working directory to /app
WORKDIR /app

RUN \
  # Reset runtime environment
  asdf reshim && \
  # Warm bootsnap cache
  bin/bootsnap precompile --gemfile app/ config/ lib/ vendor/engines/ \
  # Test that application boot to avoid the loading errors encountered with the mail gem
  # due to wrong files permissions for nonroot user (-rw-r-----   1 root  root   2.2K Dec  4 18:09 fields.rb)
  bin/rake concerto:config:test TESTING_BUILD=true RAILS_ENV=production && \
  rm -f /app/.env /app/log/*.log && \
  rm -rf /app/public/documents /app/private/documents

and set to readonlyin production:

nonroot@10382c5ab7db:/app$ ll /tmp/cache/bootsnap/
total 12
drwxr-xr-x 258 root root 4096 Jan 21 00:24 compile-cache-iseq
drwxr-xr-x 116 root root 4096 Jan 21 00:24 compile-cache-json
drwxr-xr-x 254 root root 4096 Jan 21 00:24 compile-cache-yaml
nonroot@10382c5ab7db:/app$ du -sh /tmp/cache/bootsnap/
84M /tmp/cache/bootsnap/

I'm thinking about adding all this paths to bin/bootsnap precompile but I'd like to know if there is a better of way to add this paths to bootsnap precompilation.

Thank you!

n-rodriguez commented 10 months ago

I came up with this solution :

in bin/bootsnap-paths:

#!/usr/bin/env ruby
# frozen_string_literal: true

require 'bundler'

{
  # rubygems
  'cable_ready'       => ['/app/'],
  'devise'            => ['/app/'],
  'turbo-rails'       => ['/app/', '/config/'],
  'rack-test'         => ['/lib/rack/test/'],
  'devise-otp'        => ['/app/'],
  'pghero'            => ['/app/', '/config/'],
  'rails-pg-extras'   => ['/app/', '/config/'],
  'devise_masquerade' => ['/app/'],
  'sentry-rails'      => ['/app/'],
  'punching_bag'      => ['/app/'],

  # github
  'crono'             => ['/app/', '/config/'],
  'redis_web_manager' => ['/app/', '/config/'],
  'smart_listing'     => ['/app/', '/config/'],
  'health_monitor'    => ['/config/'],
}.each do |gem_name, paths|
  spec = Bundler.load.specs.find{ |s| s.name == gem_name }
  paths.each do |path|
    puts "#{spec.full_gem_path}#{path}"
  end
end
bin/bootsnap precompile --gemfile app/ config/ lib/ vendor/engines/ $(bin/bootsnap-paths)

I no longer have any misses in container's logs.

casperisfine commented 10 months ago

Yes, this a consequence of Rails no longer adding autoloaded paths into the load path. We could probably look for app/ directories in gems, it wouldn't hurt, and is an easy fix.

But note that compiling 100% is nice but not strictly necessary.

n-rodriguez commented 10 months ago

But note that compiling 100% is nice but not strictly necessary.

Yes, but I want to avoid unecessary work at application boot.

casperisfine commented 10 months ago

Yes, but I want to avoid unecessary work at application boot.

Of course, just saying if e.g 0.2% of files aren't precompiled, it likely won't make a measurable difference.

Still, I'll see to add gem/app directories in the default compiled paths.

n-rodriguez commented 10 months ago

Still, I'll see to add gem/app directories in the default compiled paths.

So fast! Thank you!

casperisfine commented 10 months ago

So fast! Thank you!

Welcome. I probably won't cut a release just yet as I have some other changes I'm trying to make, but you can probably point your Gemfile at the repo.

n-rodriguez commented 10 months ago

but you can probably point your Gemfile at the repo.

Yep, done, merci!