Closed metaskills closed 3 years ago
I think I didn't add it because that cache is dependent on the $LOAD_PATH
, and that it's hard to predict an app $LOAD_PATH
without booting it.
But if you have some ideas or rough patches, feel free to submit them.
Another thing to note is that Lambda is a read-only file system. My method to build the load-path-cache
is to simply add something like this to the Dockerfile.
RUN bundle exec bootsnap precompile --gemfile app/ lib/ && \
LAMBY_BUILD=1 bundle exec ruby app.rb
Where app.rb
here does the typical requires for Rails boot, application, environment and defines the Lambda handler. This does generate a good looking load-path-cache
within the Lambda Container. Still testing benchmarks.
However, one big hurdle I had to overcome is that this file is opened and re-written on boot. I've done some general testing and it appears to not change. Can you share why this file is mutated and ends up @dirty
during commit_transaction
? I have to add code like this to the app.rb
to work around it while I test things further.
unless ENV['LAMBY_BUILD']
require 'bootsnap'
Bootsnap::LoadPathCache::Store.class_eval do
def commit_transaction ; end
end
end
to simply add something like this to the Dockerfile.
That's pretty much what we do too. Mostly to run rake assets:precompile
but it generates the load path cache.
Can you share why this file is mutated and ends up @dirty during commit_transaction?
Hum, that sounds weird. I'd have to dig that up. Not sure I'll have the time today though.
Does your app ever mutate $LOAD_PATH
?
Thanks. No, the application does not mutate $LOAD_PATH
Maybe you happen to have a backtrace relating to that?
Yup!
Init error when loading handler app.handler
{
"errorMessage": "Read-only file system @ rb_sysopen - /var/task/tmp/cache/bootsnap/load-path-cache.8.4481.tmp",
"errorType": "Init<Errno::EROFS>",
"stackTrace": [
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:88:in `initialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:88:in `open'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:88:in `dump_data'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:58:in `commit_transaction'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:49:in `block in transaction'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:45:in `synchronize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:45:in `transaction'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:139:in `push_paths_locked'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:122:in `block in reinitialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:116:in `synchronize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:116:in `reinitialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:16:in `initialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache.rb:44:in `new'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache.rb:44:in `setup'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap.rb:62:in `setup'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap.rb:108:in `default_setup'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/setup.rb:4:in `<top (required)>'",
"/var/task/app.rb:9:in `require'",
"/var/task/app.rb:9:in `<top (required)>'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'"
]
}
Hum, I'm afraid I'm not able to reproduce. On my test app the cache is consistent and isn't marked as dirty on subsequent boots.
If you are able to reproduce this in isolation I'm interested in digging this up, but at this stage I'm afraid this issue isn't actionable for me.
Would a different $LOAD_PATH
on the Docker container vs the Lambda runtime environment cause this? For example, I looked at both and saw these differences.
>> d - l
=> ["/var/lang/lib/ruby/gems/2.7.0/gems/bundler-2.2.16/lib"]
>> l - d
=> ["/var/task", "/var/runtime/lib", "/opt/ruby/lib", "/var/runtime/gems/bundler-2.2.16/lib"]
If not, where else could I look to reproduce this?
Would a different
$LOAD_PATH
on the Docker container vs the Lambda runtime environment cause this?
Yes. The content of $LOAD_PATH
is hashed to generate the key. If the $LOAD_PATH
isn't the same in both environments, the cache won't work.
won't work.
I meant a new entry will be written.
Thanks! Would you be open to a configuration change to avoid writing the new entry?
I'd be open to a patch that would ignore the error. That would be inline with recent changes in bootsnap where we make the cache best effort.
Thanks. Done deal. https://github.com/Shopify/bootsnap/pull/358
I think this was fixed.
I am working on optimizing some cold starts for Rails in AWS Lambda (https://lamby.custominktech.com) and wonder if you could describe why the precompilation command does not also add the
load-path-cache
. My goal is to have this in place for the Docker image deployed to Lambda so Rails boots faster.I've come up with a workaround to do just that. But hoping y'all can add some context as to how the load path cache is supposed to work.