search

muxinc / mux-ruby

Official Mux API wrapper for ruby projects, supporting both Mux Data and Mux Video.
MIT License
51 stars 14 forks source link

cannot run rails console on in production environments when mux_ruby is installed #36

Closed patmisch closed 3 years ago

patmisch commented 3 years ago

After deploying our mux_ruby implementation I can no longer run rails console. Removing the mux_ruby gem resolves this, so I'm thinking it has something to do with this gem.

Any ideas?

Here is the error output:


Traceback (most recent call last):
        92: from /workspace/bin/rails:4:in `<main>'
        91: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
        90: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
        89: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
        88: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        87: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        86: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
        85: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        84: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        83: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/railties-6.1.3.1/lib/rails/commands.rb:18:in `<main>'
        82: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/railties-6.1.3.1/lib/rails/command.rb:50:in `invoke'
        81: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/railties-6.1.3.1/lib/rails/command/base.rb:69:in `perform'
        80: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
        79: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
        78: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
        77: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/railties-6.1.3.1/lib/rails/commands/console/console_command.rb:101:in `perform'
        76: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/railties-6.1.3.1/lib/rails/command/actions.rb:14:in `require_application_and_environment!'
        75: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/railties-6.1.3.1/lib/rails/command/actions.rb:22:in `require_application!'
        74: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
        73: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
        72: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
        71: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        70: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        69: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
        68: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        67: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        66: from /workspace/config/application.rb:9:in `<main>'
        65: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bundler-2.1.4/lib/bundler.rb:174:in `require'
        64: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `require'
        63: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `each'
        62: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `block in require'
        61: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `each'
        60: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:74:in `block (2 levels) in require'
        59: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        58: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        57: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
        56: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        55: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        54: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/mux_ruby-1.9.0/lib/mux_ruby.rb:7:in `<main>'
        53: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
        52: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
        51: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
        50: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        49: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        48: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
        47: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        46: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        45: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/mux_ruby-1.9.0/lib/mux_ruby/api_client.rb:10:in `<main>'
        44: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
        43: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
        42: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
        41: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        40: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        39: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
        38: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        37: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        36: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/typhoeus-1.4.0/lib/typhoeus.rb:2:in `<main>'
        35: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
        34: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
        33: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
        32: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        31: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        30: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
        29: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        28: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        27: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon.rb:16:in `<main>'
        26: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
        25: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
        24: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
        23: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        22: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        21: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
        20: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
        19: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
        18: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon/curl.rb:9:in `<main>'
        17: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon/curl.rb:14:in `<module:Ethon>'
        16: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon/curl.rb:29:in `<module:Curl>'
        15: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `require'
        14: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:299:in `load_dependency'
        13: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.1/lib/active_support/dependencies.rb:332:in `block in require'
        12: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
        11: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
        10: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
         9: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
         8: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
         7: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon/curls/classes.rb:2:in `<main>'
         6: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon/curls/classes.rb:3:in `<module:Ethon>'
         5: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon/curls/classes.rb:27:in `<module:Curl>'
         4: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ethon-0.14.0/lib/ethon/curls/classes.rb:37:in `<class:FDSet>'
         3: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ffi-1.15.0/lib/ffi/struct.rb:220:in `layout'
         2: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ffi-1.15.0/lib/ffi/struct_layout_builder.rb:171:in `build'
         1: from /layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ffi-1.15.0/lib/ffi/struct_layout_builder.rb:171:in `new'
/layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ffi-1.15.0/lib/ffi/struct_layout_builder.rb:171:in `initialize': integer 4294967288 too big to convert to `int' (RangeError)
dylanjha commented 3 years ago

Hi @patmisch,

/layers/heroku_ruby/gems/vendor/bundle/ruby/2.7.0/gems/ffi-1.15.0/lib/ffi/struct_layout_builder.rb:171:in initialize': integer 4294967288 too big to convert toint' (RangeError)

This looks like a problem with bigint vs. int from inside the ffi gem

It appears you're running:

I'm curious why you're only running into this in your production env -- can you check your local ruby version to make sure it matches and see if you can reproduce the issue locally?

If that's not working can you try generating a new rails app that reproduces the issue when running the console on Heroku?

patmisch commented 3 years ago

@dylanjha Thanks for the quick response.

Our local and production versions definitely do match. And no, I cannot reproduce this locally.

I went ahead and generated a fresh rails app and got the same error after adding the mux_ruby gem.

FWIW we actually aren't on Heroku. Our app(s) run on Digital Ocean's App Platform, which utilizes heroku buildpacks. If it's helpful, I believe DO is using Kubernetes to power App Platform.

I'm going to experiment a little with installing mux_ruby dependencies independently to see if we can pinpoint it a little better.

Edit: The dependency in question is obviously ffi, duh.

dylanjha commented 3 years ago

I went ahead and generated a fresh rails app and got the same error after adding the mux_ruby gem.

Can you share that application & instructions to reproduce?

patmisch commented 3 years ago

Repo here: https://github.com/passageinc/mux_test

If you have access to a Digital Ocean account, you can utilize the "Deploy To DO" button provided in the readme.

From the App Platform dashboard, click the "Console" tab. You'll be able to run rails console and see the resulting error.

https://github.com/passageinc/mux_test/tree/without-mux Exact same repo, except on a branch that has mux_ruby uninstalled. Repeat same steps and find rails console working correctly.

dylanjha commented 3 years ago

I was able to repo -- thanks for that.

One thing to note is that the ffi gem installs with native extensions against the OS

[mux-test] [2021-07-13 21:26:32]        Fetching ffi 1.15.3
[mux-test] [2021-07-13 21:26:33]        Installing ffi 1.15.3 with native extensions

That explains why you couldn't repro the problem locally. It must be specific to the OS on the DO box (ubuntu of some sort, I imagine?) -- I couldn't tell from the deploy.template.yaml

1.15.3 is the latest version of ffi, so I'm not sure what else we can do with that dependency. I also saw this question in the DO forum so it looks like this isn't a totally isolated case. I have a feeling you're the one who commented there a few hours ago.

Next things I would try:

  1. Bump the ruby version to 2.7.4 (or heck, even 3.0.2). I think different ruby versions will change the way native extensions are installed, so that's worth a shot
  2. See if you can over-ride the default ubuntu version that DO installs. I looked through the docs for deploy.template.yaml, I'm unclear about how to go about that, but there must be a way
  3. Instead of over-riding the default ubuntu version, there's probably an apt-get install command you can run on the OS before running bundle install (like apt-get install libffi-dev, or something -- this is pretty common when there's a problem building native extensions). You might be able to specify something like this in the deploy.template.yaml file.

Could you also open up a support ticket with DO about this and see if they can help?

patmisch commented 3 years ago

Digital Ocean was able to get to the bottom of this...

The problem wasn't actually with ffi, that is just the furthest down the stack ruby is able to trace.

Turns out the issue was with the ethon gem. Related to https://github.com/typhoeus/ethon/issues/182

I'm able to work around it by calling ulimit -n 65535 before executing rails console

mein-beer-hu commented 2 years ago

After along a year, I am here, facing this issue on digitalocean, going to give the ulimit -n 65535 solution a try. thanks