Unable to start any service after upgrade - "ruby_executable_hooks": No such file or directory.

[s626ch]

Steps to reproduce the problem

I donʻt know if this is a fault of Chuckya or upstream Mastodon, but after an upgrade, none of my services will start. Iʻve tried upgraded Ruby to 3.3.5 using both rbenv and rvm, tried bundle exec rails yarn:install assets:clobber assets:precompile, tried following the upgrade instructions, tried gem install executable-hooks, absolutely nothing is working.

Expected behaviour

A clean upgrade?

Actual behaviour

Inability to start services.

Detailed description

The biggest piece of info I can really give is that my instance was a few months out of date (mostly because every time I do upgrade, I end up with big headaches like this that cause 12+ hours of downtime)

I run a small instance off of a Raspberry Pi 4b (8gb RAM), aarch64.

If I need to move this issue to upstream Mastodon, I gladly will.

Mastodon instance

No response

Mastodon version

No response

Technical details

If this is happening on your own Mastodon server, please fill out those:

• Ruby version: ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [aarch64-linux]
• Node.js version: v18.17.1

[s626ch]

Going to addendum that thereʻs likely commands I missed between each upgrade, if I can find them Iʻll attempt that.

[TheEssem]

What errors do the services output when attempting to start them?

[s626ch]

root@imhaater-pi:/home/mastodon/live# systemctl restart mastodon-*
root@imhaater-pi:/home/mastodon/live# journalctl -fu mastodon-sidekiq
-- Journal begins at Sun 2024-07-14 17:45:01 EDT. --
Oct 09 05:32:01 imhaater-pi systemd[1]: mastodon-sidekiq.service: Consumed 1.639s CPU time.
Oct 09 05:32:01 imhaater-pi systemd[1]: Started mastodon-sidekiq.
Oct 09 05:32:03 imhaater-pi bundle[3872794]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Oct 09 05:32:03 imhaater-pi systemd[1]: mastodon-sidekiq.service: Main process exited, code=exited, status=127/n/a
Oct 09 05:32:03 imhaater-pi systemd[1]: mastodon-sidekiq.service: Failed with result 'exit-code'.
Oct 09 05:32:03 imhaater-pi systemd[1]: mastodon-sidekiq.service: Consumed 1.609s CPU time.
Oct 09 05:32:03 imhaater-pi systemd[1]: mastodon-sidekiq.service: Scheduled restart job, restart counter is at 2.
Oct 09 05:32:03 imhaater-pi systemd[1]: Stopped mastodon-sidekiq.
Oct 09 05:32:03 imhaater-pi systemd[1]: mastodon-sidekiq.service: Consumed 1.609s CPU time.
Oct 09 05:32:04 imhaater-pi systemd[1]: Started mastodon-sidekiq.
Oct 09 05:32:05 imhaater-pi bundle[3872834]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Oct 09 05:32:05 imhaater-pi systemd[1]: mastodon-sidekiq.service: Main process exited, code=exited, status=127/n/a
Oct 09 05:32:05 imhaater-pi systemd[1]: mastodon-sidekiq.service: Failed with result 'exit-code'.
Oct 09 05:32:05 imhaater-pi systemd[1]: mastodon-sidekiq.service: Consumed 1.413s CPU time.
Oct 09 05:32:05 imhaater-pi systemd[1]: mastodon-sidekiq.service: Scheduled restart job, restart counter is at 3.
Oct 09 05:32:05 imhaater-pi systemd[1]: Stopped mastodon-sidekiq.
Oct 09 05:32:05 imhaater-pi systemd[1]: mastodon-sidekiq.service: Consumed 1.413s CPU time.
Oct 09 05:32:05 imhaater-pi systemd[1]: Started mastodon-sidekiq.

same thing as in issue title each service spouts the same ruby exec hooks error, above is sidekiq, below is web for example

root@imhaater-pi:/home/mastodon/live# journalctl -fu mastodon-web
-- Journal begins at Sun 2024-07-14 17:45:01 EDT. --
Oct 09 05:32:08 imhaater-pi bundle[3872924]: /usr/bin/env: ‘ruby_executable_hooks’: No such file or directory
Oct 09 05:32:08 imhaater-pi systemd[1]: mastodon-web.service: Main process exited, code=exited, status=127/n/a
Oct 09 05:32:08 imhaater-pi systemd[1]: mastodon-web.service: Failed with result 'exit-code'.
Oct 09 05:32:08 imhaater-pi systemd[1]: mastodon-web.service: Consumed 1.396s CPU time.
Oct 09 05:32:09 imhaater-pi systemd[1]: mastodon-web.service: Scheduled restart job, restart counter is at 5.
Oct 09 05:32:09 imhaater-pi systemd[1]: Stopped mastodon-web.
Oct 09 05:32:09 imhaater-pi systemd[1]: mastodon-web.service: Consumed 1.396s CPU time.
Oct 09 05:32:09 imhaater-pi systemd[1]: mastodon-web.service: Start request repeated too quickly.
Oct 09 05:32:09 imhaater-pi systemd[1]: mastodon-web.service: Failed with result 'exit-code'.
Oct 09 05:32:09 imhaater-pi systemd[1]: Failed to start mastodon-web.

ruby_executable_hooks is present under /home/mastodon/.rvm/gems/ruby-3.3.5@mastodon/, in both bin, and wrappers

[TheEssem]

Mastodon's docs recommend using rbenv with the ruby-build plugin; try that instead of rvm and see if it works.

[s626ch]

Iʻll try reinstalling 3.3.5 with rbenv, ruby-build is already present. ...and nuking rvm.

[s626ch]

well, getting...somewhere this is located within /usr/local/bin/

[s626ch]

i have absolutely no fucking idea what itʻs trying to do

like i said, nuked rvm and i reinstalled 3.3.5 with rbenv, and the plugin was already present, iʻm not too sure whatʻs going wrong