1.0.6 update not working with Ruby version error

[ljwrites]

Steps to reproduce the problem

1. Attempt to upgrade Hometown from 1.0.5 to 1.0.6 per instructions https://github.com/hometown-fork/hometown/releases/tag/v1.0.6%2B3.5.2 , resulting in the error that ruby wasn't the required version, 3.0.3 (version was 2.7.2)
2. Upgrade ruby to 3.0.0p0 with rvm, but trying to install 3.0.3 as instructed ends with "There has been an error fetching the ruby interpreter. Halting the installation."
3. Trying to precompile the assets after git checkout as the mastodon user results in the version error; precompile as root user, which proceeds without error.
4. The mastodon-sidekiq and mastodon-web services fail to restart, with the message that "rbenv: version `3.0.3' is not installed (set by /home/mastodon/live/.ruby-version)"
5. Change the text in .ruby-version to 3.0.0p0 (actual installed version), same thing happens with the new version number, "rbenv: version `3.0.0p0' is not installed (set by /home/mastodon/live/.ruby-version)." Same when .ruby-version is changed to 3.0.0.
6. Remove the .ruby-version file altogether, restart the Mastodon services, which results in errors relating to ruby 2.7.2, see mastodon-sidekiq.log which is the output of journalctl -u mastodon-sidekiq

Expected behaviour

All the Mastodon services would have restarted and the instance would work with the updated version.

Actual behaviour

Two of the services would not restart and the instance does not work.

Specifications

v1.0.6+3.5.2

[ljwrites]

Subsequent update: I upgraded and switched Ruby to version 3.0.3, with the output of

ruby -v

in the mastodon/live directory as the mastodon user being:

ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]

Nevertheless, trying to precompile as mastodon user in the live directory still results in the message:

rbenv: version `3.0.3' is not installed (set by /home/mastodon/live/.ruby-version)

Changing the content of .ruby-version to 3.0.0p157 results in the same error with the new version number:

rbenv: version `3.0.3p157' is not installed (set by /home/mastodon/live/.ruby-version)

Attempts to restart the services throw the same version error, this time with 3.0.3. See attached the output of journalctl -u mastodon-sidekiq :

mastodon-sidekiq.log mastodon-web.log

(The second log was taken after .ruby-version was changed to 3.0.3p157 which is why the version numbers that appear in them differ, but it doesn't seem to make a difference either way.)

[dariusk]

This will sound silly but: have you rebooted your server at any point in all this? If not, can you change .ruby-version back to its original value, reboot your server, and see if you can precompile?

[ljwrites]

I changed .ruby-version back to 3.0.3, rebooted with `sudo reboot' (I was immediately cut off of the host and could ssh back in several minutes later, confirming the reboot took place), and the same issue persists:

mastodon@ubuntu-4gb-nbg1-2:~/live$ cat .ruby-version
3.0.3
mastodon@ubuntu-4gb-nbg1-2:~/live$ ruby -v
ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]
mastodon@ubuntu-4gb-nbg1-2:~/live$ bundle exec rails assets:precompile
rbenv: version `3.0.3' is not installed (set by /home/mastodon/live/.ruby-version)

Updated service restart logs, but they just throw the same Ruby version error message: mastodon-sidekiq.log mastodon-web.log

[FiXato]

I'm wondering if the binstub for bundle might have some incorrect paths? Is there a ./bin/bundle perhaps in your project, and if so, could you show the cat for it? (Perhaps use which bundle to see which bundle command is being used)

[ljwrites]

@FiXato Thanks for taking an interest in this, I'm extremely confused by all this Ruby stuff and Mastodon was the only context in which I've touched it. Here's the output:

mastodon@ubuntu-4gb-nbg1-2:~/live/bin$ cat bundle
#!/usr/bin/env ruby
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)
load Gem.bin_path('bundler', 'bundle')
mastodon@ubuntu-4gb-nbg1-2:~/live/bin$ which bundle
/usr/local/rvm/rubies/ruby-3.0.3/bin/bundle

[dariusk]

How much RAM is on your server? Could it be related to this issue where the ruby build "completes" but actually fails due to lack of RAM?

https://github.com/mastodon/mastodon/issues/18421

If you have 1gb RAM this is probably the problem, I recommend at least 2gb generally.

[FiXato]

rbenv: version `3.0.3' is not installed (set by /home/mastodon/live/.ruby-version)

and

/usr/local/rvm/rubies/ruby-3.0.3/bin/bundle

Seems to indicate that both the rvm and rbenv ruby environment/version managers are installed (or have both at some point been installed, and one of them not removed completely), and which now seem to be at odds with each other.

The bundle command is being loaded from the rvm install, while LJ has been installing a newer ruby through rbenv.

I suspect that the /usr/local/rvm or perhaps /usr/local/rvm/rubies/ruby-3.0.3/bin/ directory is still on PATH with a higher priority than rbenv. At this point LJ probably first wants to decide whether she wants to use rvm or rbenv going forward, and remove any traces of the other from her environment before trying to run the updates again.

I've already provided LJ with some more pointers via mastodon DM before posting here (didn't feel like getting up to use my laptop at the time), but figured I'd add the relevant info here too now that I'm at my laptop.

[ljwrites]

@dariusk the output of free -h shows free as 2.7Gi so I don't think it's a memory issue--the Mastodon instance being down freed up a lot of ram probably lolsob.

@FiXato You were right, there were both rbenv and rvm on the system and I've removed rvm. The output of $PATH is now:

mastodon@ubuntu-4gb-nbg1-2:~$ echo $PATH
/home/mastodon/.rbenv/shims:/home/mastodon/.rbenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

The ruby version is now correct:

mastodon@ubuntu-4gb-nbg1-2:~/live$ ruby -v
ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]
mastodon@ubuntu-4gb-nbg1-2:~/live$ rbenv versions
  system
  2.7.2
* 3.0.3 (set by /home/mastodon/live/.ruby-version)

Trying to precompile now results in a whole new error message, which is genuinely kind of exciting!

mastodon@ubuntu-4gb-nbg1-2:~/live$ bundle exec rails assets:precompile
/usr/bin/env: ‘ruby_executable_hooks’: No such file or directory

So I definitely got rid of something necessary when I purged rvm, which is probably better in the long run than the confusion of keeping both rbenv and rvm around. I looked up this issue and tried both

gem install --user-install executable-hooks

and

gem install rubygems-update
update_rubygems

These were done as root since mastodon doesn't have sudo authority, maybe that matters?

Neither operation threw an error, but when I try to precompile as mastodon in live I still get the same error as above. I think I'm close now if I can rebuild the ruby environment based on rbenv only, any ideas for resolving this?

[FiXato]

Can you check rbenv shims? If ruby_executable_hooks isn't listed there, try running rbenv rehash to refresh the shims used by rbenv, followed by checking rbenv shims again.

If that still doesn't list it, you could try rbenv exec gem install --user-install executable-hooks before rehashing and checking the shims again.

[ljwrites]

@FiXato I was able to install executable-hooks with your instructions, thank you! I'm now getting other errors on precompile that I was getting prior to the rvm purge, but as this is clearly a Ruby setup issue and not a Hometown bug, I'll close the issue and contact you directly for help :)

[dariusk]

So so so glad to hear this. Gold star for @FiXato !!!