Using next_rails on Heroku

obromios commented 2 years ago

IMPORTANT: please make sure you ask yourself all intro questions and fill all sections of the template.

Before we start...:

[x ] I checked the documentation and found no answer
[x ] I checked to make sure that this issue has not already been filed
[ x] I'm reporting the issue to the correct repository (for multi-repository projects)

Version, Branch, or Commit: next_rails 1.2.0 What branch/commit/version of "next_rails" you are using? master (I am using the latest version of gem) Expected behavior:

This fastruby post, seems to imply the next_rails features can be used on staging by setting BUNDLE_GEMFILE=Gemfile.next

In development and test environments I used next_rails to upgrade my app from rails 6.0.5.1 to 6.1.6.1. The next_rails gem was very useful, and all my tests passed and the server ran in 6.0 and 6.1. I was expecting that when I deployed the app to the heroku staging server, that the server would work in 6.0 and when I set BUNDLE_GEMFILE=Gemfile.next using heroku config:set that the server would work with 6.1. Actual behavior:

In order to debug the problem, I put the following code in the Gemfile puts "next?: #{next?}, file: #{File.basename(__FILE__)}" which allowed me to see which state the app was meant to be.

The app deployed to staging, and the server worked with 6.0. I then set BUNDLE_GEMFILE=Gemfile.next and the app crashed with the following error messages.

2022-08-21T04:38:10.315494+00:00 heroku[web.1]: Starting process with command `bundle exec puma -p 42091 -C ./config/puma.rb`
2022-08-21T04:38:10.430822+00:00 heroku[worker.1]: Starting process with command `bundle exec rails jobs:work`
2022-08-21T04:38:11.011685+00:00 heroku[worker.1]: State changed from starting to up
2022-08-21T04:38:11.531042+00:00 app[worker.1]: bundler: failed to load command: rails (/app/vendor/bundle/ruby/2.7.0/bin/rails)
2022-08-21T04:38:11.531229+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/definition.rb:481:in `materialize': Could not find rails-6.1.6.1, actioncable-6.1.6.1, actionmailbox-6.1.6.1, actionmailer-6.1.6.1, actionpack-6.1.6.1, actiontext-6.1.6.1, actionview-6.1.6.1, activejob-6.1.6.1, activemodel-6.1.6.1, activerecord-6.1.6.1, activestorage-6.1.6.1, activesupport-6.1.6.1, railties-6.1.6.1, tzinfo-2.0.5 in any of the sources (Bundler::GemNotFound)
2022-08-21T04:38:11.531255+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/definition.rb:190:in `specs'
2022-08-21T04:38:11.531258+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/definition.rb:238:in `specs_for'
2022-08-21T04:38:11.531259+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/runtime.rb:18:in `setup'
2022-08-21T04:38:11.531261+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler.rb:151:in `setup'
2022-08-21T04:38:11.531263+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/setup.rb:20:in `block in <top (required)>'
2022-08-21T04:38:11.531272+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/ui/shell.rb:136:in `with_level'
2022-08-21T04:38:11.531274+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/ui/shell.rb:88:in `silence'
2022-08-21T04:38:11.531276+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/setup.rb:20:in `<top (required)>'
2022-08-21T04:38:11.531278+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/cli/exec.rb:56:in `require_relative'
2022-08-21T04:38:11.531287+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/cli/exec.rb:56:in `kernel_load'
2022-08-21T04:38:11.531289+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/cli/exec.rb:23:in `run'
2022-08-21T04:38:11.531291+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/cli.rb:483:in `exec'
2022-08-21T04:38:11.531293+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
2022-08-21T04:38:11.531303+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
2022-08-21T04:38:11.531305+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
2022-08-21T04:38:11.531306+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/cli.rb:31:in `dispatch'
2022-08-21T04:38:11.531315+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
2022-08-21T04:38:11.531317+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/cli.rb:25:in `start'
2022-08-21T04:38:11.531319+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/exe/bundle:48:in `block in <top (required)>'
2022-08-21T04:38:11.531321+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
2022-08-21T04:38:11.531323+00:00 app[worker.1]: from /app/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/exe/bundle:36:in `<top (required)>'
2022-08-21T04:38:11.531333+00:00 app[worker.1]: from /app/vendor/bundle/bin/bundle:113:in `load'
2022-08-21T04:38:11.531335+00:00 app[worker.1]: from /app/vendor/bundle/bin/bundle:113:in `<main>'
2022-08-21T04:38:11.532046+00:00 app[worker.1]: next?: true, file: Gemfile.next
2022-08-21T04:38:11.753017+00:00 heroku[worker.1]: State changed from up to crashed
2022-08-21T04:38:11.760087+00:00 heroku[worker.1]: State changed from crashed to starting
2022-08-21T04:38:11.929927+00:00 heroku[web.1]: Process exited with status 1

Note that at the fourth last line in the error message above, the debug code shows Gemfile.next is being used.

If I do heroku run bash -r staging and then do next bundle install, it installs successfully 6.1. If I then do next rails s then the server starts OK, so I wonder if I need to add some preparation steps to the Procfile, such as web: bundle install? Steps to reproduce:

How do I achieve this behavior? Use the following format to provide a step-by-step guide:

Produce a dual booting app that has a working server for both versions of rails
Deploy app to heroku, and check the server is working in the earlier version of rails.
heroku run rake config:set BUNDLE_GEMFILE=Gemfile.next -r staging4.
check the server is working in the later version of rails. Context and environment:

Provide any relevant information about your setup (Customize the list accordingly based on what info is relevant to this project)

Heroku stack-18
rails 6.1.5.1
rails 6.1.6.1
ruby 2.7.6

I will abide by the code of conduct

obromios commented 2 years ago

I tried adding web: bundle install to the Procfile, but it seemed to be ignored so did not help.

I tried setting 'BUNDLE_GEMFILE prior to deploying i.e

 heroku config:set BUNDLE_GEMFILE=Gemfile.next -r staging
 git push staging master

But this did not work, here is the relevant part of the log temp_next.txt

I inadvertently added the bug tag to this issue. I do not see this issue as a bug, but more an effort to provide more documentation on how to use the gem. Also to that effect, I notice there are a couple excellent articles by fastruby staff, such as this one. Perhaps references to these posts should be in the README, or even putting some of the content of the articles into the README? I would be prepared to do a pull request along those lines.

mateusdeap commented 2 years ago

@obromios It's odd that you should be facing this issue, though. I'll try to replicate and see if indeed there isn't a need for a fix, because as far as I know, you seem to be using the gem correctly. @bronzdoc or @arielj do you happen to have any input on this? I'm not that familiar with Heroku.

mateusdeap commented 2 years ago

@obromios Just so I'm sure I fully understood you:

You want to be able to just swap the env var in the running server and, once it restarts, that it will be on the next rails version? Because if that's the case, next_rails doesn't allow that. It merely allows you to have 2 set of dependencies for the same app so that you can run it in either version.

However, when using this in a remote server, unless you manually install the next version's dependencies, as you did in the first message, it will break, because it needs to install the necessary dependencies. next_rails doesn't automate that.

Now, I've setup a test app and am still going to test this in my heroku account, since I need to learn how to do it in heroku myself 😆 but, basically, what needs to be done is, if you want to deploy the next version of rails, the idea is you configure the env var so that heroku runs bundle install with that var set so that it installs the new rails version. Is that the purpose of the Procfile?

It has been suggested to me to suggest to you to try this in the Procfile:

release: BUNDLE_GEMFILE=Gemfile.next
bundle install

Let me know if that works

Just to reiterate: next_rails won't allow you to hot swap versions. If you swap the env var, the app needs to be rebuilt from scratch or you need to manually update the dependencies.

obromios commented 2 years ago

I have now moved on to doing a dual boot between version 6.1 and 7.0.

I tried the following in the procfile

release:  BUNDLE_GEMFILE=Gemfile.next
bundle install
web: bundle exec puma -p $PORT -C ./config/puma.rb
worker: bundle exec rails jobs:work

I did NOT set config with heroku config:set BUNDLE_GEMFILE=Gemfile.next.

The app deployed OK, and the server did start but the Rails.version was 6.1 not 7.0.

I hope to find more time to experiment on the weekend, or maybe you will make progress with your test app.

obromios commented 2 years ago

I have tried preceeding everything with BUNDLE_GEMFILE=Gemfile.next in the Procfile

release:  BUNDLE_GEMFILE=Gemfile.next
release: BUNDLE_GEMFILE=Gemfile.next bundle install
web: BUNDLE_GEMFILE=Gemfile.next bundle install
web: BUNDLE_GEMFILE=Gemfile.next bundle exec puma -p $PORT -C ./config/puma.rb
worker: BUNDLE_GEMFILE=Gemfile.next bundle exec rails jobs:work

but this failed to start, giving this exception.

2022-09-03T05:38:45.787816+00:00 heroku[web.1]: Starting process with command `BUNDLE_GEMFILE=Gemfile.next bundle exec puma -p 43843 -C ./config/puma.rb`
2022-09-03T05:38:47.120629+00:00 heroku[web.1]: Process exited with status 1
2022-09-03T05:38:46.996110+00:00 app[web.1]: bundler: failed to load command: puma (/app/vendor/bundle/ruby/2.7.0/bin/puma)

I am also wondering the following. If the eventual solution requires altering the Procfile for the next version, then your a version of your app that has to be manually changed back to deploy the earlier version. This is not really in the spirit of the next_rails magic.

So I am wondering in that case maybe you are just better deploying a branch where the next version is fully implemented and then merge that branch with master once you have it working on staging.

That is what I did when hit this problem upgrading from 6.1 to 6.2.

arielj commented 2 years ago

I don't think having 2 release and 2 web commands is valid. also the first release: BUNDLE_GEMFILE=Gemfile.next is not running a command or anything.

I have a question for you @obromios, I'm trying to understand what you are trying to achieve. Do you want to have one single Heroku up to run both Rails versions AT THE SAME TIME or you want the Heroku app to be configured so you can run one app or the other? I'm confused on how you expect this to work to suggest something specific

As general tips:

You can't run both apps at the same time
You'll have to stop one and start the other one when needed (you should be able to do something like: set the correct BUNDLE_GEMFILE env variable > redeploy)
You have to ensure that bundle install is run for both Rails versions during build, or you'll have missing dependencies (I'd recommend checking how the ruby build pack works for heroku, it only runs bundle install once but maybe there's a way to configure it to run both)

I'm not sure what you mean about the spirit of the next_rails magic in the last comment, I think you are expecting the gem to work in a way it's not supposed to? would be nice for you to explain what you want to achieve (without going into the details of what you tried so far)

obromios commented 2 years ago

@arielj, that is a good question.

By the 'magic of next_rails', I mean I can have on the same branch a functioning version of rails 6 and rails 7. I can ensure any changes I make to the rails 6 version does not affect the rails 7 version by doing next bundle install, next rspec spec and next rails s. This means that I do not have to be constantly maintaining a rails 7 branch whilst continuing development on the rails 6 branch.

So ideally what I would like to do is next git push staging master, and voila, the rails 7 app would on heroku to test. I understand that because the git command does not consult the gemfile that this is not possible, but hopefully gives you and idea of what would like to do.

This fastruby post states that You can even run your application in staging using the next version of Rails. Simply make sure that you set this environment variable: BUNDLE_GEMFILE but as explained above, I have tried two ways of interpreting that and neither seemed to work.

The way I have resolved this is to have a separate branch which is just for deploying to heroku, in which the gemfile uses the rails 7 gem (i.e. it does not respond to next). I only need to merge this with master at the time I want to check the rail 7 version on heroku. This pretty good, but it is not magic.

arielj commented 2 years ago

What I understand is that you are trying to tell heroku to use one version or the other depending on how you push the code to the git repo? but that's not a feature of next_rails (nor a feature next_rails could ever have, since the communication between git and heroku is not something we can change).

I think it makes more sense to have 2 apps, both listening to the same git repo updates, one with BUNDLE_GEMFILE=Gemfile and one with BUNDLE_GEMEFILE=Gemfile.next in the settings. That way you'll always have both the current and the next rails versions running at the same time to test whatever you need and you just use 2 urls to open each.

Another option to use only one app at a time is for you to make a script that will change the BUNDLE_GEMFILE env variable before pushing. For that to work you'll need to make heroku bundle both versions of the gems, but this is again not a feature of next_rails since the buildpack is not controlled by us, it's here https://github.com/heroku/heroku-buildpack-ruby/tree/main/bin.

For heroku to bundle both gemfiles during build, you can fork that buildpack to run both BUNDLE_GEMFILE=Gemfile bundle install and BUNDLE_GEMFILE=Gemfile.next bundle install so it bundles everything and then tell heroku to use your custom buildpack instead of the standard ruby buildpack.

This won't solve the issue of wanting to switch between versions (you'll still need to change the BUNDLE_GEMFILE env variable before re-deploying for it to start the desired rails version).

arielj commented 2 years ago

Just to clarify, if you want to run ONLY ONE version of rails in a single Heroku app, you can set the BUNDLE_GEMFILE env variable in the settings and it should run one of the apps depending if the value is Gemfile or Gemfile.next. That should work fine (and you don't even need to have next_rails installed or prefix the commands with next ...). Make sure you have that env variable and deploy the app and it should build the app with the correct Gemfile and start the correct version.

obromios commented 2 years ago

@arielj, I do only want to run version of rails in a single app. What you describe is the next_rails magic I am looking for. Your instructions seem clear, but somehow I seem to be misinterpreting.

In my opening post to this issue, I thought I did try deploying the app then changing the BUNDLE_GEMFILE to Gemfile.next but it crashed as described in that post. In the second post in this issue, I did try setting BUNDLE_GEMFILE to Gemfile.next before I deployed the app, and that also crashed, as described in the second post.

Since making those posts, I have progressed from upgrading from rails 6.0 to 6.1, to upgrading from 6.1 to 7.0, so I thought I would try again. If I set BUNDLE_GEMFILE to Gemfile.next before I deployed the app, I get the same error as before (as described in the second post). However if I deploy with BUNDLE_GEMFILE unset, and then set BUNDLE_GEMFILE to Gemfile.next after the deployment, I do get a different error messages from the first post;temp_changing_bundle_gemfile_after_deploy.txt. Maybe this new error message might shed more light.

@mateusdeap, in your post on this issue, you said you would try to replicate the issue. Have you had a chance to do this?

arielj commented 2 years ago

Would be nice to see the build log when a deploy is done with the BUNDLE_GEMFILE=Gemfile.next config in place. Maybe the buildpack is overriding any BUNDLE_GEMFILE configuration during build? (or maybe it's set after the build?).

arielj commented 2 years ago

Hmmm I wonder if this is a known issue https://github.com/heroku/heroku-buildpack-ruby/issues/407, maybe the buildpack is overriding anything that's set there and falling back to the original Gemfile?

obromios commented 2 years ago

@arielj, here is log for the following steps 1) unset BUNDLE_GEMFILE 2) push to staging 3) set BUNDLE_GEMFILE=Gemfile.next temp_deploy_then_change_bundle_gem.log

arielj commented 2 years ago

@obromios looks like your Gemfile.next.lock is out of sync with your Gemfile.next, do you have changes to commit if you run BUNDLE_GEMFILE=Gemfile.next bundle install in your dev computer?

I would swap these steps just in case:

set BUNDLE_GEMFILE=Gemfile.next
push to staging

you want the env variable during the build

obromios commented 2 years ago

@arielj, I did not have any changes to commit, but something is out of sync, and it seems to have to do with setting the platform. For the following I have set BUNDLE_GEMFILE=Gemfile.next on staging.

If I rm Gemfile.next.lock and then do next bundle install then commit changes and push to staging, I do not get the ensure_equivalent_gemfile_and_lockfile error, but I do get the following error

/tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/helpers/rake_runner.rb:100:in `load_rake_tasks!': Could not detect rake tasks (LanguagePack::Helpers::RakeRunner::CannotLoadRakefileError)
remote: ensure you can run `$ bundle exec rake -P` against your app
remote: and using the production group of your Gemfile.
remote: /tmp/build_c3d1462a/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/definition.rb:433:in `validate_platforms!': Your bundle only supports platforms ["x86_64-darwin-21"] but your local platform is x86_64-linux. Add the current platform to the lockfile with (Bundler::ProductionError)
remote: `bundle lock --add-platform x86_64-linux` and try again.

Now if I do bundle lock --add-platform x86_64-linux it does add the platform to Gemfile.next.lock. Now if I commit this change and push to staging, I get the ensure_equivalent_gemfile_and_lockfile error again.

remote: /tmp/codon/tmp/buildpacks/50d5eddf222a9b7326028041d4e6509f915ccf2c/lib/language_pack/helpers/rake_runner.rb:100:in `load_rake_tasks!': Could not detect rake tasks (LanguagePack::Helpers::RakeRunner::CannotLoadRakefileError)
remote: ensure you can run `$ bundle exec rake -P` against your app
remote: and using the production group of your Gemfile.
remote: /tmp/build_bc84792c/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.10/lib/bundler/definition.rb:397:in `ensure_equivalent_gemfile_and_lockfile': You are trying to install in deployment mode after changing (Bundler::ProductionError)
remote: your Gemfile. Run `bundle install` elsewhere and add the
remote: updated Gemfile.next.lock to version control.

If I do next bundle install then there is no change to Gemfile.next.lock, and if I rm Gemfile.next.lock and next bundle install the platform setting disappears and I am back to needing to add it back in.

Here are build logs for the two cases described above. temp_build_prior_to_setting_platform.log

temp_build_after_setting_platform.log

arielj commented 2 years ago

I was doing some tests, it looks like the problem is Heroku's buildpack for ruby, it ignores the Gemfile env variable and always bundles only the original Gemfile.lock, but then it tries to run rake using the env variable so it breaks

I don't think we can fix that in our side in next_rails, I think an update (or a fork) of the original buildpack is needed to add support for that. We can't control what Heroku does in that state of the build process.

arielj commented 2 years ago

I mean, the issue happens without even using the next_rails gem in the app, heroku seems to be ignoring bundler's BUNDLE_GEMFILE env variable.

obromios commented 2 years ago

Thank you, @arielj for your assistance, at least I can let this go for the time being. I have posted a message on the buildpack issue indicating I think this is a bug in the Heroku ruby buildpack.

My current workflow is as follows a) use the next_rails gem as documented for development server and development tests b) Once ready for testing deployment, create a new branch, with the only change is for the Gemfile to go from

...
if next?
  gem 'rails', '~> 7.0.3.1'
  gem "sprockets-rails"
else
  gem 'rails', '~> 6.1.6.1'
end

to

...
 gem 'rails', '~> 7.0.3.1'

And deploy that branch to staging. This branch only needs to be merged with master when you want to test out staging.

arielj commented 2 years ago

I'll close this since the issue happens even without using the next_rails gem so it's not a next_rails bug, you can always come and comment as a follow up of course.

JuanVqz commented 8 months ago

Just as a reference if somebody else came here looking for a solution on how to run your next rails application on Heroku

https://www.fastruby.io/blog/how-to-run-multiple-versions-of-rails-on-heroku.html

fastruby / next_rails

Using next_rails on Heroku #71