Fails due to using user ruby version instead of project ruby version

My setup

OS: MacOS
managing ruby versions with chruby
Rails 6.1 project running ruby 3.0.6, the workspace has a .ruby-version file with contents 3.0.6, Gemfile also mentions the ruby version
added debug to my Gemfile
no other problems with running & testing the rails project

My problem

I have installed debug gem and the vs code extension. I have set up launch.json so that when I click Server: Start it will start the rails server. I tried a few different things in this file before I figured out the problem isn't with this file.

{
        "name": "Server: Start",
        "type": "rdbg",
        "request": "launch",
        "rdbgPath": "bundle exec rdbg",
        "cwd": "${workspaceRoot}",
        "script": "${workspaceRoot}/bin/rails",
        "args": [
            "s"
        ]
    },

When I click Server: Start from the Run & Debug section, I get the following error:

Couldn't start debug session. The debuggee process exited with code 1

My investigations

I found in the terminal that it was running this command:

bin/zsh -lic bundle exec rdbg --command --open --stop-at-load -- bundle exec ruby /Users/reedstonefood/my_project/bin/rails s

So, I pasted that into terminal and ran it myself which found the code was failing at the ruby level. It gave me a huge output, the pertinent parts of which I will paste here:

LoadError: dlopen(/Users/reedstonefood/.gem/ruby/3.0.6/gems/date-3.3.4/lib/date_core.bundle, 0x0009): Symbol not found: _rb_alloc_tmp_buffer_with_count
  Referenced from: <88A96AFA-3481-3EFB-AC1D-0F1A01128D40> /Users/reedstonefood/.gem/ruby/3.0.6/gems/date-3.3.4/lib/date_core.bundle
  Expected in:     <8AA61F44-8675-31DA-8181-506F0BDA7046> /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby - /Users/reedstonefood/.gem/ruby/3.0.6/gems/date-3.3.4/lib/date_core.bundle
  /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
....
[long stacktrace, ending with:]
  /Users/reedstonefood/.gem/ruby/3.0.6/gems/bundler-2.4.15/exe/bundle:29:in `<top (required)>'
  /usr/bin/bundle:23:in `load'
  /usr/bin/bundle:23:in `<main>'
....

## Environment

Bundler       2.4.15
  Platforms   ruby, arm64-darwin-23
Ruby          2.6.10p210 (2022-04-12 revision 67958) [arm64-darwin-23]
  Full Path   /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
  Config Dir  /Library/Ruby/Site
RubyGems      3.0.3.1
  Gem Home    /Users/reedstonefood/.gem/ruby/3.0.6
  Gem Path    /Users/reedstonefood/.gem/ruby/3.0.6:/Users/reedstonefood/.rubies/ruby-3.0.6/lib/ruby/gems/3.0.0
  User Home   /Users/reedstonefood
  User Path   /Users/reedstonefood/.gem/ruby/2.6.0
  Bin Dir     /Users/reedstonefood/.gem/ruby/3.0.6/bin

So it appears that the problem is that it's running a modern version of the date gem (located within my 3.0.6 gem directory) against my "user ruby" (the default one that apple includes) which is 2.6.0.

I can't figure out why it is doing this; I want everything to be running against ruby 3.0.6? Any ideas?

ruby / vscode-rdbg