bug: switching buffers when hidden=off results in warning that previous buffer not attached to lsp client

Did you check docs and existing issues?

[X] I have read all the trouble.nvim docs
[X] I have searched the existing issues of trouble.nvim
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)

0.9.1

Operating system/version

Ubuntu 22.04

Describe the bug

When hidden is set to false navigating to a different file from Trouble results in a warning from neovim semantic token engine https://github.com/neovim/neovim/blob/54be7d6b45a9ae51c218c0e5c1c20dd608b97297/runtime/lua/vim/lsp/semantic_tokens.lua#L587

Steps To Reproduce

open a file with nvim -u repro.lua file gr on a symbol (that is across multiple files) to open the trouble buffer with lsp_references listed select a reference in a different file and hit enter. Displays (from my custom on_attach function):

on_attach1                                                                                                                                                                                          
on_attach2
Press ENTER or type command to continue

Hit enter Displays error:

[LSP] Client with id 1 not attached to buffer 1

Expected Behavior

no [LSP] Client with id 1 not attached to buffer 1 warning

Repro

-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({ "git", "clone", "--filter=blob:none", "https://github.com/folke/lazy.nvim.git", lazypath, })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  "folke/tokyonight.nvim",
  "folke/trouble.nvim",
  -- add any other plugins here
  "neovim/nvim-lspconfig",
  "nvim-tree/nvim-web-devicons",
}
require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

require('lspconfig').clangd.setup({
    on_attach = function(_, bufnr)
        print("on_attach" .. bufnr)
    end})

vim.opt.hidden = false

vim.keymap.set("n", "gr", function() require("trouble").open("lsp_references") end, {silent = true})

folke / trouble.nvim

bug: switching buffers when hidden=off results in warning that previous buffer not attached to lsp client #327