wrong cursor position when switching between buffers

[Spo0on]

That's what happens: let's say I'm editing two files: file_A and file_B and I open Goyo mode in the file_A at line 134. Then I move the cursor at line 168 and I switch to file_B using the bnext command. Then I come back to file_A using the bprev command (or even with bnext) and I expect to be at line 168, but the cursor resets at line 134. This only happens in the file I opened Goyo mode from. It's really annoying and makes impossible to use Goyo mode with multiple buffers. This is my configuration:

vim.g.goyo_width = 120
vim.g.goyo_height = 55
vim.g.goyo_linenr = 0

function Goyo_enter()
    vim.api.nvim_set_option("showmode", false)
    vim.api.nvim_set_option("showcmd", false)
    vim.api.nvim_set_option("showtabline", 0)
    vim.api.nvim_set_option("laststatus", 0)
    vim.api.nvim_set_hl(0, "StatusLine", { link = "Normal" })
    vim.opt.fillchars = vim.opt.fillchars + { eob = " ", stl = " ", vert = " " }
    vim.cmd([[exec 'hi! FoldColumn guibg=bg' . ' guifg=' . synIDattr(hlID('Comment'), 'fg')]])
    require("lualine").hide()
end

function Goyo_leave()
    vim.api.nvim_set_option("showmode", true)
    vim.api.nvim_set_option("showcmd", true)
    vim.api.nvim_set_option("showtabline", 2)
    vim.api.nvim_set_option("laststatus", 3)
    vim.opt.fillchars = vim.opt.fillchars + { eob = "~" }
end

local GoyoGroup = vim.api.nvim_create_augroup("Goyo", { clear = true })
vim.api.nvim_create_autocmd("User", {
    pattern = "GoyoEnter",
    command = "lua Goyo_enter()",
    group = GoyoGroup,
})
vim.api.nvim_create_autocmd("User", {
    pattern = "GoyoLeave",
    command = "lua Goyo_leave()",
    group = GoyoGroup,
})

This happens even with default configuration.