When going to a file from diff view some settings overwrite

[AlejandroSanchez90]

Description

When I go to the diff view (diff this), hover over a file, and use gf (go to file), I can see the file, but a lot of my config stops working. For example, :set number gets disabled, and my keys to go down and up (c-n and c-p) stop working, which key stops working () if I do :verbose set number? I get this response

Last set from /nvim/lazy/neogit/lua/neogit/lib/buffer.lua line 413 

Neovim version

nvim v0.10.1

Operating system and version

macOs

Steps to reproduce

1. :NeoGit
2. dd on a branch
3. hover any file
4. press gf
5. on the file try some common keys

Expected behavior

No response

Actual behavior

some config gets overwritten

Minimal config

return {
    "NeogitOrg/neogit",
    event = "VeryLazy",
    dependencies = {
        "nvim-lua/plenary.nvim", -- required
        "sindrets/diffview.nvim", -- optional - Diff integration

        -- Only one of these is needed, not both.
        "ibhagwan/fzf-lua",
    },
    config = function()
        local neogit = require("neogit")
        neogit.setup({})

        local keymap = vim.keymap -- for conciseness
        keymap.set("n", "<leader>go", "<cmd>Neogit<cr>", { desc = "Open NeoGit" })
        keymap.set("n", "<leader>gf", "<cmd>Neogit fetch<cr>", { desc = "Fetch" })
        keymap.set("n", "<leader>gb", "<cmd>Neogit branch<cr>", { desc = "Branch" })
        keymap.set("n", "<leader>gg", "<cmd>Neogit commit<cr>", { desc = "Commit" })
        keymap.set("n", "<leader>gp", "<cmd>Neogit push<cr>", { desc = "Push" })
        keymap.set("n", "<leader>gs", "<cmd>FzfLua git_status<cr>", { desc = "Status" })

        keymap.set("n", "<leader>gdh", "<cmd>DiffviewFileHistory % --base=LOCAL<cr>", { desc = "File github history" })

        --DiffviewFileHistory % --base=LOCAL
    end,
}