emacs eshell PATH broken in tramp due to site-start effectively prepending directory

Describe the bug

Emacs eshell apparently depends on tramp-own-remote-path being the first item in tramp-remote-path for it to use a remote server or docker containers PATH. I don't see this in the info manual 4.16 How TRAMP finds and uses programs on the remote host but can confirm that PATH settingi only works in a docker container from eshell if tramp-own-remote-path is first.

In nix right now the value of tramp-remote-path is:

("/run/current-system/sw/bin" tramp-own-remote-path tramp-default-remote-path "/bin" "/usr/bin" "/sbin"

Because in site-start.el it gets appended, but... its only appended after loading tramp-sh.

Many users will have a configuration including:

(use-package tramp 
  :config
  (add-to-list 'tramp-remote-path 'tramp-own-remote-path))

Since tramp-sh loads after tramp though, that means tramp-own-remote-path will never make it to the front of the list.

There is a non-intuitive workaround I'm currently using which probably breaks another thing:

(use-package tramp-sh
  :config
  (add-to-list 'tramp-remote-path 'tramp-own-remote-path))

Steps To Reproduce

Steps to reproduce the behavior:

add (add-to-list 'tramp-remote-path 'tramp-own-remote-path) to your config (don't evaluate, in open emacs order matters)
start emacs
open eshell
cd /docker:some-container:
echo $PATH
notice it's not what the container specifies aka what you get with docker exec -it some-container sh followed by echo $PATH in that container

Expected behavior

"/run/current-system/sw/bin" is inserted into tramp-remote-path /after/ any symbols like tramp-own-remote-path and tramp-default-remote-path.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

I'm using emacs 30 and emacs-overlay, but given I linked to the nixpkgs emacs source that has the same issue that shouldn't matter here.

Notify maintainers

@lovek323 @jwiegley @adisbladis @matthewbauer @atemu @the-kenny

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.15, NixOS, 23.05 (Stoat), 23.05.20230304.3c5319a`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.14.1`
 - nixpkgs: `/nix/store/qilsyakhi0qmjq9iwxn49g3qbl1rp5m7-source`

I tried and failed to make a reproduction of this, so maybe this is just misunderstanding on my part. Using the docker haskell container I assumed that /root/.cabal/bin would be added and printed out with:

(require 'tramp)
(message "")

(message "using the docker haskell image")
(message (concat "path should be: " "/root/.cabal/bin:/root/.local/bin:/opt/ghc/9.4.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"))

(message "show that using tramp-own-remote-path isn't sufficient")
(let ((docker-container-id (string-trim (shell-command-to-string "docker run -it --rm --detach haskell"))))
  (let ((default-directory (concat "/docker:" docker-container-id ":"))
    (tramp-remote-path '(tramp-own-remote-path)))
        (message docker-container-id)
    (message (concat "path is: " (truncate-string-to-width (eshell-command-result "echo $PATH") 100)))))

https://github.com/ParetoOptimalDev/nixpkgs-issue-220187-repro/blob/main/config.el

NixOS / nixpkgs