Installation of ESMValCore in dev mode with git+ssh and `pip install --no-deps .` is prone to a load of issues

[valeriupredoi]

@bouweandela found an issue with the auto testing implemented in #2994 - specifically Julia-related; then I started digging in and noticing that Julia was improperly configured when installing Core in dev mode; I intially blamed the porr Circle CI machine for messing around with paths - it proves the whole thing is not Circle's fault, but ours. Here are a few steps to reproduce the issue:

Steps to reproduce issue with Julia path

• get a fresh copy of ESMValTool from git
• create a fresh environment:

  mamba env create -n esmvaltool-julia-test -f environment.yml

• install the tool in dev mode:

  pip install -e .[develop]

• check Julia path:

  (esmvaltool-julia-test) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool_Julia$ echo $JULIA_LOAD_PATH
  @:@esmvaltool-julia-test:@stdlib

• install Core in dev mode:

  pip install --upgrade git+https://github.com/ESMValGroup/ESMValCore
  pip install --no-deps .

• check Julia path again:

  (esmvaltool-julia-test) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool_Julia$ echo $JULIA_LOAD_PATH
  @:@esmvaltool-julia-test:@stdlib

• install Julia stuffs:

  esmvaltool install Julia
  /home/valeriu/miniconda3/envs/esmvaltool-julia-test/lib/python3.10/site-packages/esmvalcore/_config/__init__.py:13: ESMValCoreDeprecationWarning: The private module `esmvalcore._config` has been deprecated in ESMValCore version 2.8.0 and is scheduled for removal in version 2.9.0. Please use the public module `esmvalcore.config` instead.
  warnings.warn(
  installing Julia packages, please wait...
  Installation successful

• check Julia path again:

  (esmvaltool-julia-test) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool_Julia$ echo $JULIA_LOAD_PATH
  @:@esmvaltool-julia-test:@stdlib

• inspect Manifest.toml in esmvaltool/install/Julia/ -> ain't there!
• append/reset Julia path:

  JULIA_LOAD_PATH="~/ESMValTool_Julia/esmvaltool/install/Julia:@:@esmvaltool:@stdlib"

  and reinstall Julia pkgs: esmvaltool install Julia

(esmvaltool-julia-test) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool_Julia$ esmvaltool install Julia
/home/valeriu/miniconda3/envs/esmvaltool-julia-test/lib/python3.10/site-packages/esmvalcore/_config/__init__.py:13: ESMValCoreDeprecationWarning: The private module `esmvalcore._config` has been deprecated in ESMValCore version 2.8.0 and is scheduled for removal in version 2.9.0. Please use the public module `esmvalcore.config` instead.
  warnings.warn(
installing Julia packages, please wait...
Installation successful
(esmvaltool-julia-test) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool_Julia$ julia esmvaltool/diag_scripts/examples/diagnostic.jl
ERROR: LoadError: ArgumentError: Package YAML not found in current path.
- Run `import Pkg; Pkg.add("YAML")` to install the YAML package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1163 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
in expression starting at /home/valeriu/ESMValTool_Julia/esmvaltool/diag_scripts/examples/diagnostic.jl:13

TL;DR - the way we tell users how to install Core in dev mode is not optimal at all:

• esmvaltool installs locally, whereas esmvalcore installs in the env's site-packages
• Julia gets immediately affect by this because, even if it creates the Manifest.toml in the site-packes location -

  (esmvaltool-julia-test) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool_Julia$ ls -la ~/miniconda3/envs/esmvaltool-julia-test/lib/python3.10/site-packages/esmvaltool/install/Julia/
  total 32
  drwxrwxr-x 2 valeriu valeriu  4096 Feb 22 11:49 .
  drwxrwxr-x 4 valeriu valeriu  4096 Feb 22 11:46 ..
  -rw-rw-r-- 1 valeriu valeriu 15504 Feb 22 11:49 Manifest.toml
  -rw-rw-r-- 1 valeriu valeriu   495 Feb 22 11:46 Project.toml
  -rw-rw-r-- 1 valeriu valeriu   201 Feb 22 11:46 setup.jl

  the actual runtime Julia can't find it and belches out with pkg YAML not found etc.

• not only Julia is a problem, but rather, if a user needs to perfom changes to Core, it's much easier if Core was somewhere in a more accessible location than all the way in Miniconda's bellows in site-packages

[valeriupredoi]

So we either tel the users to install Tool, and immediately after Julia, then do the Core dev installation, then reset the JULIA_LOAD_PATH - which is bulky, or we tell them to grab Core from git themselves, and install it with the classical pip install -e .[develop] which I have always used and had no issues with any paths or Julias

[valeriupredoi]

@bouweandela I just had an attentive read through the installation instructions of ESMValTool with ESMValCore in dev mode (git clone etc) - those are very nice and valid, and are not susceptible to the issue described here - well done us! The issue with Julia's path creeps up only when one uses git+ssh install for Core - as such, I reckon we either add a note to the doc saying don't git+ssh Core (even though we don't instruct users to do it) or, we just forget about it! I'd change the test in #2994 to reproduce exatly the install steps we tell users to follow, if I were you :+1:

[bouweandela]

Thanks for looking into this @valeriupredoi! I just tried this in the branch test-esmvalcore-development-install, but unfortunately test running a Julia diagnostic is still failing (same as on my laptop).

[valeriupredoi]

I've just pushed a change to that branch that if it doesn't solve the issue then I will officially eat my hat :tophat: on the next TLT call :grin: No, but seriously now, your steps in the PR were missing both the installation of Tool after env activation, and installing Julia after the install of Core in dev mode, which the second thing should not matter, but let's replicate exactly the install instructions in the documentation