Transitions to monorepo

[mandresm]

This merge allows for esm_versions to be able to install the monorepo version automatically.

Once version 6 is available in release the user will only need to do esm_versions upgrade to get the installation of monorepo started. Note that this only happens if esm_tools is on the release branch and the repo is clean.

For testing before version 6 is available you can follow these steps:

1. esm_tools upgrade (once this branch is merged)
2. esm_tools upgrade esm_tools=monorepo

[seb-wahl]

For testing before version 6 is available you can follow these steps:

1. esm_tools upgrade (once this branch is merged)
2. esm_tools upgrade esm_tools=monorepo

After a few attempts I got it to work with

cd esm_tools
git checkout release
esm_versions upgrade esm_version_checker=transitions_to_monorepo
cd .. # this is important because if you are in the esm_tools you get an installation failed in the end
        # because the pip install --user -e . command used fails.
module load anaconda3 # required to have python3 during the installation, I'm testing on blogin
esm_versions upgrade esm_tools=monorepo

I'll now test whether the monorepo actually works for our models. Otherwise it looks good.

[seb-wahl]

Do you have an idea?

blogin3:~/esm/esm_tools $ esm_runscripts
Traceback (most recent call last):
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pandas 0.25.1 (/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages), Requirement.parse('pandas>=1.0'), {'xarray'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shkifmsw/.local/bin/esm_runscripts", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3250, in <module>
    @_call_aside
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3263, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pandas 0.25.1 (/sw/tools/anaconda3/2019.10/skl/lib/python3.7/site-packages), Requirement.parse('pandas>=1.0'), {'xarray'})
blogin3:~/esm/esm_tools $ ls -lh ~/.local/bin/
total 0
-rwxr-xr-x 1 shkifmsw shkifmsw  243 Apr 28  2021 coloredlogs
-rwxr-x--- 1 shkifmsw shkifmsw  401 Dec  2 11:14 esm_archive
-rwxr-x--- 1 shkifmsw shkifmsw  401 Dec  2 11:14 esm_cleanup
-rwxr-x--- 1 shkifmsw shkifmsw  403 Dec  2 11:14 esm_database
-rwxr-x--- 1 shkifmsw shkifmsw  399 Dec  2 11:14 esm_master
-rwxr-x--- 1 shkifmsw shkifmsw  401 Dec  2 11:14 esm_plugins
-rwxr-x--- 1 shkifmsw shkifmsw  407 Dec  2 11:14 esm_runscripts
-rwxr-x--- 1 shkifmsw shkifmsw  397 Dec  2 11:14 esm_tests
-rwxr-x--- 1 shkifmsw shkifmsw  397 Dec  2 11:14 esm_tools
-rwxr-x--- 1 shkifmsw shkifmsw  405 Dec  2 11:14 esm_utilities
-rwxr-xr-x 1 shkifmsw shkifmsw  243 Dec  2 11:08 gfw_creator
-rwxr-xr-x 1 shkifmsw shkifmsw  245 Apr 28  2021 humanfriendly
-rwxr-xr-x 1 shkifmsw shkifmsw  234 Dec  2 11:11 pysemver
-rwxr-xr-x 1 shkifmsw shkifmsw  245 Dec  2 11:11 xdg-env
-rwxr-x--x 1 shkifmsw shkifmsw 1.2K Dec  2 11:11 xdg-env-completion.bash

I had expected that esm_runscripts et al are now removed from ~/.local/bin/?

[denizural]

Hi @seb-wahl, did you make a clean installation or just upgraded the previous one?

[mandresm]

Hi @seb-wahl, esm_runscripts, esm_master, et al are still created in the .local/bin, so I don't think that's the problem. I'll try that on blogin and see if I can reproduce that as well as the need of doing cd .. (which is weird because I think I put os.chdir(tools_dir) somewhere.

[mandresm]

@seb-wahl , cna you provide your pandas version? (pip list).

I think we might be able to solve the problem by specifying a version of pandas later than 1.0 in the requirements section of the setup.py.

[mandresm]

I just put together a fix. You can come back to the multirepo by just doing:

pip uninstall esm_tools
cd esm_tools
git checkout release
./install.sh

And then you can try again. That should install a good pandas version.

[seb-wahl]

Hi @seb-wahl, did you make a clean installation or just upgraded the previous one?

I upgraded from a previous version to test whether that works as expected.

[seb-wahl]

I just put together a fix. You can come back to the multirepo by just doing:

pip uninstall esm_tools
cd esm_tools
git checkout release
./install.sh

And then you can try again. That should install a good pandas version.

this worked, took a few minutes as blogin pandas was compiled from scratch... I'm now testing the compilation and running

[mandresm]

Installation works in:

• ollie
• mistral
• aleph
• hlrn

I'll wait for @seb-wahl to approve this merge and to confirm that their tests run in HLRN to merge.

[mandresm]

Hi @seb-wahl ,

Coming back to this issue. I managed to reproduce the behavior in mistral:

esm_versions upgrade esm_version_checker=transitions_to_monorepo does nothing.

Apparently this is happening because pip --upgrade only pulls a branch if the version defined in setup.py of that branch is superior to the current version of the package. There is a way to check that:

1. Have a look at this file: ~/.local/lib/python3.<version>/site-packages/esm_version_checker-5.1.5.dist-info/direct_url.json, you'll see that there is no transitions_to_monorepo branch specified there.
2. Change to a different version: esm_versions upgrade esm_version_checker=v5.1.4. You'll see that the commit ID of the direct_url.json has changed.
3. Execute esm_versions upgrade esm_version_checker=transitions_to_monorepo and this time it should work. You won't see the transitions_to_monorepo branch in the output of the esm_versions check, but you'll be able to see it in the direct_url.json.

All this could certainly be solved, but I am considering the following 3 points:

• most of the users I work with, use editable package installations (first cloned the package and then pip install -e .) when they need a specific branch.
• the esm_version_checker is only going to be used from now on for versions smaller than 6.
• once we bump the version of this package with this PR and merge, esm_version_checker is gonna be upgraded on the next esm_versions upgrade the user does.

So in summary, I think this fix is a) not affecting the functionality of this PR, and b) not worth of solving it.

If you agree with that, then I'll bump the version and merge.

[pgierz]

Sorry, checking in after my long holiday: thumbs up from my side. Biggest surprise was that we managed to mess up 1/0 exit codes (@seb-wahl latest commit)

Esm version checker will disappear once we are on version 6. we should also at the same time with the release ensure that's on pypi. Maybe then just pip install -U esm-Tools. But I have not checked what happens if you have one of the other sub packages already installed.

See you all on Monday!