Re-installation of message-ix fails

maximikos commented 2 years ago

Context

Having detected some irregularities in my modelling results, I decided to re-install message_ix v3.4. This, however, fails, showing a value error message (see below), and I cannot seem to get it running again. My previous installation was installed via conda on a Windows 10 machine.

Approach

My approach for re-installing the message_ix package was as follows:

A - using conda

Deactivate the current message_ix environment
Clean the cache
Remove the conda environment
Check that the conda environment was removed
(Remove remaining message_ix package listing in Anaconda3/pkgs manually with ctrl + del - I tried it once, but it did not make a difference)
Make sure that conda-forge channel is prepended
Create conda environment
Activate above conda environment
Conda-install message_ix package in above conda environment (specifying a version or not did not make a difference)
Check that the installation was successful

B - using pip (since conda did not work...)

Create another, new conda environment
Activate above conda environment
(downgrade python to v3.9 because there is a build error when using python 3.10 (JPype1 is the issue, see here))
Pip-install message_ix package in above conda environment
Install missing pyam dependencies
Check that the installation was successful

Conda

The conda commands used for the above were:

conda deactivate
conda clean --all
conda remove -n message_env --all -y
conda env list
conda config --prepend channels conda-forge
conda create --name message_env -y
conda activate message_env
conda install message-ix"=3.4"
message-ix show-versions

The last line, however, yields the following error message:

Traceback (most recent call last):
  File "C:\Users\maximiko\Anaconda3\envs\message_env\Scripts\message-ix-script.py", line 5, in <module>
    from message_ix.cli import main
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\message_ix\__init__.py", line 10, in <module>
    from .reporting import Reporter
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\message_ix\reporting\__init__.py", line 16, in <module>
    from .pyam import collapse_message_cols
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\message_ix\reporting\pyam.py", line 4, in <module>
    from genno.compat.pyam import util
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\genno\compat\pyam\__init__.py", line 2, in <module>
    import pyam  # noqa: F401
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\pyam\__init__.py", line 12, in <module>
    from pyam.core import *
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\pyam\core.py", line 66, in <module>
    from pyam._ops import _op_data
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\pyam\_ops.py", line 5, in <module>
    from iam_units import registry
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\iam_units\__init__.py", line 18, in <module>
    registry.load_definitions(str(Path(__file__).parent / "data" / "definitions.txt"))
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\pint\registry.py", line 640, in load_definitions
    for lineno, definition in parsed_files.iter_definitions():
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\pint\parser.py", line 101, in iter_definitions
    yield from self._iter_definitions(list(self))
  File "C:\Users\maximiko\Anaconda3\envs\message_env\lib\site-packages\pint\parser.py", line 84, in _iter_definitions
    raise ValueError(
ValueError: The order of the files do not match. (expected: emissions/emissions.txt, found C:\Users\maximiko\Anaconda3\envs\message_env\Lib\site-packages\iam_units\data\emissions\emissions.txt)

I tried different variations of the above, but none worked. That's when I wanted to try installing message-ix via pip.

Pip

conda create --name message
conda activate message
pip install message_ix
pip install pyam-iamc
message-ix show-versions

Although the installation itself finished smoothly, when I start Jupyter Lab and import message_ix, the same ValueError as for the above conda installation shows.

When I close Jupyter Lab and rerun the version check command, I run into the "jpype._jvmfinder.JVMNotFoundException", which is solved through conda install -c anaconda openjdk. Checking again the version yields:

Der Befehl "gams" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.

ixmp:        3.4.0
message_ix:  3.4.0
message_ix_models: None
message_data: None

click:       8.1.2
dask:        2022.04.1
genno:       installed
graphviz:    None
jpype:       1.3.0
… JVM path:  C:\Users\maximiko\Anaconda3\envs\message\Library\bin\bin\server\jvm.dll
openpyxl:    3.0.9
pandas:      1.4.2
pint:        0.18
xarray:      2022.3.0
yaml:        6.0

iam_units:   installed
jupyter:     None
matplotlib:  3.5.1
plotnine:    None
pyam:        1.4.0

GAMS:        'gams' executable not in PATH

python:      3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:18:12) [MSC v.1929 64 bit (AMD64)]
python-bits: 64
OS:          Windows
OS-release:  10
machine:     AMD64
processor:   Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      ('de_DE', 'cp1252')

This is strange, though, because GAMS is already added to the PATH. In addition, when running import message_ix in Jupyter Lab, the above ValueError is shown yet again.

What to do?

How can I get message-ix running again?

khaeru commented 2 years ago

Thanks for the thorough report!

tl;dr: install pint != 0.19.0, != 0.19.1.

The pint ValueError you show is hgrecco/pint#1498; this affects pint 0.19.0 and 0.19.1 but should be fixed in pint 0.19.2 that was just released (hgrecco/pint#1408 / hgrecco/pint#1512).

Without tracing in detail, I think whatever you did to "try installing message-ix via pip" created a strange mix of conda and pip-managed packages—although your show-versions output indicates pint 0.18, this error will definitely never occur with versions of pint before 0.19. So I guess that the package(s) loaded when you "start Jupyter Lab and import message_ix" are different from the package(s) loaded when you invoke the message_ix command-line tool for message-ix show-versions. This could also be the cause of the GAMS path issue.

You may want to:

Check with the Windows where CLI utility, like where message_ix, as well as conda list and conda info.
Cross-reference what is being used by JupyterLab.
Start with a fresh environment.

maximikos commented 2 years ago

Hi @khaeru, thanks for your reply!

Yes, you were right - when I tried setting up via pip in a conda environment, there was some mix-up of stuff, even after down-/upgrading pint. I also couldn't really figure out why Jupyter Lab calls the packages from somewhere else than where conda installs them originally, or rather: how to relocate (if you know, mind dropping a comment?). So, for instance, when having pip-installed pint=0.19.2, everything looked fine; I could import message_ix; but when calling mp = ixmp.Platform(), the kernel interrupted with the following message, after which it restarted straight away:

# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff9d2348db9, pid=16588, tid=18344
#
# JRE version:  (11.0.6+8) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.6+8-b765.1, mixed mode, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# C  0x00007ff9d2348db9
#
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Users\maximiko\Box Sync\Coursework\IAM\hs_err_pid16588.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp

Anyway, when doing everything with conda in a conda environment, it worked for me with downgrading pint to version 0.18. Version 0.19.2 is apparently not available through the conda-forge channel yet (just noticed you commented on that in #585), so I didn't try that.

khaeru commented 2 years ago

Great news that it works!

I use Jupyter only rarely, and JupyterLab not at all. So while I know the former allows a selection of kernels and it matters from which conda environment you start it, I don't know how this translates to the latter, sorry. You might try posting your experience in the Dicussions for the benefit/input of other users.

Meanwhile, hearing that this issue is resolved from your point of view, I'll close it.

Version 0.19.2 is apparently not available through the conda-forge channel yet, so I didn't try that.

We are tracking the upstream PR that will eventually make that happen; see https://github.com/iiasa/message_ix/pull/585#issuecomment-1107845032

iiasa / message_ix