search

iiasa / message_ix

The integrated assessment and energy systems model MESSAGEix
https://docs.messageix.org
Apache License 2.0
116 stars 152 forks source link

JRE error while running westeros_baseline in jupyter notebook #482

Open marek-iiasa opened 3 years ago

marek-iiasa commented 3 years ago

Problem description

Running westeros_baseline (without any modifications) in jupyter notebook (the Python [conda env:msg_env] kernel) caused error (the report in the log-file) hs_err_pid43115.log

The error occurs when running cell #34: scenario.solve()

My best guess is that the message-ix installation is inconsistent with my iMac-env or with the installed GAMS. Unfortunately, I have no idea how to explore and fix the problem.

Versions

message-ix show-versions

ixmp:        3.3.0
message_ix:  3.3.0
message_ix_models: None
message_data: None

click:       8.0.1
dask:        2021.06.0
genno:       installed
graphviz:    None
jpype:       1.2.1
… JVM path:  /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre/lib/jli/libjli.dylib
openpyxl:    3.0.7
pandas:      1.2.4
pint:        0.17
xarray:      0.18.2
yaml:        5.4.1

iam_units:   installed
jupyter:     installed
matplotlib:  3.4.2
plotnine:    None
pyam:        0.11.0

GAMS:        35.2.0

python:      3.9.4 | packaged by conda-forge | (default, May 10 2021, 22:13:15) 
[Clang 11.1.0 ]
python-bits: 64
OS:          Darwin
OS-release:  20.5.0
machine:     x86_64
processor:   i386
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      (None, 'UTF-8')
marek-iiasa commented 3 years ago

I attach also the output to console; maybe it contains useful info? console.txt

marek-iiasa commented 3 years ago

I run the a copy ipynb Muhammad presented today and got another exception (copied below) which shows that my message-ix installation is faulty. I've checked GAMS (with their transport model): it appears to work OK. Please suggest how to trace the source of the problems (the above and below summarized). My best guess from the exception listing below is that jpype continues to be the trouble-maker.

scen.solve()
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in at.ac.iiasa.ixmp.objects.MsgScenario.toGDX()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in at.ac.iiasa.ixmp.objects.Scenario.makeGamsDB()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.<init>()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.initializeAttributes()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.verifySystemDirectory()

Exception: Java Exception

The above exception was the direct cause of the following exception:

com.gams.api.GAMSException                Traceback (most recent call last)
<ipython-input-29-e0c31090e7e2> in <module>
----> 1 scen.solve()

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/message_ix/core.py in solve(self, model, solve_options, **kwargs)
    563             :class:`.GAMSModel`.
    564         """
--> 565         super().solve(model=model, solve_options=solve_options, **kwargs)
    566 
    567     def add_macro(self, data, scenario=None, check_convergence=True, **kwargs):

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/core.py in solve(self, model, callback, cb_kwargs, **model_options)
   1666         # Iterate until convergence
   1667         while True:
-> 1668             model.run(self)
   1669 
   1670             # Store an iteration number to help the callback

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/message_ix/models.py in run(self, scenario)
    361 
    362         try:
--> 363             result = super().run(scenario)
    364         finally:
    365             # Remove the optfile regardless of whether the run completed

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/model/gams.py in run(self, scenario)
    274         try:
    275             # Write model data to file
--> 276             scenario.platform._backend.write_file(
    277                 self.in_file, ItemType.SET | ItemType.PAR, **s_arg
    278             )

~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/backend/jdbc.py in write_file(self, path, item_type, **kwargs)
    576             # include_var_equ=False -> do not include variables/equations in
    577             # GDX
--> 578             self.jindex[ts].toGDX(str(path.parent), path.name, False)
    579         elif path.suffix == ".csv" and item_type is ItemType.TS:
    580             models = set(filters.pop("model"))

com.gams.api.GAMSException: com.gams.api.GAMSException: GAMS system directory is both not specified and not found from your environment!
awais307 commented 3 years ago

@marek-iiasa can you please check if you correctly specified GAMS in your environment variable as mentioned in installation instructions here.

marek-iiasa commented 3 years ago

yes, gams, as well as java, are in the path

(msg_env) Ba4:wrksp marek$ echo $PATH
/Users/marek/anaconda3/envs/msg_env/bin:/Library/Frameworks/GAMS.framework/Resources:/Applications/anaconda3/condabin:/anaconda3/bin:.:/usr/local/git/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/Java/Home/bin:/Users/marek/bin:/Users/marek/src/mcma/ScriptsMac:/Library/TeX/texbin:/Library/Apple/usr/bin

I assume the including in the $PATH all one has to do, or ?

OFR-IIASA commented 3 years ago

I did the install this morning on macOS: step1. I deleted Anaconda and GAMS. step2. rebooted machine. step3. installed GAMS 35.2 (including ticking the box to add to PATH) step4. installed anaconda3-2021.05 step5. installed messageix via anaconda following each step of the instructions with the addition of setting channel priority to strict. step5. downloaded tutorials step6. started jupyter notebook from the command-line ensuring environment 'message_env' is active.

Result -> no issues encountered as described above.

Below are the results from message-ix show-versions


ixmp:        3.3.0
message_ix:  3.3.0
message_ix_models: None
message_data: None

click:       8.0.1
dask:        2021.06.0
genno:       installed
graphviz:    None
jpype:       1.2.1
… JVM path:  /Users/oliver/opt/anaconda3/envs/message_env/lib/jli/libjli.dylib
openpyxl:    3.0.7
pandas:      1.2.4
pint:        0.17
xarray:      0.18.2
yaml:        5.4.1

iam_units:   installed
jupyter:     installed
matplotlib:  3.4.2
plotnine:    0.8.0
pyam:        0.7.0

GAMS:        35.2.0

python:      3.9.4 | packaged by conda-forge | (default, May 10 2021, 22:13:15) 
[Clang 11.1.0 ]
python-bits: 64
OS:          Darwin
OS-release:  19.6.0
machine:     x86_64
processor:   i386
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      (None, 'UTF-8')
marek-iiasa commented 3 years ago

Many thanks, Oliver. Comparing your and my installations shall help me to narrow possible problems with my installation. The key "suspect" I noticed is JVM path: your points to the message_env while mine points to the JDK (which I installed and added to $PATH). Please comment on:

  1. Have you installed JDK and added its bin to your $PATH ?
  2. How your $JAVA_HOME is defined?
  3. Should JDK be added to $PATH or defining $JAVA_HOME is enough?
  4. Confirm that a tutorial (coffee_machine or westeros) opt-task was solved by gams; the show-versions in my installation didn't show errors; the exception occurred during scen.solve()
  5. The version of pyam differ (0.7 vs 0.11); I guess should not cause problems with running tutorials
  6. There is no plotnine in my env; this might be due to uses of pip (instead of conda) for my installation
  7. I noticed that we have slightly different OS-releases (19.6 vs 20.5) but I guess this unlikely causes problem with my installation.

I will reinstall message-ix when I'll be sure that our environments are as close as possible, i.e. after you will comment on the first four questions.

OFR-IIASA commented 3 years ago

Marek,

As for questions 1, 2 and 3 -> No, I havent installed JDK.

  1. Yes, the Westeros tutorial solved.
  2. No, i believe this shouldnt create any issues.
  3. Plotnine is only required for some reporting functionalities which are not called during the actual solving.
  4. I have an older version of macOS, because the BigSur update caused macs of my type to be "bricked" and since then I have been hesitant to upgrade. But this also shouldnt make any issues.
marek-iiasa commented 3 years ago

Thanks again, Oliver,

Aha... Does it mean you have no $JAVA_HOME defined (as required by the installation instructions)?

OFR-IIASA commented 3 years ago

Marek, no i didnt define $JAVA_HOME. This is only needed if Java is installed manually. I didnt do this so it is done so as part of the MESSAGEix installation process. So when calling which java, i get:

/Users/oliver/opt/anaconda3/envs/message_env/bin/java

marek-iiasa commented 3 years ago

Oliver, thanks again. I'll reinstall message_ix later today following your path, and report result here.