Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Python Reference Queue"

ravitby commented 1 month ago

Running a notebook script getting the following error:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Python Reference Queue"

The code is running under venv environment.

The ixmp module is used as part of the message_data module and there are no issues in this environment.

Code sample or context

If possible, provide a minimal, working code example, formatted like:


import os
import pandas as pd
import ixmp as ix
import message_ix
from postprocessor import plotter
#%load_ext autoreload
#%autoreload 2

mp = ix.Platform()

current_path = os.path.dirname(os.path.abspath("__file__"))
os.chdir(current_path)

input_from_excel = True

model = "MESSAGEix-IL"
scenario = "test"

if input_from_excel:
    # The name of the Excel file containing model data
    file_name = current_path + '/V16_MESSAGEix_IL_New.xlsx'

    # Creating a new (empty) scenario
    scen = message_ix.Scenario(mp, model, scenario, version="new")

    # Reading data from Excel (takes up to a few minutes)
    scen.read_excel(file_name, add_units=True, init_items=True, commit_steps=True)
else:
    # Loading an existing (default) scenario from the database
    scen = message_ix.Scenario(mp, model, scenario)

Expected result

Continue to the next command.

Problem description

This is the output: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Python Reference Queue"

Not sure how to clear the memory issue.

Versions

'ixmp show-versions' in a terminal:

ixmp show-versions             [Sun Jul 28 14:37:05 2024]

ixmp:        3.8.1.dev3+g8a2aa91
     5713fc1 (HEAD -> main, origin/main, origin/HEAD) Merge pull request #524 from iiasa/enh/2024-W13
message_ix:  3.8.1.dev2+g693f227
     023b91a (HEAD -> main, origin/main, origin/HEAD) Bump the all-dependencies group with 1 update (#796)
message_ix_models: 2023.11.25.dev20+gaf092b1c.d20240121
     8f39459f (HEAD -> main, origin/main, origin/HEAD) Tidy data from #99
message_data: 2020.6.21.dev2619+g312f816e
     f9fa0f4da (HEAD -> il/transport-2024-01-20, origin/il/transport-2024-01-20) Revert input data to the original state

click:       8.1.6
dask:        2023.8.0
genno:       installed
graphviz:    0.20.1
jpype:       1.4.1
… JVM path:  /Library/Java/JavaVirtualMachines/jdk-20.jdk/Contents/Home/lib/libjli.dylib
openpyxl:    3.1.2
pandas:      2.0.3
pint:        0.22
xarray:      2023.7.0
yaml:        6.0.1

iam_units:   installed
jupyter:     1.0.0
matplotlib:  3.7.0
plotnine:    0.12.2
pyam:        1.9.0

GAMS:        36.1.0

python:      3.10.6 (v3.10.6:9c7b4bd164, Aug  1 2022, 17:13:48) [Clang 13.0.0 (clang-1300.0.29.30)]
python-bits: 64
OS:          Darwin
OS-release:  23.5.0
machine:     x86_64
processor:   i386
byteorder:   little
LC_ALL:      None
LANG:        en_US.UTF-8
LOCALE:      ('en_US', 'UTF-8')

glatterf42 commented 1 month ago

Thanks for opening this issue, @ravitby. While you haven't said exactly which line is causing the error (which should be part of the full error traceback you see), this error was caused in the past by trying to connect to a crowded local database. In ixmp, when a database is filled with scenarios over time, connecting to that database takes longer and longer and requires more and more energy until eventually you hit a limit. That is why it's best to store scenarios in project-specific databases. To do that, you will have to register a new ixmp platform, e.g. via the CLI: ixmp platform add <platform_name> jdbc hsqldb <path_to_platform>, where a recent example for a path on my system is /home/fridolin/.local/share/ixmp/localdb/test_feature_price_emission. If you run ixmp config show, you will find the path where your default DB is located and you want to use the same path, but with a different file/DB name for this new platform. Then in your code, you can connect to the new DB by specifying its name: mp = ix.Platform("<platform_name>").

Please let us know if this solves your issue.

ravitby commented 1 month ago

Thanks for the detailed answer, problem solved.

iiasa / ixmp