IXMP on Apple Silicon (ARM architecture)

SiddharthJoshi-Git commented 1 year ago

During Solve:

ERROR Reported: java.lang.java.lang.UnsatisfiedLinkError Error can be due to CPU/ Java architecture mismatch. Have tried both X86 and ARM builds of JAVA, still this error persists.

Code sample or context

--- Warning: The GAMS version [41.1.0] differs from the API version [24.8.3].
java.lang.UnsatisfiedLinkError: Can't load library: [/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/backend/jdbc/libgmdjni64.dylib](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/backend/jdbc/libgmdjni64.dylib)
    at java.base[/java.lang.ClassLoader.loadLibrary](https://file+.vscode-resource.vscode-cdn.net/java.lang.ClassLoader.loadLibrary)(ClassLoader.java:2393)
    at java.base[/java.lang.Runtime.load0](https://file+.vscode-resource.vscode-cdn.net/java.lang.Runtime.load0)(Runtime.java:755)
    at java.base[/java.lang.System.load](https://file+.vscode-resource.vscode-cdn.net/java.lang.System.load)(System.java:1953)
    at com.gams.api.gmd.<clinit>(gmd.java:202)
    at com.gams.api.GAMSDatabase.createGMDObject(GAMSDatabase.java:342)
    at com.gams.api.GAMSDatabase.initializeAttributes(GAMSDatabase.java:328)
    at com.gams.api.GAMSDatabase.<init>(GAMSDatabase.java:256)
    at com.gams.api.GAMSWorkspace.addDatabase(GAMSWorkspace.java:1024)
    at com.gams.api.GAMSWorkspace.addDatabase(GAMSWorkspace.java:1010)
    at at.ac.iiasa.ixmp.objects.Scenario.makeGamsDB(Scenario.java:1812)
    at at.ac.iiasa.ixmp.objects.MsgScenario.toGDX(MsgScenario.java:1328)
Unexpected exception formatting exception. Falling back to standard exception
Traceback (most recent call last):
  File "MsgScenario.java", line 1328, in at.ac.iiasa.ixmp.objects.MsgScenario.toGDX
Exception: Java Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/IPython/core/interactiveshell.py](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/IPython/core/interactiveshell.py)", line 3460, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "[/var/folders/jh/k9yd8qn1389drngmnwrj91740000gp/T/ipykernel_13760/2994721627.py](https://file+.vscode-resource.vscode-cdn.net/var/folders/jh/k9yd8qn1389drngmnwrj91740000gp/T/ipykernel_13760/2994721627.py)", line 1, in <module>
    scenario.solve()
  File "[/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/message_ix/core.py](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/message_ix/core.py)", line 676, in solve
    super().solve(model=model, solve_options=solve_options, **kwargs)
  File "[/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/core/scenario.py](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/core/scenario.py)", line 835, in solve
    model_obj.run(self)
  File "[/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/message_ix/models.py](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/message_ix/models.py)", line 377, in run
    result = super().run(scenario)
  File "[/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/model/gams.py](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/model/gams.py)", line 276, in run
    scenario.platform._backend.write_file(
  File "[/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/backend/jdbc.py](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/backend/jdbc.py)", line 579, in write_file
    self.jindex[ts].toGDX(str(path.parent), path.name, False)
java.lang.java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: Can't load library: [/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/backend/jdbc/libgmdjni64.dylib](https://file+.vscode-resource.vscode-cdn.net/Users/iiasa/opt/anaconda3/envs/env_x86/lib/python3.9/site-packages/ixmp/backend/jdbc/**libgmdjni64.dylib**)

glatterf42 commented 1 year ago

I don't know if that makes a difference, but your path includes an env_x86, while your library seems to be for 64 bit: libgmdjni64.dylib. I will have to read up on that. In the meantime, can you please make sure to check everything suggested e.g. in this answer, so check your java.library.path and PATH environment variable and see if they include the path to the ixmp library?

khaeru commented 1 year ago

I did some casual searching for "libgmdjni" and "gmdjni", and found no direct references in the ixmp_source repo, nor much on the Internet in general.

".dylib" is a macOS library file, and (as @glatterf42 suggests) I guess the one version bundled with ixmp is x86_64/amd64-specific, with no analogous file for the Apple ARM architecture.

This may be a result of the process whereby the ixmp_source JAR and its dependencies are prepared, i.e. it may pick up dependencies for certain platforms and architectures, but this set may exclude the Apple ARM architecture.

To reiterate my point during the MESSAGE meeting: since ixmp_source is effectively unmaintained (we do not have a Java developer assigned to maintain it, including fixing bugs), this may go unfixed unless boosted to higher priority.

SiddharthJoshi-Git commented 1 year ago

Thanks Paul and Fridolin. It seems that Pauls analysis is correct here. One possible solution is to recompile the Java Code (Eclipse/Xcode) while specifying the ARM architecture. Another stopgap solution that I found is to use VMWare Fusion personal license (Free for now) along with Windows 11 ARM developer preview to run the code under emulation - very low performance hit.

SiddharthJoshi-Git commented 1 year ago

Steps to enable MESSAGEix model development on Apple (ARM based CPU) --- Verified 17th April 2023.

1.) Install GAMS 40 - https://www.gams.com/40/ 2.) Install Miniconda (Miniconda3 macOS Intel x86 64-bit pkg)- https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.pkg 3.) Create conda environment (Python version 3.8)- conda create --name ME_X86_P3.8 python=3.8 3.1) Ensure that the packages being installed next are coming from "pkgs/main/osx-64" conda channel 4.) Fork and install ixmp from source - https://docs.messageix.org/projects/ixmp/en/latest/install.html#from-source 5.) Install open JDK (ver. conda-forge/osx-64::openjdk-11.0.9.1-hcf210ce_1)- conda install -c conda-forge openjdk 6.) Fork and install MESSAGEix from source - https://docs.messageix.org/en/stable/install.html#from-source 7.) Install pyam - pip install pyam-iamc 8.) Install Jupyterlab - conda install -c conda-forge jupyterlab

The above steps are tested for M1Pro processor. On 13th April 2023, GAMS native installer for Apple ARM was released as Beta. This is yet to be tested.

glatterf42 commented 1 year ago

Thanks for providing us with these steps! Do you know which of them (if any) are ARM-CPU specific? Also, do you think you'll be able to test the new GAMS native installer?

SiddharthJoshi-Git commented 1 year ago

HI Fridolin,

The beauty of these install steps is that none of the following is ARM specific: 1.) MESSAGEix,IXMP 2.) JAVA 3.) GAMS

The above steps forces the MACOS ARM to install X86(intel/AMD) platform specific code. This results in rosetta translation layer (https://developer.apple.com/documentation/apple-silicon/about-the-rosetta-translation-environment) to kick-in which translates X86 to ARM on the fly during runs. This is not ideal as there is performance penalty during translation. But, since the ARM chips from Apple are already industry leader in performance, these performance hits are not noticeable. For example, model load from Excel to IXMP takes 15 mins, Model solve takes 6 mins for a global Messageix model.

In the previous queries which I am sure will be same for any new user, the anaconda env was pulling packages from pkgs/main/osx-arm64/ which resulted in errors during model runs. With the above steps, the packages are pulled from /pkgs/main/osx-64/ which forces rosetta to work its magic.

For GAMS Arm native build, I will do some tests and revert.

Best, Sid

glatterf42 commented 1 year ago

Thanks, then we can update our installation instructions ahead of the community meeting and workshop. Please also let us know what you find for the GAMS ARM native build, whenever you find the time to do so.

glatterf42 commented 1 year ago

Hi again @SiddharthJoshi-Git, I have a few more questions before updating the installation instructions:

Looking at the miniconda installers, there are M1 ARM installers. Did they fail for you so that you had to change to the Intel x86 64-bit pkg?
Which conda channels did you end up using? Could you please post the output of conda list here?
Which command did you use to change the default conda channels?
Finally, what are the default conda channels for you? Could you please simulate starting the process from scratch and check that?

SiddharthJoshi-Git commented 1 year ago

Hi Fridolin,

- Looking at the miniconda installers, there are M1 ARM installers.Did they fail for you so that you had to change to the Intel x86 64-bit pkg?

There are currently two methods of installing anaconda/miniconda on macOS with ARM chip like APPLE M1/M2. The first method relies on installing M1/M2 native arm installer that defaults to pkgs/main/osx-arm64/. The second method relies on X86 installer that defaults to /pkgs/main/osx-64/. I tried installing using first method initially which lead to the original errors. Switching to second method lead to full compatibility using rosetta translation layer (this is automatic and we don't change any setting here). I don't yet know if we can install using first method and change packages to X86 on the fly. I presume that with first method, the python version being downloaded during venv initialisation is ARM one instead of X86. Links: https://www.anaconda.com/blog/apple-silicon-transition/ https://www.anaconda.com/blog/new-release-anaconda-distribution-now-supporting-m1 https://docs.anaconda.com/free/anaconda/reference/packages/old-pkg-lists/2022.05/py3.8_osx-arm64/

- Which conda channels did you end up using? Could you please post the output of conda list here?

With X86 installer, I did not change any default channel. The installer automatically pointed to /pkgs/main/osx-64/ repository. Some outputs:

- Which command did you use to change the default conda channels? Please see above. To summarise, the default channel is _**conda-forge/pypi**_, and the repository is _**https://repo.anaconda.com/pkgs/main/osx-64**_. No command was used to change the conda default channels and the installer takes care of this automatically

- Finally, what are the default conda channels for you? Could you please simulate starting the process from scratch and check that? Please see above. Attached is the output of conda list

Maybe a test is still required to check if ARM installer can be coerced to install packages from X86 repository. This way all the native packages can be ARM (Faster speed) and all the MESSAGEix requirements come from X86 builds.

glatterf42 commented 1 year ago

Thanks a lot, I updated the installation instructions accordingly.

setupelz commented 5 months ago

Steps to enable MESSAGEix model development on Apple (ARM based CPU) --- Verified 11th April 2024.

1.) Install GAMS 46.4 using the package installer for macOS on Intel CPUs. Ensure you select 'customize' during installation and set the tick box to add to your Path variable, your .zprofile should have an entry that looks like this after installation:

PATH="/Library/Frameworks/GAMS.framework/Resources:${PATH}" export PATH

Manual edits are likely to fail.

2.) Install Miniconda (Miniconda3 macOS Intel x86 64-bit pkg)- https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.pkg

3.) Create and activate conda environment (Python version 3.10) - $ conda create --name ME_X86_P3.10 python=3.10 $ conda activate ME_X86_P3.10

4.) Check that the packages being installed next are coming from the "pkgs/main/osx-64" conda channel using $ conda info

5.) Fork or clone and install ixmp from source - https://docs.messageix.org/projects/ixmp/en/latest/install.html#from-source

6.) Install open JDK (ver. conda-forge/osx-64::openjdk-11.0.9.1-hcf210ce_1) using $ conda install -c conda-forge openjdk

7.1) Run $ ixmp show-versions. This should show gams 46.4. 7.2) Navigate to ixmp repository and run $ pytest. This should pass.

8.) Fork or clone and install MESSAGEix from source - https://docs.messageix.org/en/stable/install.html#from-source

9.) Install pyam - $ pip install pyam-iamc

10.) Install Jupyterlab - $ conda install -c conda-forge jupyterlab

11.) Navigate to message_ix repository and run $ pytest. This should pass.

iiasa / ixmp

IXMP on Apple Silicon (ARM architecture) #473

Code sample or context