search

MHKiT-Software / MHKiT-MATLAB

MHKiT-MATLAB provides the marine renewable energy (MRE) community tools for data processing, visualization, quality control, resource assessment, and device performance.
https://mhkit-software.github.io/MHKiT/
BSD 3-Clause "New" or "Revised" License
15 stars 23 forks source link

MHKiT with MATLAB R2022b , Anaconda, Python 3.9, on Ubuntu 22.04 LTS #129

Open pstansell opened 6 months ago

pstansell commented 6 months ago

I'm trying to get MHKiT working with MATLAB R2022b and Anaconda with Python 3.9 on Ubuntu 22.04 LTS, but I'm having some problems.

I cloned both MHKiT-Python and MHKiT-MATLAB and then installed them both (i.e., mhkit and mhkit-python-utils) into my Anaconda environment using pip install -e .

Then I start MATLAB R2022b and installed the MHKiT toolbox which reports success, however, the following does not work in MATLAB:

[x,y]=circular(30)

Error using _linprog_highs><module>
Python Error: ImportError: /opt/MATLAB/R2022b/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6:
version `GLIBCXX_3.4.29' not found (required by
~/anaconda3/lib/python3.9/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-39-x86_64-linux-gnu.so)

I have GLIBCXX_3.4.29 in ~/anaconda3/lib/libstdc++.so.6 but GLIBCXX_3.4.28 is highest version in /opt/MATLAB/R2022b/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6

If I try to tell MATLAB to use ~/anaconda3/lib/libstdc++.so.6 by doing

LD_PRELOAD=/home/ps/anaconda3/lib/libstdc++.so.6 /opt/MATLAB/R2022b/bin/matlab

MATLAB crashes when I run the circular(30) command.

I also tried to make the pip install build against /opt/MATLAB/R2022b/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6 by uninstalling mhkit, mhkit-python-utils and the MHKit toolbox and repeating the whole process above after first setting

export LD_PRELOAD=/opt/MATLAB/R2022b/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6

However, this made no difference to the error message above.

I'd welcome any suggestions on how to fix the problem or what to try next, thanks.

simmsa commented 6 months ago

Hi @pstansell, thank you for the detailed issue report.

You did everything right. We have seen similar failures with our unit tests and the root cause seems to be issues between the MATLAB, Python, and system path.

Could you try the following troubleshooting steps:

  1. Close MATLAB
  2. Create a new anaconda environment
  3. In this environment:
    • Install C based python packages with conda
      • conda install numpy cython pip pytest hdf5 libnetcdf cftime netcdf4
    • Install the previous version of MHKiT-Python
      • pip install mhkit==0.7.0
    • Install the latest version of mhkit_python_utils
      • pip install -e . inside of the MHKiT-MATLAB folder
  4. Verify that mhkit-python is working:
    • python -c "import mhkit; print(mhkit.__version__)"
    • Output should v0.7.0
  5. Install the MHKiT-MATLAB add-on
    • Restart MATLAB
    • Double click on mhkit.mltbx
    • Verify the version is 0.4.1 in your MATLAB add-ons
  6. Find your conda environment python executable:
    • python -c "import sys; print(sys.executable)
  7. Setup MATLAB python to use python out of process
    • In MATLAB Command Window:
    • pyenv(Version="<path to python executable>", ExecutionMode="OutOfProcess")
  8. Test MHKiT-MATLAB
    • In MATLAB Command Window:
    • [x, y] = circular(30)

Expected output is (or similar, not sure why my MATLAB is using exponential notation here)

>> [x, y] = circular(30)

x =

    30

y =

     7.068583470577034e+02

In MATLAB using Python OutOfProcess seems to fix some MATLAB <-> Python issues related to library access.

Let us know if this is helpful!

pstansell commented 6 months ago

Hello @simmsa,

Thanks for your detailed instructions on what to try. I followed them carefully, but it didn't seem to make any difference to the results I had before. Following is a summary of what I did and the outputs and errors.

Created and activated a new (Python 3.9) Anaconda environment with

bash Anaconda3-2022.10-Linux-x86_64.sh
eval "$(/home/ps/anaconda3/bin/conda shell.bash hook)"

Installed packages with

conda install numpy cython pip pytest hdf5 libnetcdf cftime netcdf4

  Collecting package metadata (current_repodata.json): done
  Solving environment: done

  ## Package Plan ##

    environment location: /home/ps/anaconda3

    added / updated specs:
      - cftime
      - cython
      - hdf5
      - libnetcdf
      - netcdf4
      - numpy
      - pip
      - pytest

  The following packages will be downloaded:

      package                    |            build
      ---------------------------|-----------------
      cftime-1.6.2               |   py39h7deecbd_0         263 KB
      hdf4-4.2.13                |       h3ca952b_2         714 KB
      libnetcdf-4.8.1            |       h42ceab0_1         1.0 MB
      libzip-1.8.0               |       h5cef20c_0         102 KB
      netcdf4-1.6.2              |   py39h89d13dc_0         505 KB
      ------------------------------------------------------------

  The following NEW packages will be INSTALLED:

    cftime             pkgs/main/linux-64::cftime-1.6.2-py39h7deecbd_0 None
    hdf4               pkgs/main/linux-64::hdf4-4.2.13-h3ca952b_2 None
    libnetcdf          pkgs/main/linux-64::libnetcdf-4.8.1-h42ceab0_1 None
    libzip             pkgs/main/linux-64::libzip-1.8.0-h5cef20c_0 None
    netcdf4            pkgs/main/linux-64::netcdf4-1.6.2-py39h89d13dc_0 None

  Proceed ([y]/n)? y

Installed MHKit with

pip install mhkit==0.7.0
    Successfully installed NREL-rex-0.2.79 adal-1.2.7 azure-core-1.30.1 cachetools-4.2.4 fatpack-0.7.7 google-api-core-2.10.2 google-api-python-client-2.129.0 google-auth-1.35.0 google-auth-httplib2-0.2.0 google-auth-oauthlib-0.5.3 googleapis-common-protos-1.63.0 h5pyd-0.10.3 httplib2-0.22.0 isodate-0.6.1 mhkit-0.7.0 msrest-0.7.1 msrestazure-0.6.4.post1 netCDF4-1.5.8 oauthlib-3.2.2 pecos-0.3.1 protobuf-4.25.3 requests-oauthlib-2.0.0 requests-unixsocket-0.3.0 rsa-4.9 typing-extensions-4.11.0 uritemplate-4.1.1

Installed mhkit-python-utils with

cd ~/git_clones/MHKiT-MATLAB
pip install -e .
  Successfully installed mhkit-python-utils-0.2.0

Checked MHKiT version and path with

python -c "import mhkit; print(mhkit.__version__)"
  v0.7.0

python -c "import sys; print(sys.executable)"
  /home/ps/anaconda3/bin/python

Started MATLAB with

/opt/MATLAB/R2022b/bin/matlab

I reinstalled the MTKiT toolbox and then I ran MATLAB commands

pyenv(Version="/home/ps/anaconda3/bin/python", ExecutionMode="OutOfProcess")

ans = 

  PythonEnvironment with properties:

          Version: "3.9"
       Executable: "/home/ps/anaconda3/bin/python"
          Library: "/home/ps/anaconda3/lib/libpython3.9.so"
             Home: "/home/ps/anaconda3"
           Status: NotLoaded
    ExecutionMode: OutOfProcess

>> [x, y] = circular(30)
  Error using _linprog_highs><module>
  Python Error: ImportError:
  /opt/MATLAB/R2022b/interprocess/bin/glnxa64/pycli/../../../../sys/os/glnxa64/libstdc++.so.6:
  version `GLIBCXX_3.4.29' not found (required by
  /home/ps/anaconda3/lib/python3.9/site-packages/scipy/optimize/_highs/_highs_wrapper.cpython-39-x86_64-linux-gnu.so)
simmsa commented 6 months ago

@pstansell, thank you for the detailed output. Looking further into your specific error it could be due to an older version of pandas that we are using. This issue suggests that running pip install --upgrade pandas in your mhkit conda environment may fix your issue. If that does not work I suggest trying the fixes suggested here next.

simmsa commented 6 months ago

@pstansell, this Answer from Mathworks also looks like a promising solution. These types of MATLAB/Python errors are frustrating, but IMO you are very close to getting this working.

We do have tests on ubuntu verifying that you should be able to install MHKiT-MATLAB. They are based on this GitHub Actions workflow file. There may be some commands there that may help you troubleshoot this further.

pstansell commented 6 months ago

@simmsa, based on the various suggestions, I've found a solution that works with MATLAB R2022b (but not with the latest MATLAB version R2024a). Here is a summary.

Install Anaconda with

bash Anaconda3-2022.10-Linux-x86_64.sh

Activate the Anaconda environment and install some packages with

conda install cftime hdf4 libnetcdf libzip netcdf4

Install MHKiT with

pip install mhkit==0.7.0

Install MHKiT-MATLAB with

git clone https://github.com/MHKiT-Software/MHKiT-MATLAB
cd MHKiT-MATLAB
pip install -e .

Change a MALTAB library link with

cd /opt/MATLAB/R2022b/sys/os/glnxa64
mv libstdc++.so.6 libstdc++.so.6_original
ln -s /home/ps/anaconda3/lib/libstdc++.so.6.0.29 libstdc++.so.6

Start MATLAB with

/opt/MATLAB/R2022b/bin/matlab

In MATLAB, install the MHKiT toolbox MHKiT-MATLAB//mhkit.mltbx and then run the MHKiT test command

>> [x, y] = circular(30)
/home/ps/anaconda3/lib/python3.9/site-packages/mhkit/tidal/resource.py:96: SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert(all(directions>=0) and all(directions<=360),

x =

    30

y =

  706.8583

I'd previously tried something very similar with MATLAB R2024a, but it didn't work. I tried again with exactly the same steps as above but MATLAB R2024a crashed when I ran the circular with the following error

>> [x,y]=circular(30)
/home/ps/anaconda3/lib/python3.9/site-packages/h5py/__init__.py:36: UserWarning: h5py is running against HDF5 1.8.12 when it was built against 1.10.6, this may cause problems
  _warn(("h5py is running against HDF5 {0} when it was built against {1}, "
simmsa commented 6 months ago

@pstansell, nice work finding a solution! We appreciate your detailed troubleshooting efforts, and we are planning to incorporate your findings into an updated version of the MHKiT-MATLAB installation instructions.

We appreciate you bring this issue to our attention and working through it. To ensure the accuracy of MHKiT's output, we rely on MHKiT-Python as the reference for all computations, calling its functions from within MATLAB. As a result, to use MHKiT-MATLAB users must install MHKiT-Python and understand how to correctly configure MATLAB to use Python. This can be a barrier to entry for using MHKiT-MATLAB. If you have any thoughts or suggestions on how we can streamline this process, we would greatly value your feedback.

A more robust potential solution is manually adding the Anaconda executable directory to the first position MATLAB path. Internally, it seems like MATLAB adjusts the path to include MATLAB-specific binaries. This solution adjusts the path to put the Anaconda binaries first, allowing Python to correctly find the prebuilt binaries. Although you're not obligated to try this, as the first Linux user to report an issue, your input could help us devise a robust solution for all Linux users.

Below are the steps to add the Anaconda Python binaries to the MATLAB path:

  1. Close MATLAB.

  2. Locate the Anaconda environment bin directory:

    Inside your MHKiT-Python Anaconda environment, in the terminal, determine the location of the Python binary directory by running:

    python -c "import sys; import os; print(os.path.dirname(sys.executable))"

  3. Restart MATLAB and add the output directory from the previous step to the first position in the MATLAB path:

    setenv('path', ['<path to Anaconda dir>', getenv('path')])

  4. Reset the MATLAB Python Environment:

    Inside your MHKiT-Python Anaconda environment, in the terminal, determine the location of the Python executable by running:

    python -c "import sys; print(sys.executable)"

    In the MATLAB Command Window, execute:

    pyenv(Version="<path to Anaconda Python executable>", ExecutionMode="OutOfProcess")

  5. Run the circular example again.