search

BYU-PRISM / GEKKO

GEKKO Python for Machine Learning and Dynamic Optimization
https://machinelearning.byu.edu
Other
612 stars 107 forks source link

Getting OSError: [Errno 8] Exec format error #177

Open todorvelichkov opened 9 months ago

todorvelichkov commented 9 months ago

On AWS t4g.medium machine with Graviton2 CPU we are getting the following error:

Traceback (most recent call last):
    # ...
    model.solve(disp=False)
  File "/srv/.virtualenvs/python3.11/lib/python3.11/site-packages/gekko/gekko.py", line 2097, in solve
    app = subprocess.Popen([apm_exe, self._model_name], stdout=subprocess.PIPE, \
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/srv/.virtualenvs/python3.11/lib/python3.11/site-packages/gekko/bin/apm'

The model initialization looks like this:

# Initialize the model
model = GEKKO(remote=False)

# Add Variables
# ...

# Add Objective Function
# ...

# Add Constraints
# ...

# Solve the model
model.options.SOLVER = 1
model.solve(disp=False)

The OS is Ubuntu 22.04.2 LTS, Python is 3.11.7 and gekko==1.0.6

Any help would be appreciated.

UPDATE

I found issue #160, tried to install gekko with the following command: pip install -e git+https://github.com/BYU-PRISM/GEKKO.git@e6c8121015611907003f86cffdc9d8ad11d6da3d#egg=gekko

After that I got:

PermissionError: [Errno 13] Permission denied: '/srv/.virtualenvs/python3.11/src/gekko/gekko/bin/apm_aarch64'

so I updated the permissions of the apm_aarch64 because they looked like this:

(python3.11) $ ls -l  /srv/.virtualenvs/python3.11/src/gekko/gekko/bin/
total 25248
-rw-rw-r-- 1 ubuntu ubuntu     450 Feb 19 13:05 README.md
-rwxrwxr-x 1 ubuntu ubuntu 7082704 Feb 19 13:05 apm
-rwxrwxr-x 1 ubuntu ubuntu 8675840 Feb 19 13:05 apm.exe
-rwxrw-r-- 1 ubuntu ubuntu 1979360 Feb 19 13:05 apm_aarch64
-rw-rw-r-- 1 ubuntu ubuntu 1830772 Feb 19 13:05 apm_arm
-rwxrwxr-x 1 ubuntu ubuntu 6300032 Feb 19 13:05 apm_mac

but now I'm getting:

Error: 'results.json' not found. Check above for additional error details
Traceback (most recent call last):
    # 
    model.solve(disp=False)
  File "/srv/.virtualenvs/python3.11/src/gekko/gekko/gekko.py", line 2252, in solve
    self.load_JSON()
  File "/srv/.virtualenvs/python3.11/src/gekko/gekko/gk_post_solve.py", line 13, in load_JSON
    f = open(os.path.join(self._path,'options.json'))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmprbbkkiw2gk_model0/options.json'

This is how the folder looks like:

(python3.11) $ ls -l /tmp/tmprbbkkiw2gk_model0
total 20
-rw-rw-r-- 1 ubuntu ubuntu 8693 Feb 19 13:09 gk_model0.apm
-rw-rw-r-- 1 ubuntu ubuntu  711 Feb 19 13:09 gk_model0.csv
-rw-rw-r-- 1 ubuntu ubuntu    0 Feb 19 13:09 gk_model0.info
-rw-rw-r-- 1 ubuntu ubuntu 3121 Feb 19 13:09 measurements.dbs
APMonitor commented 9 months ago

Thanks for including the follow-up comment. The executable for aarch64 will be available with the new release. Could you also try:

pip install https://github.com/BYU-PRISM/GEKKO/archive/master.zip

Include ---upgrade if there is an existing installation. The new release has a version that should run on the 64-bit ARM processors. It appears that there is still a problem with Graviton2 CPU. I'll leave this issue open until it is resolved.

One way around this is to set up your own APM Linux server on Amazon and then use m=gekko(remote=True,server='http://10.0.0.10') and replace the IP address with the AWS EC2 server address. Another option is to use m=gekko(remote=True) and the public server will perform the calculations if it isn't overloaded.

todorvelichkov commented 9 months ago

Could you also try: pip install https://github.com/BYU-PRISM/GEKKO/archive/master.zip

I'm not sure why, but for some reason there is no apm_aarch64 file if I install the package that way. Here is how the installation log looks like (there is no previous installation, because I uninstall it).

$ pip install --upgrade https://github.com/BYU-PRISM/GEKKO/archive/master.zip
Collecting https://github.com/BYU-PRISM/GEKKO/archive/master.zip
  Downloading https://github.com/BYU-PRISM/GEKKO/archive/master.zip
     \ 16.6 MB 8.9 MB/s 0:00:02
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy>=1.8 in /srv/.virtualenvs/python3.11/lib/python3.11/site-packages (from gekko==1.0.7) (1.26.4)
Building wheels for collected packages: gekko
  Building wheel for gekko (setup.py) ... done
  Created wheel for gekko: filename=gekko-1.0.7-py3-none-any.whl size=12228119 sha256=25dc12453d7167e213aa35f83268531ee4c441a02a8a9ed09c07f30e2e7627e7
  Stored in directory: /tmp/pip-ephem-wheel-cache-rci880bn/wheels/79/0e/ba/0dec06fd0e989c0464e1e0e41257721b8e99ba171d03bf80c2
Successfully built gekko
Installing collected packages: gekko
Successfully installed gekko-1.0.7

The error after that is that the apm_aarch64 file is missing.

Traceback (most recent call last):
    # ...
    model.solve(disp=False)
  File "/srv/.virtualenvs/python3.11/lib/python3.11/site-packages/gekko/gekko.py", line 2123, in solve
    app = subprocess.Popen([apm_exe, self._model_name], stdout=subprocess.PIPE, \
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1950, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/srv/.virtualenvs/python3.11/lib/python3.11/site-packages/gekko/bin/apm_aarch64'

This is how the bin folder looks like:

$ ls -l /srv/.virtualenvs/python3.11/lib/python3.11/site-packages/gekko/bin/
total 23340
-rwxrwxr-x 1 ubuntu ubuntu 7082704 Feb 20 07:24 apm
-rwxrwxr-x 1 ubuntu ubuntu 8675840 Feb 20 07:24 apm.exe
-rw-rw-r-- 1 ubuntu ubuntu 1830772 Feb 20 07:24 apm_arm
-rwxrwxr-x 1 ubuntu ubuntu 6300032 Feb 20 07:24 apm_mac

One way around this is to set up your own APM Linux server on Amazon and then use m=gekko(remote=True,server='http://10.0.0.10') and replace the IP address with the AWS EC2 server address. Another option is to use m=gekko(remote=True) and the public server will perform the calculations if it isn't overloaded.

We will probably try to do that.

The new release has a version that should run on the 64-bit ARM processors. It appears that there is still a problem with Graviton2 CPU. I'll leave this issue open until it is resolved.

I will try to help with whatever I can until the issue is resolved.

APMonitor commented 9 months ago

Thanks for reporting that. I needed to include apm_aarch64 in the setup.py file so now it lists the extra files that should be included with the pip install:

extra_files = gui_files + ['bin/apm.exe','bin/apm','bin/apm_aarch64','bin/apm_arm','bin/apm_mac']

From your prior message, it sounds like we still need a custom executable for Graviton2 CPUs anyway. I've added this issue as a bug report to be resolved for a future release.