search

BYU-PRISM / GEKKO

GEKKO Python for Machine Learning and Dynamic Optimization
https://machinelearning.byu.edu
Other
573 stars 102 forks source link

Library not loaded: libquadmath #102

Closed talsaiag closed 3 years ago

talsaiag commented 3 years ago

Error running using local m = Gekko(remote=False)

Error: dyld: Library not loaded: /usr/local/opt/gcc/lib/gcc/9/libquadmath.0.dylib
  Referenced from: /Users/<my-local-user>/Library/Python/3.9/lib/python/site-packages/gekko/bin/apm_mac
  Reason: image not found

and then of course (a little less indicative error):

Error: 'results.json' not found. Check above for additional error details
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)

--> 158                 m.solve()

~/Library/Python/3.9/lib/python/site-packages/gekko/gekko.py in solve(self, disp, debug, GUI, **kwargs)
   2143         if timing == True:
   2144             t = time.time()
-> 2145         self.load_JSON()
   2146         if timing == True:
   2147             print('load JSON', time.time() - t)

~/Library/Python/3.9/lib/python/site-packages/gekko/gk_post_solve.py in load_JSON(self)
     11 ## options.JSON has all APM options
     12 def load_JSON(self):
---> 13     f = open(os.path.join(self._path,'options.json'))
     14     data = json.load(f)
     15     f.close()

FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/v3/433gqqkj4fd3yl1ljdzdb_lh0000gn/T/tmpsw80nnazgk_model0/options.json'

Same code worked on same machine before upgrading from python3.8 to python3.9 (using brew install python3)

Environment:

APMonitor commented 3 years ago

We now have an Apple M1 processor (MacBook Pro) for testing and development to provide a local executable. Until then, there is still the public server available with remote=True or you can install an APM Linux or Windows server and run locally without an Internet connection with remote=True and server='http://10.0.0.10' with your local server address. The target is to release the new version of the local executable with dual ARM and x86_64 support. One of the complications is that gcc is still under development for the new platform: https://developer.apple.com/forums/thread/651476

talsaiag commented 3 years ago

Note: I am using 2.9 GHz 6-Core Intel Core i9 and not the new apple processor. This worked as expected right before I update the python on the same host

talsaiag commented 3 years ago

@APMonitor I think that the issue is the missing libquadmath library which is not statically compiled with the apm_mac and is is not found in the gcc version that the python3 was compiled for using the brew's gcc

talsaiag commented 3 years ago

I know that it works correctly on Python3.9 (on another intel macbook)

Python 3.9.0 (default, Oct 27 2020, 14:15:17)
[Clang 12.0.0 (clang-1200.0.32.21)] on darwin

Where he has /usr/local/opt/gcc/lib/gcc/9

But not on mine:

Python 3.9.1 (default, Jan  6 2021, 12:44:33)
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

where I have /usr/local/opt/gcc/lib/gcc/10 only

talsaiag commented 3 years ago

@APMonitor is there a way to compile the binary locally or make it statically linked?

APMonitor commented 3 years ago

I've been working with the Apple M1 MacBook Pro but the gcc compilers aren't ready yet to provide a universal architecture executable. I'll let you know as soon as one is available. An alternative is to create a Windows or Linux APM server and then use that server locally with m = GEKKO(remote=True, server='http://10.0.0.10') with the correct local IP address. I'll make it a point to statically link all the libraries on this next release.

bjrnfrdnnd commented 3 years ago

I tried to install a Linux server; it did not work. For example, instructions on the referenced webpage are wrong: sudo cp -n apm_linux/lib/* /usr/lib/ (there is no lib Directory on https://github.com/APMonitor/apm_server.git). When running the python test, the system does not succeed because it cannot load the file "results.csv" which should've presumably been produced by running apm(s,a). At this moment, the suggested workaround does not work. Or did I do something wrong?

APMonitor commented 3 years ago

Another way to run Gekko locally is to run with remote=False on Linux or Windows. The /lib file is only needed when IPOPT is linked with the executable. You can skip that step for your install. The results.csv file is generated when you issue a solve command.

bjrnfrdnnd commented 3 years ago

Thanks.

Do you mind trying to help me installing a local server? I can give you the commands I used and the errors/observations that follow. No results.csv is produced; an error log is written in /var/log/apache2/error.log :

[Sat Apr 03 15:33:14.926845 2021] [mpm_prefork:notice] [pid 1736] AH00163: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations [Sat Apr 03 15:33:14.926879 2021] [core:notice] [pid 1736] AH00094: Command line: '/usr/sbin/apache2' [Sat Apr 03 15:37:58.714331 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: mkdir(): No such file or directory in /var/www/html/apm_server/apm_linux/online/apm_line.php on line 24 [Sat Apr 03 15:37:58.714371 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: fopen(127.0.0.1_test_python/127.0.0.1_test_python.apm): failed to open stream: No such file or directory in /var/www/html/apm_server/apm_linux/online/apm_line.php on line 38 [Sat Apr 03 15:37:58.714379 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: fclose() expects parameter 1 to be resource, bool given in /var/www/html/apm_server/apm_linux/online/apm_line.php on line 64 [Sat Apr 03 15:37:58.714389 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: opendir(/var/www/html/online/127.0.0.1_test_python/): failed to open dir: No such file or directory in /var/www/html/apm_server/apm_linux/online/rrmdir.php on line 4 [Sat Apr 03 15:37:58.714394 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: readdir() expects parameter 1 to be resource, bool given in /var/www/html/apm_server/apm_linux/online/rrmdir.php on line 6 [Sat Apr 03 15:37:58.714405 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: unlink(/var/www/html/online/127.0.0.1_test_python/): No such file or directory in /var/www/html/apm_server/apm_linux/online/rrmdir.php on line 10 [Sat Apr 03 15:37:58.714410 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: readdir() expects parameter 1 to be resource, bool given in /var/www/html/apm_server/apm_linux/online/rrmdir.php on line 6 [Sat Apr 03 15:37:58.714417 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: unlink(/var/www/html/online/127.0.0.1_test_python/): No such file or directory in /var/www/html/apm_server/apm_linux/online/rrmdir.php on line 10 ...

The error log coninues with a repetitition of the last two lines until the disk is full: [Sat Apr 03 15:37:58.714498 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: readdir() expects parameter 1 to be resource, bool given in /var/www/html/apm_server/apm_linux/online/rrmdir.php on line 6 [Sat Apr 03 15:37:58.714513 2021] [php7:warn] [pid 1757] [client 127.0.0.1:44816] PHP Warning: unlink(/var/www/html/online/127.0.0.1_test_python/): No such file or directory in /var/www/html/apm_server/apm_linux/online/rrmdir.php on line 10

APMonitor commented 3 years ago

Does it work to put the directory /var/www/html/apm_server/apm_linux/online/ in /var/www/html/online/ instead?

bjrnfrdnnd commented 3 years ago

No. Still the same error: [Sun Apr 04 17:18:14.338475 2021] [php7:warn] [pid 317093] [client 127.0.0.1:37722] PHP Warning: unlink(/var/www/html/online/127.0.0.1_test_python/): No such file or directory in /var/www/html/online/rrmdir.php on line 10 [Sun Apr 04 17:18:14.338479 2021] [php7:warn] [pid 317093] [client 127.0.0.1:37722] PHP Warning: readdir() expects parameter 1 to be resource, bool given in /var/www/html/online/rrmdir.php on line 6

APMonitor commented 3 years ago

Those look like warnings. Can you see if the directory: /var/www/html/online/127.0.0.1_test_python/ is created and if it contains the .apm file or result.csv file? It looks like you are running with Matlab?

bjrnfrdnnd commented 3 years ago

Thanks. No, python. No results.csv is produced. Not even the directory 127.0.0.1_test_python is produced.

(base) bn@bn-ubuntu-i7-focal:/var/www/html$ ls /var/www/html/online/ apm_line.php apm_t0.php dbs.php get_tag.php index.php info.php meas.php plot.php plot_select.php plot_top.php plt.php rrmdir.php view_pass.php (base) bn@bn-ubuntu-i7-focal:/var/www/html$ ls /var/www/html/apm_server/apm_linux/online/ apm_line.php apm_t0.php dbs.php get_tag.php index.php info.php meas.php plot.php plot_select.php plot_top.php plt.php rrmdir.php view_pass.php

bjrnfrdnnd commented 3 years ago

log: [Mon Apr 05 19:46:19.155956 2021] [php7:warn] [pid 954529] [client 127.0.0.1:33884] PHP Warning: unlink(/var/www/html/online/127.0.0.1_test_python/): No such file or directory in /var/www/html/online/rrmdir.php on line 10 [Mon Apr 05 19:46:19.155960 2021] [php7:warn] [pid 954529] [client 127.0.0.1:33884] PHP Warning: readdir() expects parameter 1 to be resource, bool given in /var/www/html/online/rrmdir.php on line 6

APMonitor commented 3 years ago

It looks like a permission problem. Could you run ls -la in the directory /var/www/html/online and for the online directory to ensure that the php files can write a new directory in that folder? You can test that a new folder is created with this Gekko script:

from gekko import GEKKO
m = GEKKO(server='http://127.0.0.1',remote=True)
x = m.Var()            # define new variable, default=0
y = m.Var()            # define new variable, default=0
m.Equations([3*x+2*y==1, x+2*y==0])  # equations
m.solve(disp=False)    # solve
print(x.value,y.value) # print solution

There may also be a write permission error with Ubuntu. This thread discusses some of the issues and potential solutions: https://askubuntu.com/questions/246437/how-to-give-apache2-permission-to-write-and-read-files

APMonitor commented 3 years ago

The local executable in Gekko v1.0.0 now runs without dependencies. Please let us know if there are any additional issues.

jerrytim commented 2 years ago

happy to see that the new version Gekko v1.0.0 resolved this issue. I have came across this issue so many times, and have this workaround:

  1. Navigate to usr/local/opt/gcc/lib/gcc
  2. Run the command cp -R 11 9

but yeah this is very annoying should be fixed in new version