trouble to install on OSX 10.12.6

Hello,

I have some problems installing on Sierra, not the same as #11

I first installed gsl and compiled it. I had to change the permissions for the library and include directories after the make install (maybe useless) and I had to create a symlink to /usr/local/include/gsl inside /src to have the compiler find the gsl*.h files. Why do I need to do that since gsl was installed in its default directory ?

I used gcc = gcc.8.2.0 installed via macports so that openmp is accepted (with setenv CC = gcc in tcsh)

make, make install, and make idlwrapper seem to work.

make pywrapper fails with a message which seems to say that it is unhappy with the extremedeconvolution.dylib it found.

The error came from the fact that make install did not give the permissions to the user to access the library!

make rpackage fails: clang: error: unsupported option '-fopenmp'

why is it calling clang instead of gcc like the rest of the Makefile ?

I can't run IDL because the test tries to run it into sh and in my installation, IDL has to be setup first before being launched. I have launched it by hand but it failed:

% Compiled module: NEAREST_NEIGHBORS. % Compiled module: MATCH_2D. % Compiled module: REPCHR. % Compiled module: GET_HFIBOLO_LIST. % Compiled module: CO_CONVERSION. % Compiled module: HFI_CO_CORRECTION. % Compiled module: PLANCK_RAD_PUBLIC. % Compiled module: MOD_BLACKBODY. % Compiled module: COLOUR_CORRECTION. % Compiled module: COLOR_CORRECTION. % Compiled module: HFI_COLOUR_CORRECTION. % Compiled module: HFI_COLOR_CORRECTION. % Compiled module: HFI_LFI_TEST_SCRIPT_PLA_RIMO. % Compiled module: HFI_READ_AVG_BANDPASS. % Compiled module: HFI_READ_BANDPASS. % Compiled module: UNIT_CONVERSION. % Compiled module: HFI_UNIT_CONVERSION.

tfdata= read_ascii('TF.dat',num_records=nline,template=template) ^ % Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 42

eigenvals= EIGENQL(xcovar,eigenvectors=eigenvectors)
                                       ^

% Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 117

    eigenvals= EIGENQL(xcovar,eigenvectors=eigenvectors)
                                           ^

% Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 146

squared_err= (ndata-1.)^2./double(ndata)*VARIANCE(jack_slopes,/double)
                                                               ^
                                                               ^

% Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 150

squared_err= (ndata-1.)^2./double(ndata)*VARIANCE(jack_int,/double)
                                                            ^

% Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 152 % Compiled module: FIT_TF. % Attempt to call undefined procedure: 'FIT_TF'. % Execution halted at: $MAIN$

running pytest:

make testpy

it mentions a pair of errors:

File "extreme_deconvolution.py", line 348, in main.extreme_deconvolution Failed example: ydata.flags['C_CONTIGUOUS'] Expected: False Got: True

Thanks for any suggestion/help

(continuing)

I tried to run the idl test by hand but it also fails. It compiles all the .pro in the /pro directory but the compilation or execution of fit_tf fails:

IDL> .compile fit_tf

tfdata= read_ascii('TF.dat',num_records=nline,template=template) ^ % Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 42

eigenvals= EIGENQL(xcovar,eigenvectors=eigenvectors)
                                       ^

% Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 117

    eigenvals= EIGENQL(xcovar,eigenvectors=eigenvectors)
                                           ^

% Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 146

squared_err= (ndata-1.)^2./double(ndata)*VARIANCE(jack_slopes,/double)
                                                               ^

% Syntax error. At: /Users/laurent/XNICER/extreme-deconvolution/examples/fit_tf.pro, Line 150

Suggestions welcome.

Sorry you are having these difficulties. Without seeing what happened during the installation, it's difficult to know what might be going wrong here (the IDL errors seem unrelated to the package, given that they are all syntax errors; I haven't used IDL in eight years, so cannot help out with this really). Please start from a fresh version of the code and post all of the installation output (e.g., using pastebin if there is too much output).

One normally tells the compiler where the GSL library can be found using the CFLAGS and LDFLAGS environment variables (see the installation FAQ in the README).

Hi Jo,

sorry to not have answered earlier. I was trying to install several pieces of software in parallel and ended up in a mess. Starting from scratch again, it all worked but I have some problems left :

1) There is a extreme-deconvolution-1.4 version which does not compile correctly (among other things the Wl option has turned into a W1 —Wone— option but correcting that was not enough) and in the python installer it says it is still version 1.3.

2) Returning to extreme-deconvolution-master version, the python setup.py does not seem to work (it cannot find the library at the end, one has still to go through make/make install).

3) On my Mac with OSX 10.12.6 with Python 3.6, the openmp option hardly helps : the maximum it gets is 117% of CPU. I just installed it on a OSX 10.13 with Python 3.7 and the maximum it gets is 148% of CPU. A colleague of mine mentioned that he could obtain full CPU usage on his own Mac. I use gcc-8.3 to compile your program which knows openmp and the option -fopenmp is visible in the execution of your Makefile :

extreme-deconvolution-master>make gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/bovy_randvec.c -o src/bovy_randvec.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/calc_splitnmerge.c -o src/calc_splitnmerge.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/logsum.c -o src/logsum.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/minmax.c -o src/minmax.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/normalize_row.c -o src/normalize_row.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/proj_EM.c -o src/proj_EM.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/proj_EM_step.c -o src/proj_EM_step.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/proj_gauss_mixtures.c -o src/proj_gauss_mixtures.o -I src/ src/proj_gauss_mixtures.c: In function 'proj_gauss_mixtures': src/proj_gauss_mixtures.c:126:3: warning: 'tmpnam' is deprecated: This function is provided for compatibility reasons only. Due to security concerns inherent in the design of tmpnam(3), it is highly recommended that you use mkstemp(3) instead. [-Wdeprecated-declarations] char tmpfilename= tmpnam(NULL);; ^~~~ In file included from src/proj_gauss_mixtures.c:44: /usr/include/stdio.h:277:7: note: declared here char tmpnam(char ); ^~ src/proj_gauss_mixtures.c:60:21: warning: variable 'startgaussians' set but not used [-Wunused-but-set-variable] struct gaussian startgaussians; ^~~~~~ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/splitnmergegauss.c -o src/splitnmergegauss.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/bovy_det.c -o src/bovy_det.o -I src/ gcc -arch x86_64 -I /usr/local/include -fopenmp -fpic -Wall -c src/proj_gauss_mixtures_IDL.c -o src/proj_gauss_mixtures_IDL.o -I src/ gcc -dynamiclib -Wl,-single_module -o build/libextremedeconvolution.dylib \ -arch x86_64 -fopenmp \ src/bovy_randvec.o src/calc_splitnmerge.o src/logsum.o src/minmax.o src/normalize_row.o src/proj_EM.o src/proj_EM_step.o src/proj_gauss_mixtures.o src/splitnmergegauss.o src/bovy_det.o \ src/proj_gauss_mixtures_IDL.o \ -lm -lgsl -lgslcblas -lgomp 2>/dev/null; \ case "$?" in \ 0);; \ *) \ gcc -dynamiclib -Wl,-single_module -o build/libextremedeconvolution.dylib \ -arch x86_64 -fopenmp \ src/bovy_randvec.o src/calc_splitnmerge.o src/logsum.o src/minmax.o src/normalize_row.o src/proj_EM.o src/proj_EM_step.o src/proj_gauss_mixtures.o src/splitnmergegauss.o src/bovy_det.o \ src/proj_gauss_mixtures_IDL.o \ -lm -lgsl -lgslcblas ;; \ esac

If you have any idea how to obtain full CPU usage… I would be extremely glad.

Thanks, Laurent

"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème" (devise Shadok)

Dear Laurent,

1) There is a extreme-deconvolution-1.4 version which does not compile correctly (among other things > the Wl option has turned into a W1 —Wone— option but correcting that was not enough) and in the python installer it says it is still version 1.3.

Yes, I believe the latest release (which is very old) is broken, but this Wl issue should be fixed in master. I should really release a new version at some point...

2) Returning to extreme-deconvolution-master version, the python setup.py does not seem to work (it > cannot find the library at the end, one has still to go through make/make install).

Could you let me know in what exact way the python setup.py install does not work? #25 seems to be able to install with python setup.py (at least up until loading the module...). Is it the same issue?

3) On my Mac with OSX 10.12.6 with Python 3.6, the openmp option hardly helps : the maximum it gets is 117% of CPU. I just installed it on a OSX 10.13 with Python 3.7 and the maximum it gets is 148% of CPU. A colleague of mine mentioned that he could obtain full CPU usage on his own Mac. I use gcc-8.3 to compile your program which knows openmp and the option -fopenmp is visible in the execution of your Makefile :

I don't think there is much useful I can suggest here, beyond setting OMP_NUM_THREADS appropriately and trying to make sure that your machine isn't otherwise busy. The exact speed-up will depend on the problem, the number of CPUs, and whether you are running other processes at the time.

dear Jo,

Returning to extreme-deconvolution-master version, the python setup.py does not seem to work (it > cannot find the library at the end, one has still to go through make/make install). Could you let me know in what exact way the python setup.py install does not work? #25 https://github.com/jobovy/extreme-deconvolution/issues/25 seems to be able to install with python setup.py (at least up until loading the module...). Is it the same issue?

It runs smoothly but it does not find the library at the end when one tries to import extreme_deconvolution. I checked and the library was not installed in /usr/local/lib/. Don’t know if it should have been found somewhere else when only using python setup.py. I would have to deinstall extreme-dec to redo all the steps to tell you more because using the make/make install way, it works, so I don’t have error message anymore to show you.

If it is important to you, I can do that.

On my Mac with OSX 10.12.6 with Python 3.6, the openmp option hardly helps : the maximum it gets is 117% of CPU. I just installed it on a OSX 10.13 with Python 3.7 and the maximum it gets is 148% of CPU. A colleague of mine mentioned that he could obtain full CPU usage on his own Mac. I use gcc-8.3 to compile your program which knows openmp and the option -fopenmp is visible in the execution of your Makefile : I don't think there is much useful I can suggest here, beyond setting OMP_NUM_THREADS appropriately and trying to make sure that your machine isn't otherwise busy. The exact speed-up will depend on the problem, the number of CPUs, and whether you are running other processes at the time.

OMP_NUM_THREADS is set to 8, the machine is doing nothing fancy and I cannot go beyond 117%. I have written some code with openmp where I could get close to 800% CPU. So it is not a problem with my environment.

Cheers, Laurent

Hi Laurent,

Thanks for the extra info! When using the python setup.py option, the library should be installed where normal python packages go (something like /usr/local/lib/Python3.6/site-packages). But it looks like there was another problem with the python setup.py option that is now resolved (see #25).

I'm sorry about the parallelization not working better, but it's probably because your problem doesn't lend itself to much speed-up from this.

Hi Jo,

OK, let’s sum up everything :

1) I try to install a given software (xnicer by Marco Lombardi) which makes use of extreme_deconvolution 2) I try to install it on 3 different Macs under Python 3.6 and 3.7, sierra and high sierra, all python versions installed with Macports, and on two linux machines under Py3.5 and 3.6

3) When I first installed it on my Mac under Py36 and sierra, I did not meet the problem mentioned here ( #25 https://github.com/jobovy/extreme-deconvolution/issues/25). The problem appeared once and disappeared but I don’t remember what I did to make it disappear. After I reboot my mac or after I upgraded Macports, the problem became permanent and indeed the ( #25 https://github.com/jobovy/extreme-deconvolution/issues/25) fix fixes it. The problem is present with both py36 and py37.

4) on the two other macs, both under high sierra and python 3.7, the problem never appeared (but after having exercised on my mac, the install was straightforward, I did not have to make various attempts). Reboots do not change the situation.

5) on the linux machines, the bug is not present either.

6) if I don’t run make / make install on top of python setup.py install, I get this message when I try to run xnicer :

Traceback (most recent call last): File "map_2MASS.py", line 33, in from xnicer import * File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/xnicer-0.1.0-py3.7-macosx-10.12-x86_64.egg/xnicer/init.py", line 6, in from .xdcv import XDCV File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/xnicer-0.1.0-py3.7-macosx-10.12-x86_64.egg/xnicer/xdcv.py", line 6, in from extreme_deconvolution import extreme_deconvolution File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/extreme_deconvolution/init.py", line 1, in from .extreme_deconvolution import extreme_deconvolution File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/extreme_deconvolution/extreme_deconvolution.py", line 21, in raise IOError(_libraryname+' library not found') OSError: libextremedeconvolution.dylib library not found

Looking at the install_python log (which I attach here as install.log), it seems that you create a library named :

_extreme_deconvolution.cpython-37m-darwin.so

Is it the equivalent of libextremedeconvolution.dylib which is built via make/make install ? Because if it is, the python program still tries to find the libextremedeconvolution.dylib and not the python one since the extreme_deconvolution.py in the site-packages still looks for the .dylib. Therefore the problem seems to sit there.

On the other hand, as you discuss in #25, the "make pywrapper” command is only a local install inside extreme-deconvolution-master/ which is not found by Python if not added in the pythonpath (and there is no mention to do that in the INSTALL protocol).

7) this problem is identical on Linux machines (need both make/make install and python setup.py install)

8) the -no-openmp function does not work in the python install since the library which is built is not the one which is used. I had to mask the -fopenmp option in the Makefile to run the test without openmp.

9) the openmp issue :

9-a) the person who wrote xnicer has no problem running with openmp on his Mac, using all 12 cores when needed. 9-b) the test program takes approximately 2 min. on the Linux machines using all the available cores/threads (CPU usage : 4800%) 9-c) On my mac under Sierra, with py36, the program takes 1.5 hour, 80% of which is spent by the system with max usage of CPU of 115%

So there is really enough data crunching to do to use the openmp option at full speed of the multicore. It just does not work on my mac. Since it is too long to make speed tests with the original test program (too many stars), I switched to a smaller program afterwards :

10-a) Sierra/py36/openmp activated : 206.502u 1078.374s 18:38.42 114.8% 0+0k 1129+30io 14pf+0w 10-b) Linux machine/openmp activated : 35.896u 0.156s 0:14.63 246.3% 0+0k 0+1616io 0pf+0w 10-c) Sierra/py37/openmp activated : 29.414u 98.813s 1:49.48 117.1% 0+0k 951+43io 58pf+0w 10-d) Sierra/py36/openmp deactivated : 31.345u 1.453s 0:34.04 96.3% 0+0k 1580+284io 2371pf+0w 10-e) Sierra/py37/openmp deactivated : 11.945u 0.241s 0:12.31 98.9% 0+0k 4+28io 0pf+0w

The conclusion is obvious : when openmp is activated on my Mac or the two other macs, with py36 it is catastrophic, a huge amount of time is spent in swapping I guess. Total is almost 19 minutes. Linux machines run in 15 sec. with 246% CPU usage on average. Py37 is doing already much better, almost a factor 10 improvement but still spending a lot of time in system to swap. Masking the openmp option in extreme_deconvolution, the user time is divided by 7 and the system time by 1000, total real time becomes 34 sec. while with py37, we gain almost a factor 3 compared to py36/no openmp and to py37/openmp in user time and a factor 6 and 400 for system time respectively.

Therefore the openmp in my case is slowing the machine down…

I can tell you I had exactly the same problem but reversed once : I introduced openmp in a program in .c which ran full speed on my mac (but without Python interface) and when recompiled on linux machines took more time than without openmp, with huge system time usage. I asked a few gurus who found no answer. So I cannot say that my openmp is bad and others are good. My openmp is different!

I hope that everything is clear now! Sorry for the length of the mail. Best, Laurent

Le 11 juin 2019 à 17:53, Jo Bovy notifications@github.com a écrit :

Hi Laurent,

Thanks for the extra info! When using the python setup.py option, the library should be installed where normal python packages go (something like /usr/local/lib/Python3.6/site-packages). But it looks like there was another problem with the python setup.py option that is now resolved (see #25 https://github.com/jobovy/extreme-deconvolution/issues/25).

I'm sorry about the parallelization not working better, but it's probably because your problem doesn't lend itself to much speed-up from this.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jobovy/extreme-deconvolution/issues/23?email_source=notifications&email_token=ADJVTNE3Z7TEMDIQY6ULW7LPZ7DARA5CNFSM4HCY22LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNTNQI#issuecomment-500905665, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVTNC44YOPK4QAAOUJKATPZ7DARANCNFSM4HCY22LA.

"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème" (devise Shadok)

Hi Laurent,

Thanks for this very helpful overview! I think the only outstanding issue (that I can resolve, so leaving out the weird OpenMP behavior...) is that the file created by python setup.py is named

_extreme_deconvolution.cpython-37m-darwin.so

which makes sense, because that is the new naming convention in Python 3, but I had not thought of this yet. Yes, this is the same as the libextremedeconvolution.dylib file.

I'm currently considering the python setup.py installation method to be unsupported until I have time to sit down and fix it properly (I removed it from the README.rst, README, and put a note at the top of INSTALL_PYTHON).

I will close the issue, but please re-open if you feel like there is still another outstanding issue.

jobovy / extreme-deconvolution

trouble to install on OSX 10.12.6 #23