KxSystems / embedPy

Allows the kdb+ interpreter to call Python functions
https://code.kx.com/q/interfaces
Apache License 2.0
89 stars 42 forks source link

Configuration fails on macOS #13

Closed abalkin closed 6 years ago

abalkin commented 6 years ago

I am using a universal Python build from

$ brew install sashkab/python/python36 --universal

and configuration fails with the following diagnostic

$ q configure.q
KDB+ 3.4 2016.06.14 Copyright (C) 1993-2016 Kx Systems
m32/ 8()core 16384MB a a-mbp.local 172.20.10.2 NONEXPIRE

Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: sequence item 0: expected str instance, int found
{system string[$[count w;pp first w;e`python]]," -c \"",x,"\""}
'os
@
.,["\\"]
"python3 -c \"import sysconfig as c,os,sys;v=c.get_config_var;first=lambda x:..
abalkin commented 6 years ago

In order to troubleshoot this, I patched configure.q as follows

diff --git a/configure.q b/configure.q
index 3113074..da4f4bd 100755
--- a/configure.q
+++ b/configure.q
@@ -3,7 +3,7 @@ e:{-2 string x;exit 1}
 w:where{"B"$first@[system;string[x]," -c 'print(1)'";"0"]}@'pp:`python3`python                                                                      /python3 first
 py:{system string[$[count w;pp first w;e`python]]," -c \"",x,"\""}
 s :"import sysconfig as c,os,sys;v=c.get_config_var;first=lambda x:len(x) and x[0];d=v('LDLIBRARY');P,p=sys.exec_prefix,sys.prefix;"           /find dylib
-s,:"w=lambda x:first([a[0] for a in os.walk(x) if d in a[2]]);L=w(P+'/lib')or w(v('LIBDIR'));I=c.get_path('include');print('\\n'.join([L,d,P,p,I]))"
+s,:"w=lambda x:first([a[0] for a in os.walk(x) if d in a[2]]);L=w(P+'/lib')or w(v('LIBDIR'))or '<fixme>';I=c.get_path('include');print('\\n'.join([L,d,P,p,I]))"
 `L`d`P`p`I set'0N!py s;l:$[a=`l;-3_3_d;a=`w;-4_d;-6_3_d]
 I:"-I",I," -I",first py"import numpy as n;print(n.get_include())"                                                                                   /numpy dir
 `r`D set\:"";if[rpath:1;r:raze " -Xlinker ",/:("-rpath";L," ");d:L,"/",d;D:"-DPH='L\"",P,":",p,"\"' ";]

Now,

$ q configure.q -q
("<fixme>";"Python.framework/Versions/3.6/Python";"/Users/a/.virtualenvs/3/bi..

Apparently there is some problem with the library search logic. The python3.6m-config script reports the library path correctly

$ python3.6m-config --ldflags
-L/usr/local/opt/python36/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation
$ ls -l /usr/local/opt/python36/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib
lrwxr-xr-x  1 a  admin  15 Nov  8 12:12 /usr/local/opt/python36/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib -> ../../../Python
abalkin commented 6 years ago

I was able to compile p.so after patching the generated makefile as follows:

--- makefile.orig   2017-11-12 14:35:22.000000000 -0500
+++ makefile    2017-11-12 14:39:46.000000000 -0500
@@ -1,8 +1,8 @@
-d=<fixme>/Python.framework/Versions/3.6/Python
-C=gcc -g3 -Os -DKXVER=3
-D=-DPH='L"/Users/a/.virtualenvs/3/bin/..:/Users/a/.virtualenvs/3/bin/.."' -DRP=1 -DDY='"<fixme>/Python.framework/Versions/3.6/Python"'
+d=/usr/local/opt/python36/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin
+C=gcc -g3 -Os -DKXVER=3 -m32
+D=-DPH='L"/Users/a/.virtualenvs/3/bin/..:/Users/a/.virtualenvs/3/bin/.."' -DRP=1 -DDY='"libpython3.6m.dylib"'
 I=-I/Users/a/.virtualenvs/3/include/python3.6m -I/Users/a/.virtualenvs/3/lib/python3.6/site-packages/numpy/core/include
-L= -L<fixme> -lhon.framework/Versions/3.6/ -Xlinker -rpath -Xlinker <fixme> -bundle -undefined dynamic_lookup
+L= -L$d -lpython3.6m -Xlinker -rpath -Xlinker $d -bundle -undefined dynamic_lookup
 p.so:p.c k.h makefile
    $C $< -o $@ $D $I $L
 k.h:
effbiae commented 6 years ago

@abalkin what do you think of this pull request? merge with your setup.py work?

initial configure.py. uses python-config --ldflags if not anaconda #15

jhanna-kx commented 6 years ago

Configure is no longer required as there is no compile time dependency on python, so closing out