search

trufanov-nok / ta-lib-rt

TA-Lib RT is a fork of TA-Lib that provides additional API for incremental calculation of indicators without reprocessing whole data.
http://ta-lib.org
83 stars 30 forks source link

_wrap_TA_RealList_data_get array_size1 undeclared when building python. #17

Closed xiaoluffy closed 3 years ago

xiaoluffy commented 3 years ago

I edit swig/make/python/linux/g++/Makefile to target path:

 makefile for building a Python ta_lib.py module
# CMR (commandline multithreaded release only)

TOPDIR=../../../..
WRAPDIR=$(TOPDIR)/temp/python/wrap
TEMPDIR=$(TOPDIR)/temp/python/wrap
INSTDIR=/opt/anaconda/lib/python3.7/site-packages

CPPFLAGS   = -I$(TOPDIR)/../c/include -I/opt/anaconda/include/python3.7m -I/usr/local/include/ta-lib-rt
CFLAGS     = -pipe -Wall -Wno-unused -O2 -fno-strict-aliasing -fPIC
...

and make failed with output:

gcc -pipe -Wall -Wno-unused -O2 -fno-strict-aliasing -fPIC -m64 -D64BIT -march=x86-64 -I../../../../../c/include -I/opt/anaconda/include/python3.7m -I/usr/local/include/ta-lib-rt -c -o ../../../../temp/python/wrap/ta_libc_wrap.o ../../../../temp/python/wrap/ta_libc_wrap.c ../../../../temp/python/wrap/ta_libc_wrap.c: In function ‘_wrap_TA_RealList_data_get’: ../../../../temp/python/wrap/ta_libc_wrap.c:177171:28: error: ‘array_size1’ undeclared (first use in this function) resultobj = PyList_New(array_size1); ^~~ ../../../../temp/python/wrap/ta_libc_wrap.c:177171:28: note: each undeclared identifier is reported only once for each function it appears in ../../../../temp/python/wrap/ta_libc_wrap.c: In function ‘_wrap_TA_IntegerList_data_get’: ../../../../temp/python/wrap/ta_libc_wrap.c:177231:28: error: ‘array_size1’ undeclared (first use in this function) resultobj = PyList_New(array_size1); ^~~

: recipe for target '../../../../temp/python/wrap/ta_libc_wrap.o' failed make: *** [../../../../temp/python/wrap/ta_libc_wrap.o] Error 1

Any idea what is going on and how it might be fixed? Thanks!

MORE details:

swig -version SWIG Version 3.0.12 Compiled with g++ [x86_64-pc-linux-gnu] Configured options: +pcre Please see http://www.swig.org for reporting bugs and further information

g++ -v Using built-in specs. COLLECT_GCC=g++ COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper OFFLOAD_TARGET_NAMES=nvptx-none OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04

trufanov-nok commented 3 years ago

Hi, Frankly I've never tested a SWIG and some other bindings (C#, Java) that are inherited from original TA-Lib project. As I understand the SWIG is worse then python wrapper made with Cypthon. The original project has such wrapper made by John Benediktsson: https://github.com/mrjbq7/ta-lib It can be installed via pip install. Check out its readme. I've forked it and adapted for TA-Lib RT: https://github.com/trufanov-nok/ta-lib-py-wrapper It's experimental as I don't know python well enough to wrap Abstract API, but other stuff works. So it's not final and can't be installed via pip. But manual installation works. The general discussion about this wrapper and its performance is here.

So if you just need a wrapper for python I would recommend to start with this instead of SWIG. But if you really need SWIG or want to help to debug it and polish to working state - I can try to fix it.

xiaoluffy commented 3 years ago

Thanks a lot for your reply. I built the ta-lib-rt and ta-lib-py-wrapper successfully, but there is still some issues, something wrong with my environment?

python ta-lib-py-wrapper/tools/perf_talib.py, (vector functions is fine and incremental functions fail.)

test_len: 10000, loops: 1000
time spend: 0.618861
time per loop: 0.000619

Test batch state functions:

Segmentation fault (core dumped)

python ta-lib-py-wrapper/tools/examples.py

Traceback (most recent call last):
  File "example.py", line 8, in <module>
    from talibrt.abstract import Function
  File "/opt/anaconda/lib/python3.7/site-packages/TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg/talibrt/abstract.py", line 2, in <module>
    from ._ta_lib import Function as _Function, FunctionRT as _FunctionRT, __TA_FUNCTION_NAMES__, _get_defaults_and_docs
ImportError: cannot import name 'FunctionRT' from 'talibrt._ta_lib' (/opt/anaconda/lib/python3.7/site-packages/TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg/talibrt/_ta_lib.cpython-37m-x86_64-linux-gnu.so)

building the wrapper projects by 'python setup.py install'.

                                ^~~~~~~~~~~~~~~~~~~~~~~~
gcc -pthread -shared -B /opt/anaconda/compiler_compat -L/opt/anaconda/lib -Wl,-rpath=/opt/anaconda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/talibrt/_ta_lib.o -L/usr/lib -L/usr/local/lib -L/usr/lib64 -L/usr/local/lib64 -L/opt/lib -L/opt/local/lib -L/usr/local/lib -Wl,-R/usr/local/lib -lta-lib-rt -o build/lib.linux-x86_64-3.7/talibrt/_ta_lib.cpython-37m-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/_ta_lib.cpython-37m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/deprecated.py -> build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/test_pandas.py -> build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/__init__.py -> build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/abstract.py -> build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/test_abstract.py -> build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/test_func.py -> build/bdist.linux-x86_64/egg/talibrt
copying build/lib.linux-x86_64-3.7/talibrt/test_data.py -> build/bdist.linux-x86_64/egg/talibrt
byte-compiling build/bdist.linux-x86_64/egg/talibrt/deprecated.py to deprecated.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/talibrt/test_pandas.py to test_pandas.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/talibrt/__init__.py to __init__.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/talibrt/abstract.py to abstract.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/talibrt/test_abstract.py to test_abstract.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/talibrt/test_func.py to test_func.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/talibrt/test_data.py to test_data.cpython-37.pyc
creating stub loader for talibrt/_ta_lib.cpython-37m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/talibrt/_ta_lib.py to _ta_lib.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying TA_Lib_RT.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying TA_Lib_RT.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying TA_Lib_RT.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying TA_Lib_RT.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying TA_Lib_RT.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
talibrt.__pycache__._ta_lib.cpython-37: module references __file__
creating 'dist/TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg
removing '/opt/anaconda/lib/python3.7/site-packages/TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg' (and everything under it)
creating /opt/anaconda/lib/python3.7/site-packages/TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg
Extracting TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg to /opt/anaconda/lib/python3.7/site-packages
TA-Lib-RT 0.8.1a0 is already the active version in easy-install.pth

Installed /opt/anaconda/lib/python3.7/site-packages/TA_Lib_RT-0.8.1a0-py3.7-linux-x86_64.egg
Processing dependencies for TA-Lib-RT==0.8.1a0
Searching for numpy==1.19.5
Best match: numpy 1.19.5
Adding numpy 1.19.5 to easy-install.pth file
Installing f2py script to /opt/anaconda/bin
Installing f2py3 script to /opt/anaconda/bin
Installing f2py3.7 script to /opt/anaconda/bin

Using /opt/anaconda/lib/python3.7/site-packages
Finished processing dependencies for TA-Lib-RT==0.8.1a0

My python info is conda info

     active environment : base
    active env location : /opt/anaconda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.9.2
    conda-build version : 3.17.8
         python version : 3.7.3.final.0
       virtual packages : __glibc=2.27=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/anaconda  (writable)

          package cache : /opt/anaconda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/anaconda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.1 CPython/3.7.3 Linux/4.15.0-88-generic ubuntu/18.04.5 glibc/2.27
                UID:GID : 0:0
             netrc file : None
           offline mode : False
trufanov-nok commented 3 years ago

Hm, For python ta-lib-py-wrapper/tools/perf_talib.py I'm getting valid :

$ python ta-lib-py-wrapper/tools/perf_talib.py
test_len: 10000, loops: 1000
time spend: 1.158334
time per loop: 0.001158

Test batch state functions:

test_len: 10000, loops: 1000
time spend: 4.102825
time per loop: 0.004103

Test state functions:

test_len: 10000, loops: 1000
time spend: 97.075456
time per loop: 0.097075

As for python ta-lib-py-wrapper/tools/example.py - yes, it's not working as Abstract API wasn't done in python wrapper for TA-Lib. Only direct calls to TA indicator functions are available.

Try to launch you python and execute following code in its terminal:

import talibrt
import numpy as np

closes = np.asarray( [2,3,4,5], dtype=np.float64)

res, macd, signal, hist = talibrt.MACD(closes, fastperiod=3, slowperiod=2, signalperiod=1)
print("old api", res, signal)

res, state = talibrt.MACD_StateInit(fastperiod=3, slowperiod=2, signalperiod=1)
res, macd, signal, hist = talibrt.MACD_BatchState(state, closes)
print("state api", res, signal)
xiaoluffy commented 3 years ago

Hi, all State or BatchState methods will raise a segmentation fault. so what's the version of your ta-lib-rt, python, cython and g++? I'm trying to fix it by rebuilding packages in a different env.

In [1]: import talibrt
In [2]: import numpy as np
In [3]: closes = np.asarray( [2,3,4,5], dtype=np.float64)
In [4]: res, macd, signal, hist = talibrt.MACD(closes, fastperiod=3, slowperiod=2, signalperiod=1)
In [5]: print("old api", res, signal)
old api TALibResult.OK [nan nan 0.5 0.5]
In [6]: res, state = talibrt.MACD_StateInit(fastperiod=3, slowperiod=2, signalperiod=1)
In [7]: res, macd, signal, hist = talibrt.MACD_BatchState(state, closes)

Segmentation fault (core dumped)
...
In [9]: res, macd, signal, hist = talibrt.MACD_State(state, closes[0])
Segmentation fault (core dumped)
xiaoluffy commented 3 years ago

I rebuilt the packages in the env below, and still raise a segmentation fault. Then I found that talibrt in python(0.7.0Alpha) , is different from talibrt lib(0.8.1Alpha), may cause the issues.

ls /usr/local/lib/libta*
lrwxrwxrwx   libta-lib-rt.so -> libta-lib-rt.so.0.8
lrwxrwxrwx libta-lib-rt.so.0.8 -> libta-lib-rt.so.0.8.1-alpha
-rw-r--r--  1  libta-lib-rt.so.0.8.1-alpha

conda info

     active environment : base
    active env location : /opt/anaconda
            shell level : 1
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.9.2
    conda-build version : 3.20.5
         python version : 3.8.5.final.0
       virtual packages : __glibc=2.27=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/anaconda  (writable)
          package cache : /opt/anaconda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/anaconda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Linux/5.4.0-58-generic ubuntu/18.04.5 glibc/2.27
                UID:GID : 0:0
             netrc file : None
           offline mode : False

g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)

>>> import numpy as np
>>> np.__version__
'1.19.2'
>>> import cython
>>> cython.__version__
'0.29.21'
>>> import talibrt
>>> talibrt.__version__
'0.7.0alpha'
trufanov-nok commented 3 years ago

I've installed anaconda and was able to reproduce this seg fault in its environment. Need to debug it now...

trufanov-nok commented 3 years ago

Hi, Should be fixed now, could you pull up-to-date wrapper sources from master branch, recompile and check on your side?

xiaoluffy commented 3 years ago

python perf_talib.py works fine now! many thanks!