Bundle Ta-lib

[bizso09]

Would it be possible to bundle Talib with this wrapper so that it works out of the box when installing with pip install ta-lib?

[mrjbq7]

Hmm, so could we create platform specific bundles with the ta-lib binaries somehow included and linked to? I'm not sure.

[bizso09]

How about only supporting Linux as a fallback option when no local Talib installation is found?

[bizso09]

(I'm trying to find a way to deploy your code to Heroku)

[mrjbq7]

We would need to build a library that statically linked to its dependencies then? I'm not sure how easy or not that would be with their Makefile. Doesn't Heroku have options to deploy binaries separately?

[bizso09]

Could you not execute the same steps as you specified in your installation instructions. Only difference would be to set the Talib make prefix and the include dir path to the installation path where your wrapper is being installed.

I've looked at including binaries with Heroku. This is what I found https://github.com/dbrgn/heroku-buildpack-python-sklearn How would I add Talib to it, this seems like a very complicated process...

[mrjbq7]

Might be building a heroku-buildpack-talib would be a good idea.

https://devcenter.heroku.com/articles/buildpacks#buildpack-urls

[aneesh-neelam]

I have created a custom Heroku buildpack for Python + TA-Lib. The GitHub repository is here

Perhaps this can be used to bundle the libraries and wrapper for Heroku.

[ghost]

How about that have a copy of c sources and apply c sources relative locations to parameters? https://docs.python.org/2/extending/building.html

[bizso09]

@aneesh-neelam I can't get your heroku-buildpack working, and there's no "Issues" tab on your github repository to discuss it. I'm getting

 ImportError: No module named Cython.Distutils

It also starts installing numpy and Cython from source. Can you show me your requirements.txt file?

emote:     warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests'
emote:     warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests'
emote:     warning: no files found matching '*.pxd' under directory 'Cython/Utility'
emote:     Installing cython script to /app/.heroku/python/bin
emote:     Installing cygdb script to /app/.heroku/python/bin
emote:     Installing cythonize script to /app/.heroku/python/bin
emote: Successfully installed Cython
emote: Cleaning up...
emote: Searching for TA-Lib
emote: Reading https://pypi.python.org/simple/TA-Lib/
emote: Reading http://github.com/mrjbq7/ta-lib
emote: Reading https://github.com/mrjbq7/ta-lib/releases
emote: Best match: TA-Lib 0.4.8
emote: Downloading https://github.com/mrjbq7/ta-lib/archive/TA_Lib-0.4.8.zip
emote: Processing TA_Lib-0.4.8.zip
emote: Writing /tmp/easy_install-6YPAMc/ta-lib-TA_Lib-0.4.8/setup.cfg
emote: Running ta-lib-TA_Lib-0.4.8/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6YPAMc/ta-lib-TA_Lib-0.4.8/egg-dist-tmp-tKFZYz
emote: /tmp/easy_install-6YPAMc/ta-lib-TA_Lib-0.4.8/setup.py:49: UserWarning: Cannot find ta-lib library, installation may fail.
emote:   warnings.warn('Cannot find ta-lib library, installation may fail.')
emote: talib/common.c:314:28: error: ta-lib/ta_defs.h: No such file or directory
emote: talib/common.c:315:30: error: ta-lib/ta_common.h: No such file or directory
emote: talib/common.c:316:32: error: ta-lib/ta_abstract.h: No such file or directory
emote: talib/common.c:317:28: error: ta-lib/ta_func.h: No such file or directory
emote: talib/common.c: In function ‘__pyx_f_5talib_6common__ta_check_success’:
emote: talib/common.c:951: error: ‘TA_SUCCESS’ undeclared (first use in this function)
emote: talib/common.c:951: error: (Each undeclared identifier is reported only once
emote: talib/common.c:951: error: for each function it appears in.)
emote: talib/common.c: In function ‘__pyx_pf_5talib_6common_2_ta_initialize’:
emote: talib/common.c:1190: warning: implicit declaration of function ‘TA_Initialize’
emote: talib/common.c: In function ‘__pyx_pf_5talib_6common_4_ta_shutdown’:
emote: talib/common.c:1266: warning: implicit declaration of function ‘TA_Shutdown’
emote: talib/common.c: In function ‘initcommon’:
emote: talib/common.c:1912: warning: implicit declaration of function ‘TA_GetVersionString’
emote: talib/common.c:1912: warning: passing argument 1 of ‘PyString_FromString’ makes pointer from integer without a cast
emote: /app/.heroku/python/include/python2.7/stringobject.h:63: note: expected ‘const char *’ but argument is of type ‘int’
emote: error: Setup script exited with error: command 'gcc' failed with exit status 1
emote:
emote:  !     Push rejected, failed to compile Python app
emote:

[bizso09]

@aneesh-neelam looking at your fork, though I'm no bash expert, the problem might be this

easy_install TA-Lib

https://github.com/aneesh-neelam/heroku-buildpack-python-talib/blob/master/bin/steps/talib#L45 Is there any chance you could include prebuilt python-talib libraries, i.e. from this repository? Then instead of installing it with easy_install, you just copy the files into the python site-packages directory. What do you think?

[mrjbq7]

I've never used the buildpack, but those errors seem to indicate the C library is missing.

[bizso09]

@mrjbq7 Is it missing the C library for compilation, or is it missing the C library that is already compiled? As far as I can tell, @aneesh-neelam 's buildpack just copies the following files to a lib directory, and adds them to the path.

libta_lib.a
libta_lib.la
libta_lib.so
libta_lib.so.0
libta_lib.so.0.0
libta_lib.so.0.0.0

These look like compiled binaries to me. I'm wondering if easy_install TA-lib command attempts to compile the C libraries from scratch, and misses the source files, since those are not included, therefore fails the installation. Would having python-TA-Lib be precompiled as well solve the issue?

Or maybe there's a way to check for the presence of already compiled TA-lib C files, and thus not attempt to re-compile them?

Or a third option could be just to copy the C files from the TA-Lib C library, rather than the precompiled files, and let heroku do the compilation.

[mrjbq7]

These errors:

emote: talib/common.c:314:28: error: ta-lib/ta_defs.h: No such file or directory
emote: talib/common.c:315:30: error: ta-lib/ta_common.h: No such file or directory
emote: talib/common.c:316:32: error: ta-lib/ta_abstract.h: No such file or directory
emote: talib/common.c:317:28: error: ta-lib/ta_func.h: No such file or directory

Mean the ta-lib Python library is trying to build against the ta-lib C library's header files. The libta_lib.so contains the C library binary, so presumably your issue is either not including the header files, or not deploying a precompiled Python library?

[bizso09]

There are these environment variables exported. Do these help?

export TALIB=$(pwd)/vendor/lib/libta/libta_lib.a
export LIBRARY_PATH=$(pwd)/python/lib:$(pwd)/vendor/lib:$(pwd)/vendor/lib/libta
export LD_LIBRARY_PATH=$(pwd)/python/lib:$(pwd)/vendor/lib:$(pwd)/vendor/lib/libta

where would you need to export the header files?

[aneesh-neelam]

@mrjbq7: I've had issues with easy_install and dependencies for TA-Lib. I honestly did not know the dependencies for TA-Lib before. By trial and error I found that numpy and Cython are required, I may be wrong however. Similar to pycuda, being a wrapper around a shared library, could this be used with pip on both Python 2.7.x and Python 3.4.x? It did not attempt to compile the C libraries before, is it doing so now?

@bizso09: What the buildpack does is extract pre-built TA-Lib shared library files and add them to the Environment PATH(s) and then install the TA-Lib Python wrapper using easy_install. The prebuilt shared library files are hosted here. I wanted to bundle the Python site packages first, however that would be dependent on the Python interpreter version. The last time I used this buildpack was months ago and it had worked for Python 2.7.8 and Python 3.4.2.

[mrjbq7]

@aneesh-neelam I believe it only requires numpy (for the ndarray support) and cython (to compile the .pyx files) and the ta-lib C library (headers to compile, binary to run against).

[mrjbq7]

Oh wait, I committed the .c files that Cython makes so you don't need to compile it again. Hmm, I'll have to look into why it's doing that now. Maybe because I built them with Cython 0.20 and if you have a different version the .c file will change? What version of Cython is being used?

[bizso09]

@mrjbq7 is this correct for pip to discover TA-Lib package in site-packages? https://github.com/bizso09/talib-binaries/tree/master/venv/lib/python2.7/site-packages

[mrjbq7]

@bizso09 it looks okay, is it not working?