TA-Lib / ta-lib-python

Python wrapper for TA-Lib (http://ta-lib.org/).
http://ta-lib.github.io/ta-lib-python
Other
9.46k stars 1.74k forks source link

How to add new indicators to ta-lib? #515

Open hhashim1 opened 2 years ago

hhashim1 commented 2 years ago

I am trying to figure out how to add new indicators to ta-lib. I have copied the .c file in the ta_func folder and also added a header for the function in the ta_func.h file in the include folder. I have then compiled the library. I see the .c file in the right directory however I do not see the .o or .lo file. Additionally, python is unable to see my function.

Any help would be much appreciated.

trufanov-nok commented 2 years ago

I wrote an article about that for my fork which is also valid for original TA-Lib: https://github.com/trufanov-nok/ta-lib-rt/wiki/Adding-new-TA-function

hhashim1 commented 2 years ago

@trufanov-nok this is great. Thank you for sharing this info. I am stuck though. I followed your instructions and added the file and also to the header in ta_func.c and then did a Build in VS but I do not see any changes in the indicator file which was created from the template. In other words, %%%GENCODE%%% has not been changed in the .c file.

Any suggestions on what could've gone wrong?

trufanov-nok commented 2 years ago

gencode.exe isn't launched in scope of a regular build process as it usually not needed. You need to execute it manually.

hhashim1 commented 2 years ago

How do I do that?

trufanov-nok commented 2 years ago

gencode is a tool which sources is a part of ta-lib project. It's not executed, but it's compiled during a build. If sourcecode was successfully compiled, gencode's binary should appear in ta-lib/c/bin/ folder. Launch it from there.

hhashim1 commented 2 years ago

Can you please give me instructions on how to do that as I am not too familiar with this process? I am on a Windows machine.

trufanov-nok commented 2 years ago

If you're able to build ta-lib with VS, I suppose you have opened a VS project file supplied with sourcecode? Like "ta-lib/c/ide/vs2012/lib_proj/ta_lib.sln"?

hhashim1 commented 2 years ago

I tried with VS 2022. I got some errors during the Build process. But in the end, the .c was not changed like it was supposed to. Here are some of the errors that I just noticed.

`Severity Code Description Project File Line Suppression State Error C2065 'TA_TSV_FramePP': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390 Error C2065 'TA_TSV_FramePPLB': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390 Error C2065 'TA_TSV_FramePPSI': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390 Error C2065 'TA_TSV_FramePPS': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390 Error C2065 'TA_TSV_FramePPBS': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390 Error C2065 'TA_TSV_FramePPSF': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390 Error C2065 'TA_TSV_FramePPSS': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390 Error C2065 'TA_TSV_FramePPSL': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\src\ta_abstract\tables\table_t.c 390

Severity Code Description Project File Line Suppression State Error LNK2019 unresolved external symbol _cnvtToUpperCase referenced in function _printDefines gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1
Error LNK2019 unresolved external symbol _cnvtChar referenced in function _printDefines gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1
Error LNK2019 unresolved external symbol _trimWhitespace referenced in function _gen_retcode gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1
Error LNK2019 unresolved external symbol _doubleToStr referenced in function _printOptInputValidation gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1
Error LNK2019 unresolved external symbol _printStateTestFunc referenced in function _doForEachFunctionPhase2 gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1
Error LNK2019 unresolved external symbol _print_xml referenced in function _doForEachFunctionXml gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1`

trufanov-nok commented 2 years ago

It's been a long time ago I didn't add any new indicators to ta-lib, so I need to recap with my own notes. It says: ... Ok, now we can build gencode. Is whole TA-Lib project can't be build (some code isn't generated yet), we must build gencode only. It could be done via IDE with help of project files from ta-lib/ta-lib/c/ide/ or ...
In VS you shall have a subprojects inside TA-Lib solution in a projects' tree. There should be a gencode item. Right click on it and choose "Bould this project only", or something like that. That will update the gencode executable. Let me know if there will be any issues in output. Then launch gencode from the folder to update a new .c file. Then build whole solution to include it into libraries' binary.

hhashim1 commented 2 years ago

Which subproject do you want me to right-click on and then choose Build? Or do I need to do it for each of them?

Here is what I have... image

trufanov-nok commented 2 years ago

gen_code

hhashim1 commented 2 years ago

Ok I had to download the right build tools and all and then do the Build. Here are the errors I have now.

Severity Code Description Project File Line Suppression State Error LNK2019 unresolved external symbol _cnvtToUpperCase referenced in function _printDefines gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1 Error LNK2019 unresolved external symbol _cnvtChar referenced in function _printDefines gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1 Error LNK2019 unresolved external symbol _trimWhitespace referenced in function _gen_retcode gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1 Error LNK2019 unresolved external symbol _doubleToStr referenced in function _printOptInputValidation gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1 Error LNK2019 unresolved external symbol _printStateTestFunc referenced in function _doForEachFunctionPhase2 gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1 Error LNK2019 unresolved external symbol _print_xml referenced in function _doForEachFunctionXml gen_code E:\Dropbox\Documents\Repos\ta-lib-rt\ta-lib-rt\c\ide\vs2022\lib_proj\gen_code\gen_code.obj 1

trufanov-nok commented 2 years ago

What do you mean by "the right build tools"? Please use VS 2022 you have.
Also I see "ta-lib-rt" in the paths. That's my fork of ta-lib - you don't need it. Use original sourcebase, like https://github.com/TA-Lib/ta-lib My fork has additional functions and won't work with wrapper in this project page. Just open the VS 2022, right click on "gen_code" in a tree. And select "build this project only", or something like that.

hhashim1 commented 2 years ago

I did use VS 2022 but it did not have build toolkit v143 and I had to install that. I will download the original repo and test.

trufanov-nok commented 2 years ago

Download it, build it. Check for errors. If there is no - then add a new file, apply the changes desrcibed in my notes, then build gen_code only, launch gen_code - it will update new file, then build the whole solution again.

hhashim1 commented 2 years ago

Ok the build was successful. When I run gen_code.exe from the terminal, here is the error I get.

image

trufanov-nok commented 2 years ago

Hmm.... Have the new .c file been updated?

hhashim1 commented 2 years ago

Yes. But the old repo does not have dotnet code in it so I don't know why it is looking for dotnet.

trufanov-nok commented 2 years ago

I don't know too. It's something from VS, I guess we can ignore it. Now you can build the whole solution and in terms of C library you will add a new empty function. Then, I guess if we are discussing this function under the python wrapper project page - you'll need to access it from python? We better to make sure the function is accessible from python side before filling it with a real code

hhashim1 commented 2 years ago

Well, I cannot build the whole solution either. Here are the errors I am getting. TSVis my new function.

Severity Code Description Project File Line Suppression State Error C2065 'TA_TSV_FramePP': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib\src\ta_abstract\tables\table_t.c 271 Error C2065 'TA_TSV_FramePPLB': undeclared identifier ta_abstract E:\Dropbox\Documents\Repos\ta-lib\src\ta_abstract\tables\table_t.c 271 Error C2099 initializer is not a constant ta_abstract E:\Dropbox\Documents\Repos\ta-lib\src\ta_abstract\tables\table_t.c 266 Error LNK1181 cannot open input file '.\..\..\..\..\lib\ta_abstract_cdd.lib' ta_libc E:\Dropbox\Documents\Repos\ta-lib\ide\vs2022\lib_proj\ta_libc\LINK 1 Error MSB3073 The command "lib /OUT:.\..\..\..\..\lib\ta_libc_cdd.lib .\..\..\..\..\lib\ta_abstract_cdd.lib .\..\..\..\..\lib\ta_common_cdd.lib .\..\..\..\..\lib\ta_func_cdd.lib" exited with code 1181. ta_libc D:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.MakeFile.Targets 45 Error LNK1181 cannot open input file 'ta_libc_cdd.lib' ta_regtest E:\Dropbox\Documents\Repos\ta-lib\ide\vs2022\lib_proj\ta_regtest\LINK 1

trufanov-nok commented 2 years ago

Do you have a new file listed inside ta_func sources in VS? I guess you need to right click on ta_func project under ta_lib solution in VS and add existing file to this prject. Then rebuild.

hhashim1 commented 2 years ago

The file was missing so I went ahead and added it but I am still getting the same error.

image

trufanov-nok commented 2 years ago

*_FramePP functions are supposed to be automatically generated by gencode.exe in src/ta_abstract/frames/ta_frame.c
You may check if there is a TA_TSV_FramePP declaration.
But I'm more interested in errors in ta_TSV.c which are next to this error. Is it related to your own code or the syntax error is in generated code? If first - remove your own code untill build problems would be resolved. If it's a syntax error in autogenerated code you might made some mistake in function declaration and gencode generated a wrong header. Please check what's wrong in ta_TSV.c?

Another interesting issue is "initializer is not constant" at line 266 in this file. I see "TA_G...." there and last is hidden by pop up hint (it's useless). What is it? There supposed to be one of constants from ta_def_ui.h:

extern const char TA_GroupId_MathOperatorsString[];
extern const char TA_GroupId_MathTransformString[];
extern const char TA_GroupId_OverlapStudiesString[];
extern const char TA_GroupId_VolatilityIndicatorsString[];
extern const char TA_GroupId_MomentumIndicatorsString[];
extern const char TA_GroupId_CycleIndicatorsString[];
extern const char TA_GroupId_VolumeIndicatorsString[];
extern const char TA_GroupId_PatternRecognitionString[];
extern const char TA_GroupId_StatisticString[];
extern const char TA_GroupId_PriceTransformString[];

which are

const char TA_GroupId_MathOperatorsString[]        = "Math Operators";
const char TA_GroupId_MathTransformString[]        = "Math Transform";
const char TA_GroupId_OverlapStudiesString[]       = "Overlap Studies";
const char TA_GroupId_VolatilityIndicatorsString[] = "Volatility Indicators";
const char TA_GroupId_MomentumIndicatorsString[]   = "Momentum Indicators";
const char TA_GroupId_CycleIndicatorsString[]      = "Cycle Indicators";
const char TA_GroupId_VolumeIndicatorsString[]     = "Volume Indicators";
const char TA_GroupId_PatternRecognitionString[]   = "Pattern Recognition";
const char TA_GroupId_StatisticString[]            = "Statistic Functions";
const char TA_GroupId_PriceTransformString[]       = "Price Transform";

Have you tried to add a new group?

hhashim1 commented 2 years ago

There is no mention of TA_TSV_FramePP in table_t.c file. I dont have any code in ta_TSV.c file. I copied over the template and renamed it. I have searched for that word and I cannot find it so I don't know where it is coming from. Here is the content of that file.

`/* TA-LIB Copyright (c) 1999-2008, Mario Fortier

/ List of contributors:

/ START GENCODE SECTION 1 - DO NOT DELETE THIS LINE / %%%GENCODE%%% / END GENCODE SECTION 1 - DO NOT DELETE THIS LINE / { / insert local variable here /

/ START GENCODE SECTION 2 - DO NOT DELETE THIS LINE / %%%GENCODE%%% / END GENCODE SECTION 2 - DO NOT DELETE THIS LINE /

/ insert lookback code here. /

return 0; }

/ START GENCODE SECTION 3 - DO NOT DELETE THIS LINE / %%%GENCODE%%% / END GENCODE SECTION 3 - DO NOT DELETE THIS LINE / { / insert local variable here /

/ START GENCODE SECTION 4 - DO NOT DELETE THIS LINE / %%%GENCODE%%% / END GENCODE SECTION 4 - DO NOT DELETE THIS LINE /

/ Insert TA function code here. /

/ Default return values / VALUE_HANDLE_DEREF_TO_ZERO(outBegIdx); VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);

return ENUM_VALUE(RetCode,TA_SUCCESS,Success); }

/ START GENCODE SECTION 5 - DO NOT DELETE THIS LINE / %%%GENCODE%%% / END GENCODE SECTION 5 - DO NOT DELETE THIS LINE /`

I don't know anything about groups so no, I have not added a new group.

hhashim1 commented 2 years ago

Would you be open to and available to do a google hangout session and work on this together so that we can troubleshoot it and fix the issue quickly?

trufanov-nok commented 2 years ago

I've just booted a Win machine and tried to add a new indicator with VS. And reproduced your problem. gen-code execution results in

 ./gen_code.exe
gen_code V0.6.0-dev (Apr 29 2022 19:22:13)
Now updating source code...

Cannot access [..\..\dotnet\src\Core\TA-Lib-Core.vcproj]

I never tried to use gencode on Win machine. It seems it's really looking for TA-Lib-Core.vcproj if was build with VS. There is a code for that: https://github.com/TA-Lib/ta-lib/blob/master/src/tools/gen_code/gen_code.c#L709
Give me some time I'll clear up with it under Win.

trufanov-nok commented 2 years ago

Ok, I figured out what was the problem. Apparently there are two versions of original TA-Lib sources. One is a complete sourcebase (with scripts to make a java, dotnet version of the lib etc.) and another is a subfolder of it. This shorten version of the code is enough to build the library from sources and seems to widely used for ex. if python wrapper need to build the lib on user machine before installation. But this minimal codebase isn't enough for gen_code to properly work and maintain the library.

So all you need is a proper sourcecode. I've made its copy long time ago and saved under original branch of my fork. To get it use: git clone -b original https://github.com/trufanov-nok/ta-lib-rt.git ta-lib. They will contain a dotnet\src\Core\TA-Lib-Core.vcproj which is required for gen_code.exe under Windows. Just do all the steps again. I tried - everything works. Except the fact that after the first launch of gen_code I found in ta_TSV.c a single %%%GENCODE%%% at the end of the file. That seems to be a gencode bug, just launch it again and it will replace this line with code properly.

trufanov-nok commented 2 years ago

I've also created an example project where last 5 commits demonstrate a steps of addition of a new function: https://github.com/trufanov-nok/ta-lib-new-indicator-example

I guess you may even copy ta_TSV.c from it into your project instead of invoking gen_code. As gen_code only purpose is to fill template with code and it's done only once. And if you need to add more indicators you can skkip gen_code step by copying this ta_TSV.c and renaming file and all TSV text in it.

hhashim1 commented 2 years ago

Ok I was able to use gen_code by cloning your backup repo and everything worked. It generated the code and all that.

I am here at this point and not sure what I need to do with Visual Studio. What option should I choose from the Build menu or when I right-click on the Solution?

and rebuild whole project:

ta-lib/ta-lib/build$ cmake ..
ta-lib/ta-lib/build$ make
trufanov-nok commented 2 years ago

You either should right click on "Solution 'ta-lib'" on the very top of the tree and click Build or just press "Ctrl+Shift+B".

hhashim1 commented 2 years ago

Ok I did a build and got an error for %%%GENCODE%%% I commented out that line and did the Build again and it was successful. How can I easily test the function to make sure it is available before I proceed further?

trufanov-nok commented 2 years ago

Ok I did a build and got an error for %%%GENCODE%%% I commented out that line and did the Build again and it was successful.

It's better not comment the %%%GENCODE%%% line. Uncomment it and launch gen_code again - it replace it with proper code. It seems there is a bug in gen_code of original ta-lib which doesn't allow it to replace all %%%GENCODE%%% at once, but after second launch it completes the replacement. But if you commented it out - it's ok too as this last piece of code is quite useless.

trufanov-nok commented 2 years ago

How can I easily test the function to make sure it is available before I proceed further?

The easiest way to check if dll contains the function (TA_TSV) is to list its export symbols and try to find it in them. I would use the old tool dependancy walker Or there is a suggestion to use a command line utility from VS command prompt: https://stackoverflow.com/a/6634722/841424

Or you can just open the dll or lib file in notepad and search for "TA_TSV" if you find something you may be pretty sure the code is in dll.

trufanov-nok commented 2 years ago

Oh, I recall that I actually made a tool for listing all the functions inside ta-lib rt. I guess it could be easely modified to list original TA-Lib functions: https://github.com/trufanov-nok/ta-gen

I'll check it.

trufanov-nok commented 2 years ago

UPD: No, it can't be compilled for Win so easy as it uses boost library...

hhashim1 commented 2 years ago

so now the question is how can I take this library and install/compile it on a linux machine? I am using WSL on my windows box for docker. I am running Kafka and other tools on docker/Linux and need to get ta-lib there.

Here is the setup.sh file that I have which downloads, extracts and compiles the ta-lib from the internet. I need to replace it so that instead of downloading from the internet, the process would use my file.

Would you happen to have any idea on how to do this?

apt-get upgrade -y
apt-get update -y
apt install build-essential wget -y
wget https://artiya4u.keybase.pub/TA-lib/ta-lib-0.4.0-src.tar.gz
tar -xvf ta-lib-0.4.0-src.tar.gz
cd ta-lib/
./configure --prefix=/usr
/* this copy statement does not work */
docker cp E:/Dropbox/Downloads/ta-lib/c/include/ta_func.h deephaven-server-1:/ta-lib/include/ta_func.h
docker cp E:/Dropbox/Downloads/ta-lib/c/src/ta_func/ta_TSV.c deephaven-server-1:/ta-lib/src/ta_func/ta_TSV.c
/* ignore above copy statements */
make
make install
apt-get install python3-dev -y
pip install ta-lib
trufanov-nok commented 2 years ago

Well, I know a little about docker. Perhaps: docker run -v C:\mypath\ta-lib\c\:/etc/ta-lib And in script replace lines

wget https://artiya4u.keybase.pub/TA-lib/ta-lib-0.4.0-src.tar.gz
tar -xvf ta-lib-0.4.0-src.tar.gz
cd ta-lib/

with just cd /etc/ta-lib/

hhashim1 commented 2 years ago

Ok thanks. So, I need to copy the whole ta-lib directory and not just a single file, right?

trufanov-nok commented 2 years ago

You 're not copying anything. This supposed to create a virtual directory /etc/ta-lib inside docker, that in real will be pointing to C:\mypath\ta-lib\c\ filder. You should change C:\mypath\ta-lib\c\ to the talib\c\ folder on your windows machine. Then inside docker we are skipping the lines that downloading archieve with a sources and unpacking it. We just go to /etc/ta-lib/ with cd /etc/ta-lib/ and launching ta-lib's build commands.

trufanov-nok commented 2 years ago

So, I need to copy the whole ta-lib directory and not just a single file, right?

Ok, I got you now. Yes, it'll be easier to pass a whole library. Actually, I've realized that you need to do one more thing. Remember we add a link to ta_TSV.c to VS project? Thats for Windows, and now you're building in Linux-like environment.
The

./configure --prefix=/usr
make 

is another buildchain and it doesn't know about ta_TSV.c like VS buildchain didn't know before. Now you need to add this file to make buildchain too. For that edit the ta-lib\c\CMakeLists.txt: find there a line ${ta-lib-rt_SOURCE_DIR}/c/src/ta_func/ta_TSF.c and insert line ${ta-lib-rt_SOURCE_DIR}/c/src/ta_func/ta_TSV.c after it.

hhashim1 commented 2 years ago

Ok...here is the latest update. I got all this stuff working. I couldn't find ta_TSF.c listing so I copied ta_SMA.c and renamed it.

I was also able to modify the docker file to create the mount. Now I can see the files in my docker container however I am still unable to use my function in my IDE. It says cannot find method TSV.

When I try to manually compile in the container, I get an error for this command. root@d2eb1d7b4e5e:/ta-lib# ./configure --prefix=/usr bash: ./configure: No such file or directory

Any idea?

trufanov-nok commented 2 years ago

When I try to manually compile in the container, I get an error for this command.

That's a result of a difference between full sourcebase and the shorten version of it which is installed from https://artiya4u.keybase.pub/TA-lib/ta-lib-0.4.0-src.tar.gz
you need to call autoreconf --install before ./configure --prefix=/usr. This command will create a configure script in this folder.

hhashim1 commented 2 years ago

I got this error now. root@d2eb1d7b4e5e:/ta-lib# autoreconf --install autoreconf: 'configure.ac' or 'configure.in' is required

trufanov-nok commented 2 years ago

what ls command says?

hhashim1 commented 2 years ago

image

Is: command not found

trufanov-nok commented 2 years ago

LS, command, in lowercase

hhashim1 commented 2 years ago

image

That's mounted to my directory on my Windows drive.

trufanov-nok commented 2 years ago

which command did you used to launch a docker? You need to be in ./c/ folder. You one directory above it...

hhashim1 commented 2 years ago

Here is the command I used to get into my container docker exec -ti deephaven_server_1 /bin/bash

hhashim1 commented 2 years ago

Ok I went to c and then ran autoreconf --install and this is what I got image