How to add new indicators to ta-lib?

[hhashim1]

I am trying to figure out how to add new indicators to ta-lib. I have copied the .c file in the ta_func folder and also added a header for the function in the ta_func.h file in the include folder. I have then compiled the library. I see the .c file in the right directory however I do not see the .o or .lo file. Additionally, python is unable to see my function.

Any help would be much appreciated.

[hhashim1]

Does the mount need to be to c? I have mounted ta-lib and not ta-lib/c

[trufanov-nok]

it need to be c in ta-lib/c as original build commands certanly need CMakeLists.txt in the folder they're executed,
Ok, let's try ./autogen.sh instead of autoreconf --install. Then ./configure --prefix=/usr

[trufanov-nok]

It's need to be said that if C sources are compiled without any problem the new function won't be accesible via python as python wrapper doesn't know about it. I mentioned it in https://github.com/mrjbq7/ta-lib/issues/515#issuecomment-1112517213 So if you reach the compilable code and import talib is working in python - let me know.

[hhashim1]

So, ./autogen.sh did not work. Here is the error. I changed the directory mount to be c

[hhashim1]

I also tried ./configure --prefix=/usr and I got the following error: configure: error: cannot find install-sh, install.sh, or shtool in "." "./.." "./../.."

[trufanov-nok]

google says the problem seems to be in ^M characters displayed in errors in output: The issue is Git converted line endings to Windows format, and now you have extra carriage returns (^M).. So linux machine is looking not for /bin/sh but for /bin/sh^M where ^M is a character with code 0x0D. It seems git on Windows does automatically convert Linux line endings to Windows line endings when you cloned my repository. And would convert them back if you would push the changes to the git server. But now you're accessing the local folder from linux and line endings are wrong for linux.

Try following on Windows machine:

1. Backup ta-lib sources, bcs we may damage them.
2. Execute in ta-lib's folder

   git config --global core.eol lf
   git config --global core.autocrlf input

   This will instruct git to use linux line endings even on windows and conver windows line endings to linux on checkout.

3. Most dangerous - execute:

   git rm -rf --cached .
   git reset --hard HEAD

This will delete (rm) recursively (r) without prompt (-f), all files except those that you have edited (--cached), from the current directory (.). The reset then returns all of those files to a state where they have their true line endings (matching what's in the repo).

repo has files with linux line endings.
Our aim is to get proper line endings in scripts like autogen.sh. I guess line endings in .c files won't have any metter for linux compiler.

[hhashim1]

Ok I was able to get all this done. What's next?

[trufanov-nok]

Now launch docker and try autogen.sh again

[hhashim1]

Didnt work. I tried all of the commands that we have been trying but nothing worked.

[trufanov-nok]

It worked, the error is different. Let me think a minute.

[trufanov-nok]

After apt install build-essential wget -y add apt install automake -y or replace apt install build-essential wget -y with apt install build-essential wget automake -y automake package should install aclocal tool.

[hhashim1]

Here is the latest error now after running ...wget automaker -y

[hhashim1]

ok check this out. I tried ./configure --prefix=/usr after running the command you suggested ....wget automate -y and I got this. It started creating the Makefile but then it failed. I don't know if you will be able to see the image. I wanted to capture the whole thing. If you cannot then let me know and I will chop up the image to make the text bigger.

[trufanov-nok]

Now libtoolize is missing. It's a tool from package libtool. YOu need to add it to the installation list: apt install build-essential wget -y with apt install build-essential wget automake libtool -y

[hhashim1]

Ok that worked. I ran apt install build-essential wget automake libtool -y followed by ./autogen.sh I then ran make which started the build process and then I ran 'make install' I uninstalled the old talib library and then reinstall using 'pip install ta-lib'.

I wanted to test in python if TSV is accessible but seems like it is not. Take a look at this screenshot. I created a dummy TSV function so whatever the default is from the template. I have NOT modified the function logic yet. What to you think I should do now?

[trufanov-nok]

Great, now you have an own TA-Lib library C code with a new function, and a process that builds and installs it in docker.

pip install ta-lib installs a python wrapper for this library from repositories that compiles in docker and allow you to access the C library from python environment. The problem is that this wrapper doesn't know about your new function. It's not discovering them in C code in compilation time - they are hardcoded. Thus you'll need to modify the python wrapper for ta-lib too. On its turn that means you need to do the same thing you did with ta-lib c codebase - you need to get your own wrapper copy and replace pip install ta-lib with building wrapper from your own local copy. That shouldn't be difficult to do.

First of all download wrapper sources (it's this github project) with git clone https://github.com/mrjbq7/ta-lib.git ta-lib-wrapper. Let's say they will be in C:\ta-lib-wrapper\ folder. Then share this folder to your docker, let's say it be in /etc/ta-lib-wrapper. Replace pip install ta-lib with

cd /etc/ta-lib-wrapper/
python setup.py install

python setup.py install is enough to compile and install ta-lib wrapper. There shouldn't be any problems with ^M endings as no .sh scripts are used. All of this will do the same as pip install ta-lib does, but with your own local copy.

Note: that python setup.py install will autodetect a ta-lib library you installed with make install in /usr/lib, but in case it fails to autodetect paths to ta-lib one may explicitly point to them by executing:

export TA_LIBRARY_PATH=/usr/lib
export TA_INCLUDE_PATH=/usr/include

before python setup.py install. That's not mandatory in your docker case, but come in handy on Windows machine where autodetection isn't working. Would be useful to learn how to build the ta-lib wrapper on Windows machine too, so you can debug your functions without a docker.
To build wrapper on windows, open its folder in command line: cd c:\ta-lib-wrapper and then set up terminal environment variables with paths to your C library:

TA_LIBRARY_PATH="C:\mypath\ta-lib\c\include\"
TA_INCLUDE_PATH="C:\mypath\ta-lib\c\lib\"
python setup.py install

The Windows build process is most risky one. In case it wails with "Cannot find ta-lib library, installation may fail." - let me know.

Once you're able to build your own local copy of the wrapper under Windows and in docker environment the next step will be Addition of wrapper of TSV function to the wrapper's sourcecode.

[hhashim1]

Ok so I ran setup.py and eventually it was successful however when I run python and import talib, I get this error.

[mrjbq7]

That looks like the installation failed to build the C modules...

[trufanov-nok]

I'm not sure, we already verified taht c sources are buildable. May it be bcs python is launched from ta-lib-wrapper folder? I'm not a python develper so not sure... How about to cd /etc/ then python then import talib ?

[mrjbq7]

It also sort of looks like you're building the pip install ta-lib version not the python3 setup.py install version you downloaded...

[hhashim1]

I am in the wrapper folder but I will redownload the repo just in case I made a mistake

[trufanov-nok]

I've just tried to git clone wrapper successfully python setup.py install then launched python and import talib didn't work with

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/ta-lib/talib/__init__.py", line 93, in <module>
    from ._ta_lib import (
ModuleNotFoundError: No module named 'talib._ta_lib'

Then I closed python, cd out of wrapper folder and did the same - it worked. I guess that's just python trying to locate module in local folders if its name matches.

[mrjbq7]

You can build it in-place so it doesn't install to your system python, and use from your ta-lib-wrapper directory

$ python3 setup.py build_ext --inplace

It makes sense that it's trying to load the un-built files when you're in that directory...

[hhashim1]

Ok I was able to make it work using the suggestion that @mrjbq7 shared and import talib worked however I still cannot access TSV.

[mrjbq7]

Did you add a def TSV(...) to talib/_func.pxi?

[hhashim1]

No. Do I do that in the wrapper?

[mrjbq7]

Right, the wrapper doesn't "automatically" wrap all the functions, it has definitions for them in talib/_func.pxi. You can either manually add your own, or use the tools/generate_func.py script to have it process the TA-Lib header files.

I'd suggest manually adding it to start?

[hhashim1]

I found the file. Are there instructions on how to add my function to _func.pxi?

[mrjbq7]

What is the C function signature of TSV? I'll show you

[hhashim1]

@mrjbq7 can I copy one of the other functions and just rename it or do I need to do more?

[hhashim1]

Well, I just copied the function from the template file and renamed it. I don't have any logic in it yet.

The inputs to be used in the real function are close, volume, length, avglength however for the dummy function I believe its only close.

I hope that makes sense.

[trufanov-nok]

Ok, great. Now we have a local copy of TA-Lib C sources with a newly added function and a local copy of TA-Lib python wrapper, both are propagated to docker and successfully buildable. Now we need to add a wrap function for TSV indicator to the python wrapper (to its local copy on Win machine). @mrjbq7 is a best expert in that as he's a creator of this wrapper.

I would note that wrapping C function would depend on this function arguments. How many optional arguments, how many input arguments (candles?) etc. This is so called function prorotype, it's generated by gen_code based on whatever you defined in ta_abstract/tables/table_#.c and declared in ta-lib/include/ta_func.h. You may open this file on Win machine (as the declaration is crossplatform) and tell us how TA_TSV declaration is looks like. I think that will help for sure to define a right wrapper function

[mrjbq7]

Well, if it only takes close then this is what you'd need:

In talib/_ta_lib.pxd, add the function references to this block:

cdef extern from "ta-lib/ta_func.h":
    ...
    TA_RetCode TA_TSV(int startIdx, int endIdx, const double inClose[], int *outBegIdx, int *outNBElement, double outReal[])
    int TA_TSV_Lookback()

And then in talib/_func.pxi:

@wraparound(False)  # turn off relative indexing from end of lists
@boundscheck(False) # turn off bounds-checking for entire function
def TSV( np.ndarray close not None ):
    """ TSV(close)

    TSV

    Inputs:
        prices: ['close']
    Outputs:
        real
    """
    cdef:
        np.npy_intp length
        int begidx, endidx, lookback
        TA_RetCode retCode
        int outbegidx
        int outnbelement
        np.ndarray outreal
    close = check_array(close)
    length = close.shape[0]
    begidx = check_begidx1(length, <double*>(close.data))
    endidx = <int>length - begidx - 1
    lookback = begidx + lib.TA_TSV_Lookback( )
    outreal = make_double_array(length, lookback)
    retCode = lib.TA_TSV( 0 , endidx , <double *>(close.data)+begidx, &outbegidx , &outnbelement , <double *>(outreal.data)+lookback )
    _ta_check_success("TA_TSV", retCode)
    return outreal 

[mrjbq7]

I generate all of these by looking through the ta_func.h and making them automatically, so if it makes sense that you'd be adding more indicators, that might be an easier way to go.

[hhashim1]

Do I add this block to the end of the file or somewhere in the middle?

cdef extern from "ta-lib/ta_func.h": ... TA_RetCode TA_TSV(int startIdx, int endIdx, const double inClose[], int *outBegIdx, int *outNBElement, double outReal[]) int TA_TSV_Lookback()

[mrjbq7]

Line 195 starts that extern block, so anywhere in that section ... it's alphabetic right now so maybe slot it in after TA_TSF?

[hhashim1]

Ok done with both. Should I run setup.py again now or is there anything else I need to do?

[hhashim1]

@trufanov-nok I just realized that this time around I did not have an entry in ta_func.h for TSV. I have copied from TSF Does this look right?

 * TA_TSV - Time Segmented Volume
 * 
 * Input  = double
 * Output = double
 * 
 * Optional Parameters
 * -------------------
 * optInTimePeriod:(From 2 to 100000)
 *    Number of period
 * 
 * 
 */
TA_LIB_API TA_RetCode TA_TSV( int    startIdx,
                              int    endIdx,
                                         const double inReal[],
                                         int           optInTimePeriod, /* From 2 to 100000 */
                                         int          *outBegIdx,
                                         int          *outNBElement,
                                         double        outReal[] );

TA_LIB_API TA_RetCode TA_S_TSV( int    startIdx,
                                int    endIdx,
                                           const float  inReal[],
                                           int           optInTimePeriod, /* From 2 to 100000 */
                                           int          *outBegIdx,
                                           int          *outNBElement,
                                           double        outReal[] );

TA_LIB_API int TA_TSV_Lookback( int           optInTimePeriod );  /* From 2 to 100000 */```

[mrjbq7]

This is where I start using the Makefile...

# generate func and stream wrappers from the talib header files
$ make generate

# regenerate the talib/_ta_lib.c file with the new functions
$ make cython

# build it in-place for testing
$ make build

# test it
$ make test

# install it to be available system wide
$ make install

# prepare for new release
$ make sdist

[mrjbq7]

I didn't have a timeperiod argument in my example code above, but you could add it...

[trufanov-nok]

I did not have an entry in ta_func.h for TSV

Why? You should not copy anything anywhere in C code - gen_code must do it for you. ta_func.h is a file generated by gen_code. You better not modify it on your own.

[hhashim1]

@trufanov-nok Well then for some reason gen_code did not do it. :-(

[trufanov-nok]

I'll doublecheck on Win machine if something wrong with ta_func.h generation. Meanwhile you may continue with a TSF copy, provided that this declaration matches ta_TSV.c

[hhashim1]

Ok I am going back and fixing the logic in my function. Is there an easy way to debug and test my logic from ta-lib or do I need to copy the code in a C Project in Visual Studio and then debug?

[trufanov-nok]

@trufanov-nok Well then for some reason gen_code did not do it. :-(

I've checked. gen_code does update ta_func.h. You also may see it in my sample: https://github.com/trufanov-nok/ta-lib-new-indicator-example/blob/master/ta-lib/c/include/ta_func.h#L5072
Perhaps you forgot to commit your changes before launching docker, bcs we've inserted a git reset HEAD in docker commands and this will reset all changes that were not commited. Have you commited the changes?

[hhashim1]

Upon make I am getting this error. What does this mean and how do I fix this?

This is what I have in table_t.c file

static const TA_InputParameterInfo *TA_TSV_Inputs[] =
{
  &TA_DEF_UI_Input_Real,
  NULL
};

static const TA_OutputParameterInfo *TA_TSV_Outputs[]   =
{
  &TA_DEF_UI_Output_Real,
  NULL
};

static const TA_OptInputParameterInfo *TA_TSV_OptInputs[] = { NULL };

static const TA_InputParameterInfo *TA_TSV_StructParams[] = { NULL };

DEF_FUNCTION( TSV,                     /* name */
              TA_GroupId_VolumeIndicators, /* groupId */
              "Price Volume Trend", /* hint */
              "Tsv",                         /* CamelCase name */
              0                              /* flags */
             );
/* TSV END */```

[trufanov-nok]

Do you still have a ta_TSV.c file in ta_func folder?

[trufanov-nok]

Perhaps you forgot to commit your changes before launching docker, bcs we've inserted a git reset HEAD in docker commands and this will reset all changes that were not commited. Have you commited the changes?

[hhashim1]

After I saved the file, I ran ./configure --prefix=/usr and then make How do I commit the code, and at what point?

[trufanov-nok]

Then you need to launch gen_code again bcs you revert back most of changes it did when get reset .. was executed from the docker.
You need to commit all changes made since the latest commit of sourcebase.
To commit:

1. If you created a new files - add them to the list of files tracked by git with command git add path_to_file. In your case this is ta_TSV.c.
2. Commit all changed tracked files with `git commit -a -m "My changes".

git commands shall be executed from sourcecode root folder or its subfolders.