search

TA-Lib / ta-lib-python

Python wrapper for TA-Lib (http://ta-lib.org/).
http://ta-lib.github.io/ta-lib-python
Other
9.46k stars 1.74k forks source link

Error when creating new indicator #524

Open hhashim1 opened 2 years ago

hhashim1 commented 2 years ago

I am creating a new indicator and have copied parts from BBANDS. I am getting an error on one of the variables and I cannot seem to figure out why I am getting this error. Can you @mrjbq7 or @trufanov-nok can either of you help plz?

tempBuffer2 is the one that is causing the issue.

I am trying to have an equivalent of the following line of code:

def rawRelVol = (volume - Average(volume, length)) / StDev(volume, length);

Here is my code:


   /* Insert TA function code here. */
    ENUM_DECLARATION(RetCode) retCode;
    double periodTotal, tempReal;
    int i, outIdx, trailingIdx, lookbackTotal;
    ARRAY_REF(tempBuffer1);
    ARRAY_REF(tempBuffer2);
    //double rawRelVol;

    /* Calculate moving average. */
    retCode = FUNCTION_CALL(SMA)(startIdx, endIdx, inVolume,
        optInTimePeriod, outBegIdx, outNBElement, tempBuffer1);

    if ((retCode != ENUM_VALUE(RetCode, TA_SUCCESS, Success)) || ((int)VALUE_HANDLE_DEREF(outNBElement) == 0))
    {
        VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);
        return retCode;
    }

    /* Calculate the Standard Deviation */
    retCode = FUNCTION_CALL(STDDEV)((int)VALUE_HANDLE_DEREF(outBegIdx), endIdx, inVolume,
        optInTimePeriod, 1.0,
        outBegIdx, outNBElement, tempBuffer2);

    if (retCode != ENUM_VALUE(RetCode, TA_SUCCESS, Success))
    {
        VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);
        return retCode;
    }

    tempReal = (inVolume - tempBuffer1);
    tempReal = tempReal / tempBuffer2;

Here is the screenshot: image

mrjbq7 commented 2 years ago

Are you trying to divide a double by a double* ?

hhashim1 commented 2 years ago

What is a double*?

This is what I am trying to do. rawRelVol = (volume - Average(volume, length)) / StDev(volume, length)

tempBuffer2 is meant to be the StDev value that is returned.

I have copied the Stdev code from BBANDS.

trufanov-nok commented 2 years ago

You need to learn how arrays in C are designed. double* is a pointer in memory to a first double element of an array. And you can't just divide array by array or make any other operations over arrays - it won't work, they are not objects. You need to iterate both arrays in a loop and devide each element. It's not a TA-Lib related issue. it's a C knowlage related issue. You may better to ask on stackoverflow or something.

hhashim1 commented 2 years ago

Ok, so I made some changes and added a loop to iterate however I have a different issue now. I don't know how else to initialize the variable.

The error is Using uninitialized memory "tempBuffer1" and Using uninitialized memory "tempBuffer2"

Here is the code:


   /* insert local variable here */
    ENUM_DECLARATION(RetCode) retCode;
    //double periodTotal;
    double tempReal, tempReal2;
    int i;// , outIdx, trailingIdx, lookbackTotal;
    ARRAY_REF(tempBuffer1);
    ARRAY_REF(tempBuffer2);

   /* Insert TA function code here. */
   #if defined(USE_SINGLE_PRECISION_INPUT) || defined( USE_SUBARRAY )
      tempBuffer1 = outRealMiddleBand;
      tempBuffer2 = outReal;
      /* Check that the caller is not doing tricky things. 
       * (like using the input buffer in two output!)
       */
      if( (tempBuffer1 == inVolume) || (tempBuffer2 == inVolume) )
         return ENUM_VALUE(RetCode,TA_BAD_PARAM,BadParam);
   #endif

    /* Calculate moving average. */
    retCode = FUNCTION_CALL(SMA)(startIdx, endIdx, inVolume,
        optInTimePeriod, outBegIdx, outNBElement, tempBuffer1);

    if ((retCode != ENUM_VALUE(RetCode, TA_SUCCESS, Success)) || ((int)VALUE_HANDLE_DEREF(outNBElement) == 0))
    {
        VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);
        return retCode;
    }

    /* Calculate the Standard Deviation */
    retCode = FUNCTION_CALL(STDDEV)((int)VALUE_HANDLE_DEREF(outBegIdx), endIdx, inVolume,
        optInTimePeriod, 1.0,
        outBegIdx, outNBElement, tempBuffer2);

    if (retCode != ENUM_VALUE(RetCode, TA_SUCCESS, Success))
    {
        VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);
        return retCode;
    }

    /* No standard deviation multiplier needed. */
    for (i = 0; i < (int)VALUE_HANDLE_DEREF(outNBElement); i++)
    {
        tempReal = (inVolume - tempBuffer1);
        tempReal2 = tempBuffer2[i];
        outReal[i] = tempReal / tempReal2;
    }

   return ENUM_VALUE(RetCode,TA_SUCCESS,Success);
}

Here is the screenshot: image

trufanov-nok commented 2 years ago

You are probably using ARRAY_REF. That's a template which is declared in ta_memory.h. And it's value is type *name. Which means ARRAY_REF(tempBuffer1); will be substituted by ARRAY_VTYPE_REF(double, name) and its turn by a double* tempBuffer1; during compilation by a C preprocessor. That only a pointer to a first element in array. Pointer to memory. But memory isn't allocated. So you need to use ARRAY_ALLOC(name,size) and ARRAY_FREE(name) templates instead to not only declare a pointer, but also allocate and deallocate a memory for an array at the end. name is tempBuffer1. The question is: what is the size? For ex., for tempBuffer1 I would try endIdx-startIdx-LOOKBACK_CALL(SMA)(optInTimePeriod);. Similar for a second buffer.

hhashim1 commented 2 years ago

For lookback, here is what I have. return LOOKBACK_CALL(SMA)(optInTimePeriod);

Here are my inputs:

/* Generated */ enum class Core::RetCode Core::RvStdev( int    startIdx,
/* Generated */                                         int    endIdx,
/* Generated */                                         SubArray<double>^ inVolume,
/* Generated */                                         int           optInTimePeriod, /* From 2 to 100000 */
/* Generated */                                         [Out]int%    outBegIdx,
/* Generated */                                         [Out]int%    outNBElement,
/* Generated */                                         SubArray<double>^  outReal )
hhashim1 commented 2 years ago

@trufanov-nok given the below psuedo code, how would you code it in talib?

input length = 50;
rawRelVol = (volume - Average(volume, length)) / StDev(volume, length);
trufanov-nok commented 2 years ago

I'll answer tomorrow.

hhashim1 commented 2 years ago

Thank you

trufanov-nok commented 2 years ago

@trufanov-nok given the below psuedo code, how would you code it in talib?

input length = 50;
rawRelVol = (volume - Average(volume, length)) / StDev(volume, length);

I would start with something like that (didn't test it):

/* insert local variable here */
ENUM_DECLARATION(RetCode) retCode;
int outBegIdx1, outBegIdx2;
int currentIdx, currentIdx1, currentIdx2;
ARRAY_ALLOC(tempBuffer1, 50);
ARRAY_ALLOC(tempBuffer2, 50);

       /* Calculate moving average. */
    retCode = FUNCTION_CALL(SMA)(startIdx, endIdx, inVolume,
        optInTimePeriod, &outBegIdx1, outNBElement, tempBuffer1);

    if ((retCode != ENUM_VALUE(RetCode, TA_SUCCESS, Success)) || ((int)VALUE_HANDLE_DEREF(outNBElement) == 0))
    {
        ARRAY_FREE(tempBuffer1);
        ARRAY_FREE(tempBuffer2);
        VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);
        return retCode;
    }

    /* Calculate the Standard Deviation */
    retCode = FUNCTION_CALL(STDDEV)(startIdx, endIdx, inVolume,
        optInTimePeriod, 1.0,
        &outBegIdx2, outNBElement, tempBuffer2);

    if ((retCode != ENUM_VALUE(RetCode, TA_SUCCESS, Success)) || ((int)VALUE_HANDLE_DEREF(outNBElement) == 0))
    {
        ARRAY_FREE(tempBuffer1);
        ARRAY_FREE(tempBuffer2);
        VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);
        return retCode;
    }

    // startIdx is a MAX(startIdx, outBegIdx1, outBegIdx2)
    if (startIdx < outBegIdx1)
       startIdx = outBegIdx1;
    if (startIdx < outBegIdx2)
       startIdx = outBegIdx2;

    VALUE_HANDLE_DEREF(outBegIdx) = startIdx;

    if (startIdx > endIdx)
    {
        ARRAY_FREE(tempBuffer1);
        ARRAY_FREE(tempBuffer2);
        VALUE_HANDLE_DEREF_TO_ZERO(outNBElement);
        return retCode;
    }

    currentIdx = 0;
    currentIdx1 = startIdx - outBegIdx1;
    currentIdx2 = startIdx - outBegIdx2;

    for (i = startIdx; i < endIdx; i++)
    {
        outReal[currentIdx++] = (inVolume[i] - tempBuffer1[currentIdx1++]) / tempBuffer2[currentIdx2++];
    }

    ARRAY_FREE(tempBuffer1);
    ARRAY_FREE(tempBuffer2);
    (int)VALUE_HANDLE_DEREF(outNBElement) = currentIdx;

return ENUM_VALUE(RetCode,TA_SUCCESS,Success);
hhashim1 commented 2 years ago

@trufanov-nok thank you. I was getting an error saying tempBuffer1 and 2 are uninitialized or something like that. I had to add the following code and it got rid of the error however I wanted to ask if this was the right thing to do or not.

    ARRAY_REF(tempBuffer1);
    ARRAY_REF(tempBuffer2);

Here is my full variable declaration:

   /* insert local variable here */
    ENUM_DECLARATION(RetCode) retCode;
    int outBegIdx1, outBegIdx2;
    int currentIdx, currentIdx1, currentIdx2;
    int i;
    ARRAY_REF(tempBuffer1);
    ARRAY_REF(tempBuffer2);
    ARRAY_ALLOC(tempBuffer1, optInTimePeriod);
    ARRAY_ALLOC(tempBuffer2, optInTimePeriod);
trufanov-nok commented 2 years ago

You're rignt. ARRAY_REF is a type *name. The ARRAY_ALLOC is only a name = (type *)TA_Malloc( sizeof(type)*(size)). So both are required.

hhashim1 commented 2 years ago

Ok, thanks @trufanov-nok I do have another error that I dont know how to fix.

image

Also, while running make after running ./configure --prefix=/usr, I get the following error. I dont know if the two are related.

image

Thoughts?

trufanov-nok commented 2 years ago

could you upload a whole code at pastebin?

hhashim1 commented 2 years ago

Here you go...

https://pastebin.com/daPXyTJ1

trufanov-nok commented 2 years ago

you have two declarations of after section 3 and 4

    ARRAY_REF(tempBuffer1);
    ARRAY_REF(tempBuffer2);

try to leave only one of them

hhashim1 commented 2 years ago

I still have the same error. I think I had copied the section together to share it with you but forgot to undo it. The compiled code had the correct declaration which was only once.

image

hhashim1 commented 2 years ago

Here is the updated pastebin https://pastebin.com/q41gRu85

trufanov-nok commented 2 years ago

instead of (int)VALUE_HANDLE_DEREF(outNBElement) = currentIdx; try VALUE_HANDLE_DEREF(outNBElement) = currentIdx;

hhashim1 commented 2 years ago

I still have the same error. image

in my ta_func.h file, all of my function calls show an error and say that the definition is missing. I don't recall having this issue before. I am not sure if I accidentally deleted something. image

Here is the error for 'make testforta-lib-wrapper`

_______________________________________ ERROR collecting talib/test_abstract.py ________________________________________
ImportError while importing test module '/ta-lib-wrapper/talib/test_abstract.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
talib/__init__.py:93: in <module>
    from ._ta_lib import (
E   ImportError: /ta-lib-wrapper/talib/_ta_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: TA_RVSTDEV
_________________________________________ ERROR collecting talib/test_data.py __________________________________________
ImportError while importing test module '/ta-lib-wrapper/talib/test_data.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
talib/__init__.py:93: in <module>
    from ._ta_lib import (
E   ImportError: /ta-lib-wrapper/talib/_ta_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: TA_RVSTDEV
_________________________________________ ERROR collecting talib/test_func.py __________________________________________
ImportError while importing test module '/ta-lib-wrapper/talib/test_func.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
talib/__init__.py:93: in <module>
    from ._ta_lib import (
E   ImportError: /ta-lib-wrapper/talib/_ta_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: TA_RVSTDEV
________________________________________ ERROR collecting talib/test_pandas.py _________________________________________
ImportError while importing test module '/ta-lib-wrapper/talib/test_pandas.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
talib/__init__.py:93: in <module>
    from ._ta_lib import (
E   ImportError: /ta-lib-wrapper/talib/_ta_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: TA_RVSTDEV
________________________________________ ERROR collecting talib/test_polars.py _________________________________________
ImportError while importing test module '/ta-lib-wrapper/talib/test_polars.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
talib/__init__.py:93: in <module>
    from ._ta_lib import (
E   ImportError: /ta-lib-wrapper/talib/_ta_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: TA_RVSTDEV
________________________________________ ERROR collecting talib/test_stream.py _________________________________________
ImportError while importing test module '/ta-lib-wrapper/talib/test_stream.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
talib/__init__.py:93: in <module>
    from ._ta_lib import (
E   ImportError: /ta-lib-wrapper/talib/_ta_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: TA_RVSTDEV
=============================================== short test summary info ================================================
ERROR talib/test_abstract.py
ERROR talib/test_data.py
ERROR talib/test_func.py
ERROR talib/test_pandas.py
ERROR talib/test_polars.py
ERROR talib/test_stream.py
trufanov-nok commented 2 years ago

C6011 is just a warning that indicates that in theory malloc may fail to alloc a required piece of memory and return NULL. I guess it may be ignored or suppressed with additional checks if you replace:

    ARRAY_ALLOC(tempBuffer1, optInTimePeriod);
    ARRAY_ALLOC(tempBuffer2, optInTimePeriod);

with

    ARRAY_ALLOC(tempBuffer1, optInTimePeriod);
    if ( !tempBuffer1 )
        return ENUM_VALUE(RetCode,TA_ALLOC_ERR,AllocErr);

    ARRAY_ALLOC(tempBuffer2, optInTimePeriod);
    if ( !tempBuffer2 ) 
    {
        ARRAY_FREE(tempBuffer1);
        return ENUM_VALUE(RetCode,TA_ALLOC_ERR,AllocErr);
    }
trufanov-nok commented 2 years ago

How you are building your C library and how you're rebuilding the wrapper? Could you show just a sequence of commands without output?

hhashim1 commented 2 years ago

Ok so I no longer have the warning/error of dereferencing NULL pointer and the build goes fine except I have an error when I import talib in python.

ImportError: /ta-lib-wrapper/talib/_ta_lib.cpython-37m-x86_64-linux-gnu.so: undefined symbol: TA_RVSTDEV

image

Here are my steps for building the library and the wrapper, which are what we went through last month (or was it the month before?). Talib:

Ta-lib-wrapper:

trufanov-nok commented 2 years ago

What readelf -a /usr/lib/libta_lib.so.0.0.0 | grep TA_RVSTDEV outputs?

hhashim1 commented 2 years ago

Which directory should I run this in? ta-lib-wrapper or talib/c?

trufanov-nok commented 2 years ago

any of them

trufanov-nok commented 2 years ago

Also in file ./c/src/ta_abstract/tables/table_r.c what is the content of const TA_FuncDef *TA_DEF_TableR[] = array?

hhashim1 commented 2 years ago

No output if I run from the ta-lb-wrapper directory but if I run in ta-lib/c, here is what I get.

image

hhashim1 commented 2 years ago 

const TA_FuncDef *TA_DEF_TableR[] =
{
   ADD_TO_TABLE(ROC),
   ADD_TO_TABLE(ROCP),
   ADD_TO_TABLE(ROCR),
   ADD_TO_TABLE(ROCR100),
   ADD_TO_TABLE(RSI),
   ADD_TO_TABLE(RVSTDEV),
   NULL
};
hhashim1 commented 2 years ago

This is from table_r.c

/* RVSTDEV BEGIN */
static const TA_InputParameterInfo * TA_RVSTDEV_Inputs[] =
{
  &TA_DEF_UI_Input_Price_V,
  NULL
};

static const TA_OutputParameterInfo* TA_RVSTDEV_Outputs[] =
{
  &TA_DEF_UI_Output_Real,
  NULL
};

static const TA_OptInputParameterInfo* TA_RVSTDEV_OptInputs[] =
{   
    &TA_DEF_UI_TimePeriod_50_MINIMUM2,
    NULL 
};

static const TA_InputParameterInfo* TA_RVSTDEV_StructParams[] = { NULL };

DEF_FUNCTION(RVSTDEV,                     /* name */
    TA_GroupId_VolumeIndicators, /* groupId */
    "Relative Volume St Dev", /* hint */
    "RvStdev",                         /* CamelCase name */
    0                              /* flags */
);
/* RVSTDEV END */
trufanov-nok commented 2 years ago

Hm, it seems C library exports TA_RVSTDEV correctly. Perhaps somethng is forgotten to be added into the wrapper. Could you upload the output of make cython and make build from wrapper folder to pastebin?

hhashim1 commented 2 years ago

https://pastebin.com/VEsiVQiZ

trufanov-nok commented 2 years ago

what cat /usr/include/ta-lib/ta_func.h | grep TA_RVSTDEV outputs?

trufanov-nok commented 2 years ago

and

ls -hs /usr/local/include/ta-lib/ta_func.h
ls -hs /opt/include/ta-lib/ta_func.h
ls -hs /opt/local/include/ta-lib/ta_func.h
ls -hs /opt/homebrew/include/ta-lib/ta_func.h
ls -hs /opt/homebrew/opt/ta-lib/include/ta-lib/ta_func.h
ls -hs /opt/deephaven-venv/lib/python3.7/site-packages/numpy/core/include/ta-lib/ta_func.h
ls -hs /opt/deephaven-venv/include/ta-lib/ta_func.h
ls -hs /usr/include/python3.7m/ta-lib/ta_func.h

?

hhashim1 commented 2 years ago

image

Which directory do I need to be in to run the above commands? All of them are saying No such file or directory.

trufanov-nok commented 2 years ago

Ok, in wrappers folder do cat talib/_ta_lib.pxd | grep TA_RVSTDEV

hhashim1 commented 2 years ago 
    TA_RetCode TA_RVSTDEV(int startIdx, int endIdx, const double inVolume[], int optInTimePeriod, int *outBegIdx, int *outNBElement, double outReal[])
    int TA_RVSTDEV_Lookback(int optInTimePeriod)
trufanov-nok commented 2 years ago

Hm, I'm out of options. @mrjbq7 do you have any ideas?

mrjbq7 commented 2 years ago

I would suggest uninstalling the python ta-lib:

$ python3 -m pip uninstall ta-lib

And then build yours in-place for testing:

$ cd ta-lib
$ python3 setup.py build_ext --inplace

Then do all your testing / importing from the dev directory to make sure you're importing and using the code that builds and uses the new indicator, and not accidentally using the previous site-packages version.

The other thing is make sure that you added it to _func.pxi and included it at the end in __ALL__... you could do this by either re-running tools/generate_func.py or writing it manually.

hhashim1 commented 2 years ago

I have it included in two places in _func.pxi like you have suggested. I will uninstall and test again and will report to you.

hhashim1 commented 2 years ago

Check out the error I get when I run make test for the wrapper. Anything stands out?

https://pastebin.com/cZvwvUSN

mrjbq7 commented 2 years ago

Hmm, and you're sure the library installed in /usr/local/lib has your new function exported?

$ nm /usr/local/lib/libta_lib.dylib | grep TA_RVSTDEV
hhashim1 commented 2 years ago

Says No such file

mrjbq7 commented 2 years ago

Well, whatever path your ta-lib is installed to

mrjbq7 commented 2 years ago 
$ nm $(brew --prefix ta-lib)/lib/libta_lib.dylib | grep TA_RVSTDEV
mrjbq7 commented 2 years ago

I assume your issue is having a homebrew ta-lib, and then also your dev version which you want to override homebrew.

Maybe you should brew uninstall ta-lib so you don't have the one without TA_RVSTDEV in the path.

hhashim1 commented 2 years ago

I dont have brew on the machine and cannot get it installed either. At this point, I dont know how to fix this thing 😭

mrjbq7 commented 2 years ago

Well, where is your libta_lib.dylib installed? It has to be somewhere...

Run that command on it:

$ nm /path/to/libta_lib.dylib | grep TA_RVSTDEV
hhashim1 commented 2 years ago

This is what I got when searching for the file



find: ‘./mnt/i/$RECYCLE.BIN/S-1-5-21-1915536511-2278406390-2236943863-1001’: Permission denied```
mrjbq7 commented 2 years ago

But didn’t you edit and build a ta-lib with your new function?

Where does that install to?

On Tue, Jun 28, 2022 at 1:57 PM hhashim1 @.***> wrote:

This is what I got when searching for the file

find: ‘./mnt/i/$RECYCLE.BIN/S-1-5-21-1915536511-2278406390-2236943863-1001’: Permission denied```

— Reply to this email directly, view it on GitHub https://github.com/mrjbq7/ta-lib/issues/524#issuecomment-1169235438, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF5AZHG7P6MVXCA3UBJJLVRNRLRANCNFSM5ZVADI7Q . You are receiving this because you were mentioned.Message ID: @.***>