Literal Unroll "cannot be determined, operation: unknown operation, location: unknown location"

[quantfreedom]

I have included a loom video to explain what is happening but my assumption is that i am doing something wrong using literal unroll but also it could be a bug ... not sure https://www.loom.com/share/b8b8464144714b01a8cd79cc1eb60b4c

If you need to look at the code it can all be found here https://github.com/QuantFreedom1022/QuantFreedom/blob/dev/quantfreedom/backtester/nb/simulate.py#L482

https://github.com/QuantFreedom1022/QuantFreedom/tree/dev/tests here is my tests folder where i am running the tests

here is a snippet of the problem

    arrays = (
        np.array([0.]),
        leverage_array,
        max_equity_risk_pct_array,
        max_equity_risk_value_array,
        risk_rewards_array,
        size_pct_array,
        size_value_array,
        sl_pcts_array,
        sl_to_be_based_on_array,
        sl_to_be_trail_by_when_pct_from_avg_entry_array,
        sl_to_be_when_pct_from_avg_entry_array,
        sl_to_be_zero_or_entry_array,
        tp_pcts_array,
        tsl_based_on_array,
        tsl_pcts_init_array,
        tsl_trail_by_pct_array,
        tsl_when_pct_from_avg_entry_array,
    )

    n = 1
    for x in arrays:
        n *= x.size
    out = np.empty((n, len(arrays)))
    cart_array = np.empty(n, dtype=cart_array_dt)

    for i in range(len(arrays)):
        m = int(n / arrays[i].size)
        out[:n, i] = np.repeat(arrays[i], m)
        n //= arrays[i].size

    n = arrays[-1].size
    for k in range(len(arrays)-2, -1, -1):
        n *= arrays[k].size
        m = int(n / arrays[k].size)
        for j in range(1, arrays[k].size):
            out[j*m:(j+1)*m, k+1:] = out[0:m, k+1:]

    dtype_names = (
        'order_settings_id',
        'leverage',
        'max_equity_risk_pct',
        'max_equity_risk_value',
        'risk_rewards',
        'size_pct',
        'size_value',
        'sl_pcts',
        'sl_to_be_based_on',
        'sl_to_be_trail_by_when_pct_from_avg_entry',
        'sl_to_be_when_pct_from_avg_entry',
        'sl_to_be_zero_or_entry',
        'tp_pcts',
        'tsl_based_on',
        'tsl_pcts_init',
        'tsl_trail_by_pct',
        'tsl_when_pct_from_avg_entry',
    )

    counter = 0
    for dtype_name in literal_unroll(dtype_names):
        for col in range(n):
            cart_array[dtype_name][col] = out[col][counter]
        counter += 1
    return counter

[quantfreedom]

what is weird is i created this code in a test example and it works perfectly fine ... but when i take the same code and try to run it in my python simulate file it gives me the error

i even tried replacing the arrays tuple with the one you see below and i still get errors

from numba import njit, literal_unroll
import numpy as np
cart_array_dt = np.dtype([
    ('order_settings_id', np.float_),
    ('leverage', np.float_),
    ('max_equity_risk_pct', np.float_),
    ('max_equity_risk_value', np.float_),
    ('risk_rewards', np.float_),
    ('size_pct', np.float_),
    ('size_value', np.float_),
    ('sl_pcts', np.float_),
    ('sl_to_be_based_on', np.float_),
    ('sl_to_be_trail_by_when_pct_from_avg_entry', np.float_),
    ('sl_to_be_when_pct_from_avg_entry', np.float_),
    ('sl_to_be_zero_or_entry', np.float_),
    ('tp_pcts', np.float_),
    ('tsl_based_on', np.float_),
    ('tsl_pcts_init', np.float_),
    ('tsl_trail_by_pct', np.float_),
    ('tsl_when_pct_from_avg_entry', np.float_),
], align=True)

@njit(cache=True)
def test():
    arrays = (
        np.array([0.]),
        np.array([1., 2., 3.]),
        np.array([np.nan]),
        np.array([4., 5.]),
        np.array([np.nan]),
        np.array([np.nan]),
        np.array([1.,6.,8.]),
        np.array([0.]),
        np.array([np.nan]),
        np.array([3.]),
        np.array([np.inf]),
        np.array([np.inf]),
        np.array([np.nan]),
        np.array([np.inf]),
        np.array([np.nan]),
        np.array([np.inf]),
        np.array([np.nan]),
    )

    dtype_names = (
        'order_settings_id',
        'leverage',
        'max_equity_risk_pct',
        'max_equity_risk_value',
        'risk_rewards',
        'size_pct',
        'size_value',
        'sl_pcts',
        'sl_to_be_based_on',
        'sl_to_be_trail_by_when_pct_from_avg_entry',
        'sl_to_be_when_pct_from_avg_entry',
        'sl_to_be_zero_or_entry',
        'tp_pcts',
        'tsl_based_on',
        'tsl_pcts_init',
        'tsl_trail_by_pct',
        'tsl_when_pct_from_avg_entry',
    )

    n = 1
    for x in arrays:
        n *= x.size
    out = np.empty((n, len(arrays)))
    cart_array = np.empty(n, dtype=cart_array_dt)

    for i in range(len(arrays)):
        m = int(n / arrays[i].size)
        out[:n, i] = np.repeat(arrays[i], m)
        n //= arrays[i].size

    n = arrays[-1].size
    for k in range(len(arrays)-2, -1, -1):
        n *= arrays[k].size
        m = int(n / arrays[k].size)
        for j in range(1, arrays[k].size):
            out[j*m:(j+1)*m, k+1:] = out[0:m, k+1:]

    # literal unroll
    counter = 0
    for dtype_name in literal_unroll(dtype_names):
        for col in range(n):
            cart_array[dtype_name][col] = out[col][counter]
        counter += 1
    return cart_array
test()

So i am not quite sure what is going on with my current code ... is it me or is it a bug

[quantfreedom]

I ended up having to do this because the numba code wont work in the .py file ... but yeah super super super super weird that if i copy past my second comment into my .py file that it doesn't work but if i run it stand alone it does work

    arrays = (
        leverage_array,
        max_equity_risk_pct_array,
        max_equity_risk_value_array,
        risk_rewards_array,
        size_pct_array,
        size_value_array,
        sl_pcts_array,
        sl_to_be_based_on_array,
        sl_to_be_trail_by_when_pct_from_avg_entry_array,
        sl_to_be_when_pct_from_avg_entry_array,
        sl_to_be_zero_or_entry_array,
        tp_pcts_array,
        tsl_based_on_array,
        tsl_pcts_init_array,
        tsl_trail_by_pct_array,
        tsl_when_pct_from_avg_entry_array,
    )

    # cart array loop
    n = 1
    for x in arrays:
        n *= x.size
    out = np.empty((n, len(arrays)))

    for i in range(len(arrays)):
        m = int(n / arrays[i].size)
        out[:n, i] = np.repeat(arrays[i], m)
        n //= arrays[i].size

    n = arrays[-1].size
    for k in range(len(arrays)-2, -1, -1):
        n *= arrays[k].size
        m = int(n / arrays[k].size)
        for j in range(1, arrays[k].size):
            out[j*m:(j+1)*m, k+1:] = out[0:m, k+1:]

    # Setting variable arrys from cart arrays
    leverage_cart_array = out.T[0]
    max_equity_risk_pct_cart_array = out.T[1]
    max_equity_risk_value_cart_array = out.T[2]
    risk_rewards_cart_array = out.T[3]
    size_pct_cart_array = out.T[4]
    size_value_cart_array = out.T[5]
    sl_pcts_cart_array = out.T[6]
    sl_to_be_based_on_cart_array = out.T[7]
    sl_to_be_trail_by_when_pct_from_avg_entry_cart_array = out.T[8]
    sl_to_be_when_pct_from_avg_entry_cart_array = out.T[9]
    sl_to_be_zero_or_entry_cart_array = out.T[10]
    tp_pcts_cart_array = out.T[11]
    tsl_based_on_cart_array = out.T[12]
    tsl_pcts_init_cart_array = out.T[13]
    tsl_trail_by_cart_array = out.T[14]
    tsl_when_pct_from_avg_entry_cart_array = out.T[15]

[gmarkall]

Thanks for documenting the different experiments and the explanations in the video. I get the impression that maybe the literal unroll pass is mis-transforming the code leading to a variety of different odd error messages, but I'm not well-versed in this area so I'll bring it up at the triage meeting.

[quantfreedom]

@gmarkall thanks for the reply and if you are talking about the meeting that is happening today in a couple of hours i am going to be there as well :)

[gmarkall]

@QuantFreedom1022 Does disabling cache=True in your application change anything?

[github-actions[bot]]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.

[quantfreedom]

I will com comment on this in the next week or two as I'm finishing something up and we'll have to revisit the subject

[github-actions[bot]]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.