Closed MislavSag closed 4 years ago
This is similar to https://github.com/polakowo/vectorbt/issues/15#issue-611078630
Try to cast your windows
array to np.float64
.
The next version of vectorbt will work with any integer data type.
I have just tried:
# Define params
windows = np.arange(2, 101)
windows = windows.astype(np.float64)
investment = 100 # in $
commission = 0.001 # in %
# Prepare data
ticker = yf.Ticker("BTC-USD")
price = ticker.history(period="max")['Close']
# Generate signals
fast_ma, slow_ma = vbt.MA.from_combinations(price, windows, 2)
but got the same error.
I've done a fresh install and tried the readme example and it worked. I think Numba behaves differently across different platforms, which is painful. I will issue a version 0.6 tomorrow or Wednesday, it will be type agnostic.
Yeah, numba can behave unexpectedly. I will wait for new release than.
There were no new comitts. Probably zou didn't have time to upmake upgrade?
It's in the works. Had to give it more time to refactor portfolio, will release with one bulk update once everything fits (100% on the weekend).
OK, just wanted to check out if I missed anything. I would like to see if your package would fit my needs. I want to backtest my simple ML strategy.
I released the version 0.6 without explicit typing. It requires numba 0.49.
After installation, when I try to import vectorbt, it returns:
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
c:\Users\Mislav\Documents\GitHub\trademl\trademl\modeling\random_forest\train_rf.py in
----> 395 import vectorbt as vbt
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\__init__.py in
----> 1 from vectorbt import utils, accessors, timeseries, widgets, signals, portfolio, indicators, defaults
2
3 # Most important classes
4 from vectorbt.widgets import Indicator, Bar, Scatter, Histogram, Heatmap
5 from vectorbt.portfolio import Portfolio
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\utils\__init__.py in
1 """Utility functions and classes for various parts of `vectorbt`."""
2
----> 3 from vectorbt.utils import checks, combine_fns, common, index_fns, indexing, reshape_fns, accessors
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\utils\checks.py in
3 import numpy as np
4 import pandas as pd
----> 5 from numba.core.registry import CPUDispatcher
6
7 from vectorbt.utils import reshape_fns
ModuleNotFoundError: No module named 'numba.core'
I have the latest version of numba installed.
Which exact version of numba do you have installed? I cannot reproduce this error.
I uninstall and install the package again and now I can import the package, but get following error:
---------------------------------------------------------------------------
TypingError Traceback (most recent call last)
<ipython-input-4-9abe364754ce> in <module>
1 # Generate signals
----> 2 fast_ma, slow_ma = vbt.MA.from_combinations(price, windows, 2)
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py in from_combinations(cls, ts, windows, r, ewm, names, **kwargs)
203 names = ['ma' + str(i+1) for i in range(r)]
204 windows, ewm = reshape_fns.broadcast(windows, ewm, writeable=True)
--> 205 cache_dict = cls.from_params(ts, windows, ewm=ewm, return_cache=True, **kwargs)
206 param_lists = zip(*itertools.combinations(zip(windows, ewm), r))
207 mas = []
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py in from_params(cls, ts, window, ewm, **kwargs)
99 ```
100 """
--> 101 return super().from_params(ts, window, ewm, **kwargs)
102
103 @classmethod
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in from_params(cls, name, return_raw, pipeline_kwargs, *args, **kwargs)
611 results = from_params_pipeline(
612 ts_list, param_list, level_names, len(output_names),
--> 613 custom_func, *new_args, return_raw=return_raw, **kwargs)
614 if return_raw or kwargs.get('return_cache', False):
615 return results
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in from_params_pipeline(ts_list, param_list, level_names, num_outputs, custom_func, pass_lists, pass_2d, param_product, broadcast_kwargs, return_raw, *args, **kwargs)
409 array_list = tuple(map(lambda x: np.asarray(x), ts_list))
410 if pass_lists:
--> 411 output_list = custom_func(array_list, param_list, *args, **kwargs)
412 else:
413 output_list = custom_func(*array_list, *param_list, *args, **kwargs)
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in custom_func(ts_arr_list, param_list, return_cache, cache, *args)
778 # Caching
779 if cache is None and caching_func is not None:
--> 780 cache = caching_func(*ts_arr_list, *param_list, *args)
781 if return_cache:
782 return cache
C:\ProgramData\Anaconda3\lib\site-packages\numba\core\dispatcher.py in _compile_for_args(self, *args, **kws)
399 e.patch_message(msg)
400
--> 401 error_rewrite(e, 'typing')
402 except errors.UnsupportedError as e:
403 # Something unsupported is present in the user code, add help info
C:\ProgramData\Anaconda3\lib\site-packages\numba\core\dispatcher.py in error_rewrite(e, issue_type)
342 raise e
343 else:
--> 344 reraise(type(e), e, None)
345
346 argtypes = []
C:\ProgramData\Anaconda3\lib\site-packages\numba\core\utils.py in reraise(tp, value, tb)
78 value = tp()
79 if value.__traceback__ is not tb:
---> 80 raise value.with_traceback(tb)
81 raise value
82
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of Function(<built-in function getitem>) with argument(s) of type(s): (array(float64, 1d, C), float64)
* parameterized
In definition 0:
All templates rejected with literals.
In definition 1:
All templates rejected without literals.
In definition 2:
All templates rejected with literals.
In definition 3:
All templates rejected without literals.
In definition 4:
All templates rejected with literals.
In definition 5:
All templates rejected without literals.
In definition 6:
All templates rejected with literals.
In definition 7:
All templates rejected without literals.
In definition 8:
All templates rejected with literals.
In definition 9:
All templates rejected without literals.
In definition 10:
All templates rejected with literals.
In definition 11:
All templates rejected without literals.
In definition 12:
TypeError: unsupported array index type float64 in [float64]
raised from C:\ProgramData\Anaconda3\lib\site-packages\numba\core\typing\arraydecl.py:69
In definition 13:
TypeError: unsupported array index type float64 in [float64]
raised from C:\ProgramData\Anaconda3\lib\site-packages\numba\core\typing\arraydecl.py:69
In definition 14:
All templates rejected with literals.
In definition 15:
All templates rejected without literals.
This error is usually caused by passing an argument of a type that is unsupported by the named function.
[1] During: typing of intrinsic-call at C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\timeseries\nb.py (373)
File "..\..\ProgramData\Anaconda3\lib\site-packages\vectorbt\timeseries\nb.py", line 373:
def rolling_mean_1d_nb(a, window, minp=None):
<source elided>
else:
window_len = window - (nancnt - nancnt_arr[i-window])
^
[1] During: resolving callee type: type(CPUDispatcher(<function rolling_mean_1d_nb at 0x0000014CA5F82DC8>))
[2] During: typing of call at C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\timeseries\nb.py (387)
[3] During: resolving callee type: type(CPUDispatcher(<function rolling_mean_1d_nb at 0x0000014CA5F82DC8>))
[4] During: typing of call at C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\timeseries\nb.py (387)
File "..\..\ProgramData\Anaconda3\lib\site-packages\vectorbt\timeseries\nb.py", line 387:
def rolling_mean_nb(a, window, minp=None):
<source elided>
for col in range(a.shape[1]):
result[:, col] = rolling_mean_1d_nb(a[:, col], window, minp=minp)
^
[1] During: resolving callee type: type(CPUDispatcher(<function rolling_mean_nb at 0x0000014CA5F8A558>))
[2] During: typing of call at C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py (44)
[3] During: resolving callee type: type(CPUDispatcher(<function rolling_mean_nb at 0x0000014CA5F8A558>))
[4] During: typing of call at C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py (44)
[5] During: resolving callee type: type(CPUDispatcher(<function rolling_mean_nb at 0x0000014CA5F8A558>))
[6] During: typing of call at C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py (44)
File "..\..\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py", line 44:
def ma_caching_nb(ts, windows, ewms):
<source elided>
else:
ma = timeseries.nb.rolling_mean_nb(ts, windows[i], minp=windows[i])
^
when I try to execute:
fast_ma, slow_ma = vbt.MA.from_combinations(price, windows, 2)
from the sample code from the readme.
Numba version: numba 0.49.1 pypi_0 pypi
It's getting weirder. Can you try windows = windows.astype(np.int32)
I change it:
import vectorbt as vbt
import numpy as np
import yfinance as yf
# Define params
windows = np.arange(2, 101)
windows = windows.astype(np.int32)
investment = 100 # in $
commission = 0.001 # in %
# Prepare data
ticker = yf.Ticker("BTC-USD")
price = ticker.history(period="max")['Close']
fast_ma, slow_ma = vbt.MA.from_combinations(price, windows, 2)
and now it returns:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-10-18625ececc3b> in <module>
13 price = ticker.history(period="max")['Close']
14
---> 15 fast_ma, slow_ma = vbt.MA.from_combinations(price, windows, 2)
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py in from_combinations(cls, ts, windows, r, ewm, names, **kwargs)
208 for i, param_list in enumerate(param_lists):
209 i_windows, i_ewm = zip(*param_list)
--> 210 mas.append(cls.from_params(ts, i_windows, ewm=i_ewm, cache=cache_dict, name=names[i], **kwargs))
211 return tuple(mas)
212
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py in from_params(cls, ts, window, ewm, **kwargs)
99 ```
100 """
--> 101 return super().from_params(ts, window, ewm, **kwargs)
102
103 @classmethod
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in from_params(cls, name, return_raw, pipeline_kwargs, *args, **kwargs)
611 results = from_params_pipeline(
612 ts_list, param_list, level_names, len(output_names),
--> 613 custom_func, *new_args, return_raw=return_raw, **kwargs)
614 if return_raw or kwargs.get('return_cache', False):
615 return results
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in from_params_pipeline(ts_list, param_list, level_names, num_outputs, custom_func, pass_lists, pass_2d, param_product, broadcast_kwargs, return_raw, *args, **kwargs)
409 array_list = tuple(map(lambda x: np.asarray(x), ts_list))
410 if pass_lists:
--> 411 output_list = custom_func(array_list, param_list, *args, **kwargs)
412 else:
413 output_list = custom_func(*array_list, *param_list, *args, **kwargs)
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in custom_func(ts_arr_list, param_list, return_cache, cache, *args)
793 typed_param_tuples,
794 *args,
--> 795 *cache)
796 else:
797 if num_outputs > 1:
C:\ProgramData\Anaconda3\lib\site-packages\numba\typed\dictobject.py in impl()
731 ix, val = _dict_lookup(d, castedkey, hash(castedkey))
732 if ix == DKIX.EMPTY:
--> 733 raise KeyError()
734 elif ix < DKIX.EMPTY:
735 raise AssertionError("internal dict error during lookup")
KeyError:
That's a bit better. Can you run this after importing all the other stuff:
import itertools
def from_combinations(ts, windows, r, ewm=False, names=None, **kwargs):
if names is None:
names = ['ma' + str(i+1) for i in range(r)]
print(windows.dtype)
windows, ewm = vbt.utils.reshape_fns.broadcast(windows, ewm, writeable=True)
print(windows.dtype)
cache_dict = vbt.MA.from_params(ts, windows, ewm=ewm, return_cache=True, **kwargs)
param_lists = zip(*itertools.combinations(zip(windows, ewm), r))
mas = []
for i, param_list in enumerate(param_lists):
i_windows, i_ewm = zip(*param_list)
print(np.asarray(i_windows).dtype)
mas.append(vbt.MA.from_params(ts, i_windows, ewm=i_ewm, cache=cache_dict, name=names[i], **kwargs))
return tuple(mas)
from_combinations(price, windows, 2)
Code:
import vectorbt as vbt
import numpy as np
import yfinance as yf
# Define params
windows = np.arange(2, 101)
windows = windows.astype(np.int32)
investment = 100 # in $
commission = 0.001 # in %
# Prepare data
ticker = yf.Ticker("BTC-USD")
price = ticker.history(period="max")['Close']
import itertools
def from_combinations(ts, windows, r, ewm=False, names=None, **kwargs):
if names is None:
names = ['ma' + str(i+1) for i in range(r)]
print(windows.dtype)
windows, ewm = vbt.utils.reshape_fns.broadcast(windows, ewm, writeable=True)
print(windows.dtype)
cache_dict = vbt.MA.from_params(ts, windows, ewm=ewm, return_cache=True, **kwargs)
param_lists = zip(*itertools.combinations(zip(windows, ewm), r))
mas = []
for i, param_list in enumerate(param_lists):
i_windows, i_ewm = zip(*param_list)
print(np.asarray(i_windows).dtype)
mas.append(vbt.MA.from_params(ts, i_windows, ewm=i_ewm, cache=cache_dict, name=names[i], **kwargs))
return tuple(mas)
from_combinations(price, windows, 2)
Error:
int32
int32
int32
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-13-d2cd9b19a7d9> in <module>
30 return tuple(mas)
31
---> 32 from_combinations(price, windows, 2)
<ipython-input-13-d2cd9b19a7d9> in from_combinations(ts, windows, r, ewm, names, **kwargs)
27 i_windows, i_ewm = zip(*param_list)
28 print(np.asarray(i_windows).dtype)
---> 29 mas.append(vbt.MA.from_params(ts, i_windows, ewm=i_ewm, cache=cache_dict, name=names[i], **kwargs))
30 return tuple(mas)
31
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\indicators.py in from_params(cls, ts, window, ewm, **kwargs)
99 ```
100 """
--> 101 return super().from_params(ts, window, ewm, **kwargs)
102
103 @classmethod
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in from_params(cls, name, return_raw, pipeline_kwargs, *args, **kwargs)
611 results = from_params_pipeline(
612 ts_list, param_list, level_names, len(output_names),
--> 613 custom_func, *new_args, return_raw=return_raw, **kwargs)
614 if return_raw or kwargs.get('return_cache', False):
615 return results
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in from_params_pipeline(ts_list, param_list, level_names, num_outputs, custom_func, pass_lists, pass_2d, param_product, broadcast_kwargs, return_raw, *args, **kwargs)
409 array_list = tuple(map(lambda x: np.asarray(x), ts_list))
410 if pass_lists:
--> 411 output_list = custom_func(array_list, param_list, *args, **kwargs)
412 else:
413 output_list = custom_func(*array_list, *param_list, *args, **kwargs)
C:\ProgramData\Anaconda3\lib\site-packages\vectorbt\indicators\factory.py in custom_func(ts_arr_list, param_list, return_cache, cache, *args)
793 typed_param_tuples,
794 *args,
--> 795 *cache)
796 else:
797 if num_outputs > 1:
C:\ProgramData\Anaconda3\lib\site-packages\numba\typed\dictobject.py in impl()
731 ix, val = _dict_lookup(d, castedkey, hash(castedkey))
732 if ix == DKIX.EMPTY:
--> 733 raise KeyError()
734 elif ix < DKIX.EMPTY:
735 raise AssertionError("internal dict error during lookup")
KeyError:
And this?
import itertools
def from_combinations(ts, windows, r, ewm=False, names=None, **kwargs):
if names is None:
names = ['ma' + str(i+1) for i in range(r)]
windows, ewm = vbt.utils.reshape_fns.broadcast(windows, ewm, writeable=True)
cache_dict = vbt.MA.from_params(ts, windows, ewm=ewm, return_cache=True, **kwargs)
print(cache_dict)
param_lists = zip(*itertools.combinations(zip(windows, ewm), r))
mas = []
for i, param_list in enumerate(param_lists):
i_windows, i_ewm = zip(*param_list)
print(i_windows)
print(i_ewm)
mas.append(vbt.MA.from_params(ts, i_windows, ewm=i_ewm, cache=cache_dict, name=names[i], **kwargs))
return tuple(mas)
from_combinations(price, np.arange(1, 5), 2)
After change:
KeyError Traceback (most recent call last)
<ipython-input-14-6b500a098cf6> in <module>
30 return tuple(mas)
31
---> 32 from_combinations(price, np.arange(1, 5), 2)
<ipython-input-14-6b500a098cf6> in from_combinations(ts, windows, r, ewm, names, **kwargs)
20 windows, ewm = vbt.utils.reshape_fns.broadcast(windows, ewm, writeable=True)
21 cache_dict = vbt.MA.from_params(ts, windows, ewm=ewm, return_cache=True, **kwargs)
---> 22 print(cache_dict)
23 param_lists = zip(*itertools.combinations(zip(windows, ewm), r))
24 mas = []
C:\ProgramData\Anaconda3\lib\site-packages\numba\typed\typeddict.py in __str__(self)
181 def __str__(self):
182 buf = []
--> 183 for k, v in self.items():
184 buf.append("{}: {}".format(k, v))
185 return '{{{0}}}'.format(', '.join(buf))
C:\ProgramData\Anaconda3\lib\_collections_abc.py in __iter__(self)
742 def __iter__(self):
743 for key in self._mapping:
--> 744 yield (key, self._mapping[key])
745
746 ItemsView.register(dict_items)
C:\ProgramData\Anaconda3\lib\site-packages\numba\typed\typeddict.py in __getitem__(self, key)
149 raise KeyError(key)
150 else:
--> 151 return _getitem(self, key)
152
153 def __setitem__(self, key, value):
C:\ProgramData\Anaconda3\lib\site-packages\numba\typed\dictobject.py in impl()
731 ix, val = _dict_lookup(d, castedkey, hash(castedkey))
732 if ix == DKIX.EMPTY:
--> 733 raise KeyError()
734 elif ix < DKIX.EMPTY:
735 raise AssertionError("internal dict error during lookup")
KeyError:
Plus warning:
C:\ProgramData\Anaconda3\lib\site-packages\numba\typed\typeddict.py:39: NumbaTypeSafetyWarning:
unsafe cast from UniTuple(int64 x 2) to Tuple(int32, int64). Precision may be lost.
I guess I know where the error is coming from. Can you try whether the following code will succeed and what's your output?
from numba import njit
from vectorbt import timeseries
@njit
def ma_caching_nb(ts, windows, ewms):
"""Numba-compiled caching function for `MA`."""
cache_dict = dict()
for i in range(windows.shape[0]):
h = hash((windows[i], ewms[i]))
if h not in cache_dict:
if ewms[i]:
ma = timeseries.nb.ewm_mean_nb(ts, windows[i], minp=windows[i])
else:
ma = timeseries.nb.rolling_mean_nb(ts, windows[i], minp=windows[i])
cache_dict[h] = ma
return cache_dict
print(ma_caching_nb(price.vbt.to_2d_array(), np.asarray([1, 2, 3, 3]), np.asarray([True, False, False, False])))
That code works. Here is the output:
{3713081631935493181: [[ 457.33]
[ 424.44]
[ 394.8 ]
...
[9593.9 ]
[8756.43]
[8791.9 ]], 3713082714464823281: [[ nan]
[ 440.885]
[ 409.62 ]
...
[9718.285]
[9175.165]
[8774.165]], 3713083796993070856: [[ nan]
[ nan]
[ 425.52333333]
...
[9796.03 ]
[9397.66666667]
[9047.41 ]]}
The fix is now on master branch. You can reinstall vectorbt completely and try again. Hopefully it works now.
I can reproduce a readme example now. I think this can be closed. great work.
Finally, great!
I am trying to reproduce example from readme. I get error in this line
The error message is:
I haven't change anything in the code. Looks like numba error?