I got a time-series dataset containing 27 columns X 782 rows. I found that once the dataset exceeded 20000 float-point data. fastdtw will throw IndexError: tuple index out of range.
Is there a way to fix this because I need to cluster at least 50 columns X 782 rows.
David Fung
Environment: Windows 10 Home, Python 3.8.3, IPython 7.16.1
The following is the dump generated by the Spyder IDE:
runfile('C:/Users/dfoo1/anaconda3/AIOPython/SP500MK02M02_test.py', wdir='C:/Users/dfoo1/anaconda3/AIOPython')
Scaling...
Applying Trend Clustering...
Dataset dimensions: 21087
15176786 function calls (15176752 primitive calls) in 4.859 seconds
i-camouflage
commented
2 months ago
I got a time-series dataset containing 27 columns X 782 rows. I found that once the dataset exceeded 20000 float-point data. fastdtw will throw IndexError: tuple index out of range. Is there a way to fix this because I need to cluster at least 50 columns X 782 rows. David Fung Environment: Windows 10 Home, Python 3.8.3, IPython 7.16.1
The following is the dump generated by the Spyder IDE: runfile('C:/Users/dfoo1/anaconda3/AIOPython/SP500MK02M02_test.py', wdir='C:/Users/dfoo1/anaconda3/AIOPython') Scaling...
Applying Trend Clustering...
Dataset dimensions: 21087 15176786 function calls (15176752 primitive calls) in 4.859 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function) 5 0.000 0.000 0.000 0.000:1017(_handle_fromlist)
1 0.000 0.000 4.859 4.859 :1()
2 0.000 0.000 4.836 2.418 SP500MK02M02_test.py:144(fastdtw_distance)
1 0.000 0.000 4.859 4.859 SP500MK02M02_test.py:57(fit)
1 0.022 0.022 0.022 0.022 SP500MK02M02_test.py:77()
1 0.000 0.000 4.836 4.836 SP500MK02M02_test.py:81()
2 0.000 0.000 4.836 2.418 SP500MK02M02_test.py:82()
8/4 0.000 0.000 0.000 0.000 _asarray.py:16(asarray)
8 0.000 0.000 0.000 0.000 _asarray.py:88(asanyarray)
3 0.000 0.000 0.000 0.000 _dtype.py:319(_name_includes_bit_suffix)
3 0.000 0.000 0.000 0.000 _dtype.py:333(_name_get)
3 0.000 0.000 0.000 0.000 _dtype.py:36(_kind_name)
6 0.000 0.000 0.000 0.000 _internal.py:830(npy_ctypes_check)
4 0.000 0.000 0.000 0.000 _ufunc_config.py:139(geterr)
4 0.000 0.000 0.000 0.000 _ufunc_config.py:39(seterr)
2 0.000 0.000 0.000 0.000 _ufunc_config.py:437(init)
2 0.000 0.000 0.000 0.000 _ufunc_config.py:441(enter)
2 0.000 0.000 0.000 0.000 _ufunc_config.py:446(exit)
17 0.000 0.000 0.000 0.000 abc.py:96(instancecheck)
1 0.000 0.000 0.000 0.000 base.py:1026(iter)
1 0.000 0.000 0.000 0.000 base.py:1186(name)
14 0.000 0.000 0.000 0.000 base.py:247(is_dtype)
2 0.000 0.000 0.000 0.000 base.py:5294(ensure_index)
1 0.000 0.000 0.000 0.000 base.py:5388(default_index)
5 0.000 0.000 0.000 0.000 base.py:5394(maybe_extract_name)
1 0.000 0.000 0.000 0.000 base.py:592(_reset_identity)
3 0.000 0.000 0.000 0.000 blocks.py:118(init)
3 0.000 0.000 0.000 0.000 blocks.py:129(_check_ndim)
2 0.000 0.000 0.000 0.000 blocks.py:187(is_categorical_astype)
2 0.000 0.000 0.000 0.000 blocks.py:202(external_values)
3 0.000 0.000 0.000 0.000 blocks.py:213(internal_values)
4 0.000 0.000 0.000 0.000 blocks.py:219(array_values)
2 0.000 0.000 0.000 0.000 blocks.py:225(get_values)
5 0.000 0.000 0.000 0.000 blocks.py:247(mgr_locs)
3 0.000 0.000 0.000 0.000 blocks.py:251(mgr_locs)
2 0.000 0.000 0.000 0.000 blocks.py:2597(init)
3 0.000 0.000 0.000 0.000 blocks.py:2987(get_block_type)
3 0.000 0.000 0.000 0.000 blocks.py:3033(make_block)
2 0.000 0.000 0.000 0.000 blocks.py:3059(_extend_blocks)
2 0.000 0.000 0.000 0.000 blocks.py:335(shape)
5 0.000 0.000 0.000 0.000 blocks.py:339(dtype)
2 0.000 0.000 0.000 0.000 blocks.py:554(astype)
1 0.000 0.000 0.000 0.000 cast.py:1088(maybe_castable)
1 0.000 0.000 0.000 0.000 cast.py:1102(maybe_infer_to_datetimelike)
1 0.000 0.000 0.000 0.000 cast.py:1209(maybe_cast_to_datetime)
1 0.000 0.000 0.000 0.000 cast.py:1483(construct_1d_ndarray_preserving_na)
2 0.000 0.000 0.000 0.000 cast.py:799(astype_nansafe)
14 0.000 0.000 0.000 0.000 common.py:1565(is_extension_array_dtype)
4 0.000 0.000 0.000 0.000 common.py:1672(_get_dtype)
16 0.000 0.000 0.000 0.000 common.py:1708(_is_dtype_type)
2 0.000 0.000 0.000 0.000 common.py:179(ensure_python_int)
1 0.000 0.000 0.000 0.000 common.py:183(all_none)
9 0.000 0.000 0.000 0.000 common.py:1844(pandas_dtype)
2 0.000 0.000 0.000 0.000 common.py:187()
15 0.000 0.000 0.000 0.000 common.py:206(classes)
15 0.000 0.000 0.000 0.000 common.py:208()
1 0.000 0.000 0.000 0.000 common.py:211(classes_and_not_datetimelike)
1 0.000 0.000 0.000 0.000 common.py:216()
7 0.000 0.000 0.000 0.000 common.py:222(is_object_dtype)
3 0.000 0.000 0.000 0.000 common.py:252(is_sparse)
1 0.000 0.000 0.000 0.000 common.py:275(maybe_iterable_to_list)
3 0.000 0.000 0.000 0.000 common.py:339(is_categorical)
4 0.000 0.000 0.000 0.000 common.py:372(is_datetime64_dtype)
3 0.000 0.000 0.000 0.000 common.py:403(is_datetime64tz_dtype)
4 0.000 0.000 0.000 0.000 common.py:441(is_timedelta64_dtype)
3 0.000 0.000 0.000 0.000 common.py:472(is_period_dtype)
3 0.000 0.000 0.000 0.000 common.py:506(is_interval_dtype)
5 0.000 0.000 0.000 0.000 common.py:542(is_categorical_dtype)
2 0.000 0.000 0.000 0.000 common.py:685(is_dtype_equal)
1 0.000 0.000 0.000 0.000 common.py:775(is_integer_dtype)
1 0.000 0.000 0.000 0.000 construction.py:337(extract_array)
1 0.000 0.000 0.000 0.000 construction.py:388(sanitize_array)
1 0.000 0.000 0.000 0.000 construction.py:506(_try_cast)
3 0.000 0.000 0.000 0.000 construction.py:570(is_empty_data)
3 0.000 0.000 0.000 0.000 dtypes.py:1124(is_dtype)
16 0.000 0.000 0.000 0.000 dtypes.py:75(find)
3 0.000 0.000 0.000 0.000 dtypes.py:917(is_dtype)
12 0.780 0.065 1.599 0.133 fastdtw.py:133(dtw)
2 0.000 0.000 0.000 0.000 fastdtw.py:136()
502384 0.093 0.000 0.093 0.000 fastdtw.py:137()
6140 0.002 0.000 0.002 0.000 fastdtw.py:138()
1507116 0.117 0.000 0.117 0.000 fastdtw.py:143()
2 0.004 0.002 4.836 2.418 fastdtw.py:15(fastdtw)
20 0.000 0.000 0.002 0.000 fastdtw.py:153( reduce_by_half)
20 0.002 0.000 0.002 0.000 fastdtw.py:154()
10 1.660 0.166 3.097 0.310 fastdtw.py:157(expand_window)
5219612 0.755 0.000 0.755 0.000 fastdtw.py:160()
12/2 0.134 0.011 4.832 2.416 fastdtw.py:64(fastdtw)
4 0.000 0.000 0.000 0.000 fastdtw.py:78(prep_inputs)
2 0.000 0.000 0.004 0.002 fastdtw.py:98(dtw)
66 0.000 0.000 0.000 0.000 generic.py:10(_check)
3 0.000 0.000 0.000 0.000 generic.py:190(init)
2 0.000 0.000 0.000 0.000 generic.py:238(attrs)
2 0.000 0.000 0.000 0.000 generic.py:5235(finalize)
11 0.000 0.000 0.000 0.000 generic.py:5257(getattr)
3 0.000 0.000 0.000 0.000 generic.py:5276(setattr)
2 0.000 0.000 0.000 0.000 generic.py:5563(astype)
5 0.000 0.000 0.000 0.000 inference.py:299(is_dict_like)
11 0.000 0.000 0.000 0.000 inference.py:325()
12 0.000 0.000 0.000 0.000 inference.py:358(is_hashable)
2 0.000 0.000 0.000 0.000 inspect.py:72(isclass)
3 0.000 0.000 0.000 0.000 managers.py:1467( init)
12 0.000 0.000 0.000 0.000 managers.py:1520(_block)
2 0.000 0.000 0.000 0.000 managers.py:1544(index)
3 0.000 0.000 0.000 0.000 managers.py:1548(dtype)
2 0.000 0.000 0.000 0.000 managers.py:1562(external_values)
3 0.000 0.000 0.000 0.000 managers.py:1565(internal_values)
2 0.000 0.000 0.000 0.000 managers.py:1582(_consolidate_inplace)
3 0.000 0.000 0.000 0.000 managers.py:232(items)
3 0.000 0.000 0.000 0.000 managers.py:314(len)
2 0.000 0.000 0.000 0.000 managers.py:368(apply)
2 0.000 0.000 0.000 0.000 managers.py:419()
2 0.000 0.000 0.000 0.000 managers.py:581(astype)
3 0.000 0.000 0.000 0.000 numeric.py:155(is_all_dates)
1 0.000 0.000 0.000 0.000 numeric.py:83(_validatedtype)
10 0.000 0.000 0.000 0.000 numerictypes.py:293(issubclass)
5 0.000 0.000 0.000 0.000 numerictypes.py:365(issubdtype)
4 0.000 0.000 0.000 0.000 numpy.py:138(init)
4 0.000 0.000 0.000 0.000 numpy.py:183( array)
4 0.000 0.000 0.000 0.000 numpy_.py:42(init)
1 0.000 0.000 0.000 0.000 range.py:131(_simple_new)
5 0.000 0.000 0.000 0.000 range.py:675(len)
1 0.000 0.000 0.000 0.000 range.py:83(new)
3 0.000 0.000 0.000 0.000 series.py:183(init)
2 0.000 0.000 0.000 0.000 series.py:359(_constructor)
2/1 0.000 0.000 4.836 4.836 series.py:3719(apply)
3 0.000 0.000 0.000 0.000 series.py:376(_set_axis)
3 0.000 0.000 0.000 0.000 series.py:403(_set_subtyp)
3 0.000 0.000 0.000 0.000 series.py:414(dtype)
5 0.000 0.000 0.000 0.000 series.py:428(name)
5 0.000 0.000 0.000 0.000 series.py:432(name)
2 0.000 0.000 0.000 0.000 series.py:438(values)
3 0.000 0.000 0.000 0.000 series.py:480(_values)
4 0.000 0.000 0.000 0.000 series.py:515(array)
3 0.000 0.000 0.000 0.000 series.py:548(len)
4 0.000 0.000 0.000 0.000 series.py:707(array)
502372 0.206 0.000 0.247 0.000 utils.py:25(abs_distance)
1 0.000 0.000 0.000 0.000 {built-in method new__ of type object at 0x00007FF9F575C810}
17 0.000 0.000 0.000 0.000 {built-in method _abc._abc_instancecheck}
502372 0.041 0.000 0.041 0.000 {built-in method builtins.abs}
6 0.000 0.000 0.000 0.000 {built-in method builtins.all}
2 0.000 0.000 0.000 0.000 {built-in method builtins.callable}
1 0.000 0.000 4.859 4.859 {built-in method builtins.exec}
102 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}
19 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
12 0.000 0.000 0.000 0.000 {built-in method builtins.hash}
218 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
75 0.000 0.000 0.000 0.000 {built-in method builtins.issubclass}
138/128 0.000 0.000 0.000 0.000 {built-in method builtins.len}
502372 0.359 0.000 0.475 0.000 {built-in method builtins.min}
1 0.000 0.000 0.000 0.000 {built-in method numpy.arange}
18/10 0.000 0.000 0.000 0.000 {built-in method numpy.array}
8 0.000 0.000 0.000 0.000 {built-in method numpy.geterrobj}
4 0.000 0.000 0.000 0.000 {built-in method numpy.seterrobj}
5928830 0.653 0.000 0.653 0.000 {method 'add' of 'set' objects}
504311 0.030 0.000 0.030 0.000 {method 'append' of 'list' objects}
4 0.000 0.000 0.000 0.000 {method 'astype' of 'numpy.ndarray' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 {method 'format' of 'str' objects}
2 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects}
2 0.000 0.000 0.000 0.000 {method 'ravel' of 'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {method 'reshape' of 'numpy.ndarray' objects}
11 0.000 0.000 0.000 0.000 {method 'reverse' of 'list' objects}
7 0.000 0.000 0.000 0.000 {pandas._libs.lib.is_list_like}
2 0.000 0.000 0.000 0.000 {pandas._libs.lib.is_scalar}
2/1 0.000 0.000 4.836 4.836 {pandas._libs.lib.map_infer}
Traceback (most recent call last):
File "C:\Users\dfoo1\anaconda3\AIOPython\SP500MK02M02_test.py", line 219, in
cProfile.run('trendy.fit(stockPriceList)')
File "C:\Users\dfoo1\anaconda3\lib\cProfile.py", line 16, in run return _pyprofile._Utils(Profile).run(statement, filename, sort)
File "C:\Users\dfoo1\anaconda3\lib\profile.py", line 53, in run prof.run(statement)
File "C:\Users\dfoo1\anaconda3\lib\cProfile.py", line 95, in run return self.runctx(cmd, dict, dict)
File "C:\Users\dfoo1\anaconda3\lib\cProfile.py", line 100, in runctx exec(cmd, globals, locals)
File "", line 1, in
File "C:\Users\dfoo1\anaconda3\AIOPython\SP500MK02M02_test.py", line 80, in fit X_idx.apply(
File "C:\Users\dfoo1\anaconda3\lib\site-packages\pandas\core\series.py", line 3848, in apply mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas_libs\lib.pyx", line 2329, in pandas._libs.lib.map_infer
File "C:\Users\dfoo1\anaconda3\AIOPython\SP500MK02M02_test.py", line 81, in
lambda x: X_idx.apply(
File "C:\Users\dfoo1\anaconda3\lib\site-packages\pandas\core\series.py", line 3848, in apply mapped = lib.map_infer(values, f, convert=convert_dtype)
File "pandas_libs\lib.pyx", line 2329, in pandas._libs.lib.map_infer
File "C:\Users\dfoo1\anaconda3\AIOPython\SP500MK02M02_test.py", line 82, in
lambda y: self.fastdtw_distance(X[x], X[y]))))
File "C:\Users\dfoo1\anaconda3\AIOPython\SP500MK02M02_test.py", line 166, in fastdtw_distance return fastdtw(x, y, radius, dist=d)[0]
File "C:\Users\dfoo1\anaconda3\lib\site-packages\fastdtw\fastdtw.py", line 53, in fastdtw return __fastdtw(x, y, radius, dist)
File "C:\Users\dfoo1\anaconda3\lib\site-packages\fastdtw\fastdtw.py", line 75, in fastdtw return dtw(x, y, window, dist=dist)
File "C:\Users\dfoo1\anaconda3\lib\site-packages\fastdtw\fastdtw.py", line 148, in __dtw i, j = D[i, j][1], D[i, j][2]
IndexError: tuple index out of range