pollen-robotics / dtw

DTW (Dynamic Time Warping) python module
GNU General Public License v3.0
1.16k stars 233 forks source link

Unable to allocate 60.3 GiB for an array with shape (89997, 89997) and data type float64 #55

Open rybd111 opened 1 year ago

rybd111 commented 1 year ago
def composite_dis(df_wave, wave_all):
   dtw_dis = dict()
   for i in range(0, len(df_wave)):
        dtw_ij = dict()
        for j in range(0, len(df_wave[i])):
            krec = []
            for k in df_wave[i][j].keys():
                krec.append(k)
                for l in df_wave[i][j].keys():
                    if l not in krec:
                        if k == l:
                            continue
                        else:
                            wave_npk = np.empty( dtype=int, shape=[len(df_wave[i][j][k][k].values), 2] )
                            wave_npl = np.empty( dtype=int, shape=[len(df_wave[i][j][k][k].values), 2] )
                            wave_npk[:, 0] = df_wave[i][j][k]['data_time'].values
                            wave_npk[:, 1] = df_wave[i][j][k][k].values
                            wave_npl[:, 0] = df_wave[i][j][k]['data_time'].values
                            wave_npl[:, 1] = df_wave[i][j][l][l].values

                            manhattan_distance = lambda x, y: np.abs(wave_npk[:, 1] - wave_npl[:, 1])
                            dtw_ij[k+'_'+l] = dtw(wave_npk[:, 1], wave_npl[:, 1], dist=manhattan_distance)
        dtw_dis[str(i)] = dtw_ij
    return dtw_dis`

"I use dtw package, but the np.array emerge the above problem." what should I do?