Closed BaiBaiHi closed 9 months ago
Speedup of Tickstore read datetime index conversion by ~40x.
Overall runtime improvement is around 3-4x on average, but this can vary depending on the mongo latency.
%timeit pd.DatetimeIndex(np.concatenate(rtn[INDEX]).astype('datetime64[ms]'), tz='UTC') 1.76 s ± 59.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit pd.to_datetime(np.concatenate(rtn[INDEX]), utc=True, unit='ms') 1min 16s ± 1.74 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
In the current implementation, the datetime conversion takes ~60% of the total runtime:
The new implementation drops it down to ~5%
Output of both implementations is identical (tested on 22.5 million values).
Thanks!
Speedup of Tickstore read datetime index conversion by ~40x.
Overall runtime improvement is around 3-4x on average, but this can vary depending on the mongo latency.
In the current implementation, the datetime conversion takes ~60% of the total runtime:
The new implementation drops it down to ~5%
Output of both implementations is identical (tested on 22.5 million values).