spectriclabs / elastic_datashader

:earth_americas: Datashader enabled TMS server with ElasticSearch backend
Apache License 2.0
4 stars 1 forks source link

Add python-datemath for Kibana date-math parsing #2

Closed mrecachinas closed 4 years ago

mrecachinas commented 4 years ago

python-datemath seems to be a fully compliant, still maintained Python implementation of Kibana's date-math.

Note: it's built on arrow, which is Python's version of moment.js (i.e., a human-readable date parsing library).

Also note, it does appear using python-datemath is about 16x slower. See below for some performance metrics:

Before python-datemath

In [1]: from tms_pixellock import convertKibanaTime                                                                                                     
In [2]: import datetime                                                                                                                                 
In [3]: now = datetime.datetime(2020, 5, 12, 15, 0, 0)                                                                                                  
In [4]: %%timeit 
   ...: convertKibanaTime("now-3m", now) 

3.01 µs ± 111 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

After python-datemath

In [1]: from tms_pixellock import convertKibanaTime                                                                                                     
In [2]: import datetime 
In [3]: now = datetime.datetime(2020, 5, 12, 15, 0, 0) 
In [4]: %%timeit 
   ...: convertKibanaTime("now-3m", now)

48.9 µs ± 2.11 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

This is likely driven by relying on Arrow, which has its own overhead. If this is too much of a performance hit, I can roll the fully-compliant, but not yet fully-tested version I built this morning.

mrecachinas commented 4 years ago

For reference, my fully-compliant (but not yet fully tested) implementation (not including the most outer, wrapper function) is

In [8]: from mike_datemath import parse_kibana_time
In [9]: %%timeit 
   ...: parse_kibana_time("-3d", now) 
   ...:  
   ...:                                                                                                                                                 
3.79 µs ± 73.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

With the outer function:

In [1]: from mike_datemath import convert_kibana_time                                                                                                   
In [2]: import datetime                                                                                                                                 
In [3]: now = datetime.datetime(2020, 5, 12, 15, 0, 0)                                                                                                  
In [4]: %%timeit 
   ...: convert_kibana_time("now-3d", now) 
   ...:  
   ...:                                                                                                                                                 
4.59 µs ± 170 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
mrecachinas commented 4 years ago

FWIW, my implementation is in the mike-datemath branch in datemath.py.