taosdata / taos-connector-python

TDengine connector for Python
https://www.taosdata.com/cn/documentation/connector#python
MIT License
48 stars 14 forks source link

taos_fetch_block _v3 is slow #204

Closed hadrianl closed 1 year ago

hadrianl commented 1 year ago

https://github.com/taosdata/taos-connector-python/blob/4ab57be79615accd9bdc5868fe361a87b91642c1/taos/cinterface.py#L504

I just runned some test and found that taos_fetch_block_v3 is pretty slow here, especaily :

it seems too much cast, too much python loop.it cost at least 80% time to parse the data from c to python. is it any plan to improve this?

sangshuduo commented 1 year ago

Would you like to contribute to improving it? :)

hadrianl commented 1 year ago

I would try to submit a PR when I`m available

hadrianl commented 1 year ago

is it ok to accept the code like this to improve the performance:

markcheney1 commented 1 year ago

is it ok to accept the code like this to improve the performance:

  • Cython ,setuptools and compiler are needed
  • Maybe make it optional when fetch data into python object.
  • In my benchmark test, 2~4 times faster than using ctypes

用了pr合入的代码,速度有至少100%的提升,非常了不起。目前我也在探索数据读取的加速,要不然用taospy做回测就要非常久