search

lalinsky / python-phoenixdb

Phoenix database adapter for Python (migrated to the Apache Phoenix repo)
https://issues.apache.org/jira/browse/PHOENIX-4636
Apache License 2.0
26 stars 24 forks source link

support protobuf c++ implementation? #4

Closed yuz11 closed 6 years ago

yuz11 commented 6 years ago

I pull a large number of data(70w+ rows)，found it pretty slow when it runs to client.py:506 (1~2 second dealing with one bugket data in a fetch())

        response_data = self._apply(request)
        response = responses_pb2.FetchResponse()
        response.ParseFromString(response_data)

I guess pure python protobuf make it slow. python protobuf c++ implement may help would u share the .proto files so that I can make a test by myself?

Regards, Rafer

lalinsky commented 6 years ago

You can get the PB files here: https://github.com/apache/calcite-avatica/tree/master/core/src/main/protobuf

yuz11 commented 6 years ago

thx~I'll take a shot

yuz11 commented 6 years ago

u can close this issue now. I just run a test, protobuf with c++ implement is about 10 times faster than pure python version.But it's just a matter of python protobuf lib version, do not have to rebuild the message. python 3.0 or ++ pip install will do; python 2.x, will have to setup by source code. Regards.Sorry for taking this long.Have been busy these days