python-pinot-dbapi / pinot-dbapi

Python DB-API and SQLAlchemy dialect for Pinot
MIT License
19 stars 33 forks source link

Optimize Cursor.fetchall #62

Closed rostan-t closed 10 months ago

rostan-t commented 1 year ago

The method Cursor.fetchall has the following docstring :

Fetch all (remaining) rows of a query result, returning them as a sequence of sequences (e.g. a list of tuples). Note that the cursor's arraysize attribute can affect the performance of this operation. However, the method's implementation is simply :

return list(self)

which creates a list by calling self.fetchone, i.e. self._results.pop(0), once for each element of the list self._results.

This is a performance issue and can be done more straightforwardly.

navina commented 1 year ago

@RostanTabet : Iiuc, you are proposing to return a copy of the internal list object (_results) instead of letting the python's in-built list to construct a list instance by iterating through the objects. Is that right?

I am not that familiar with python internals. Does results = self._results create a copy of the list and assign to the lhs results variable ? Or is it just returning a reference to the list _results ?

xiangfu0 commented 1 year ago

Can you please rebase and we can merge this.