vapor / postgres-nio

🐘 Non-blocking, event-driven Swift client for PostgreSQL.
https://api.vapor.codes/postgresnio/documentation/postgresnio/
MIT License
304 stars 70 forks source link

Database cursor support #486

Closed enquora closed 2 weeks ago

enquora commented 2 weeks ago

Database fetches for batch reporting and other analytic purposes often involve data volumes sufficient to require query results be made available incrementally. Although PostgreSQL supports fetch cursors for these scenarios, Postgres-nio does not appear to support them.

Supporting server cursors would better accommodate usage cases beyond acting as a web services data store.

It is certainly possible to write client queries which loop over specified batch sizes for large result sets. This is, however, a clumsy and ad-hoc solution. Without cursor support, asyncpg, for example, is likely to remain a superior choice for such cases, with superior economy and clarity of code, and better performance — despite being an interpreted Python package.

It may be the authors consider usage cases beyond web server backing stores as a non-goal. If so, this should be noted.

Joannis commented 2 weeks ago

PostgresNIO definitely does support cursors. The readme shows you how to iterate over this cursor of results through an AsyncSequence called PostgresRowSequence. AsyncSequences are a common way to represent cursors in Swift database drivers.

In PostgresNIO, the driver fetches more data at the rate your application consumes it. If the rate of consumption decreases, the driver slows down the rate at which is loads rows from the server as to not bloat memory usage. If the rate of consumption increases, the driver loads more data from the server to prevent artificial bottlenecks.