search

CartoDB / CartoDB-SQL-API

CartoDB SQL API
BSD 3-Clause "New" or "Revised" License
62 stars 63 forks source link

Disable streaming mode by redis field "streaming_mode" #669

Closed manmorjim closed 3 years ago

manmorjim commented 3 years ago

:warning: This PR has to be merged and deployed before merging this.

Changes

Reading the field streaming_mode from db parameters of redis which indicates the operation mode for the SQL API. If true (default) then the SQL API streams the results (default behaviour). If this field is set to false the the SQL API will return all the results at once.

More info in CH: https://app.clubhouse.io/cartoteam/story/106733/turn-sql-api-transactional-and-improve-the-error-handling

dgaubert commented 3 years ago

What happens when the user that has the streaming mode disabled sends a query for a table of 10GB?

manmorjim commented 3 years ago

What happens when the user that has the streaming mode disabled sends a query for a table of 10GB?

That's a good point! We should establish a quota limit for those cases where the streaming_mode is disabled. This limit could be defined in the config such as db_max_row_size.

dgaubert commented 3 years ago

what happens when the available memory is less than the threshold or even the memory needed to handle that query?.

thedae commented 3 years ago

Closing since these changes are not going to be used