supabase / pg_net

A PostgreSQL extension that enables asynchronous (non-blocking) HTTP/HTTPS requests with SQL
https://supabase.github.io/pg_net
Apache License 2.0
226 stars 18 forks source link

Increase default request rate #160

Open steve-chavez opened 1 week ago

steve-chavez commented 1 week ago

Problem

The current request rate of 200 req/s (pg_net.batch_size) is too low for several cases. It's limited this way to save on CPU usage, but this can be improved.

For example, right now the net._http_response tables store the response headers as jsonb. For this, processing is needed on each request/response to convert the headers to jsonb, which consumes CPU.

Solution

Store the response headers as binary and then provide a view that converts them to json on demand. This can be done without a breaking change.

begank commented 1 week ago

+1, I think the value of pg_net.batch_size is far less than 200. By the way, how can I check the value of pg_net.batch_size? @steve-chavez

steve-chavez commented 1 week ago

@begank Try

show pg_net.batch_size;