search

libsql / sqld

LibSQL with extended capabilities like HTTP protocol, replication, and more.
https://libsql.org
903 stars 38 forks source link

Consider moving to a stream replication for libsql or adding pagination to `BatchLogEntries` #647

Open LucioFranco opened 1 year ago

LucioFranco commented 1 year ago

It is possible for larger databases for BatchLogEntries to be large. This will consume a lot of memory and network bandwidth without the ability to apply back pressure (as you would get with discrete smaller messages). We should consider moving the client to either a stream based api or a paginated BatchLogEntries.

For now the temporary fix is to remove the max decoding message size on the client side.

neubaner commented 1 year ago

@LucioFranco pushed a PR to libsql that might close this! https://github.com/libsql/libsql/pull/350