benthosdev / benthos

Fancy stream processing made operationally mundane
https://www.benthos.dev
MIT License
7.68k stars 752 forks source link

Allow sqlite max_page_count to be set #2511

Open pudalangai opened 1 month ago

pudalangai commented 1 month ago

Allow setting max_page_count pragma for sqlite. This acts similar to limit in memory buffer. Once set will limit the db size to page_size * max_page_count. While page_size pragma can also be set, it requires a new db or VACUUM to be done, and also the default 4096 is recommended. For ref: https://www.sqlite.org/pragma.html#pragma_max_page_count, https://www.sqlite.org/limits.html

I've thought of adding field similar to limit in memory buffer that takes in bytes. The conversions of bytes to page_count wont be exact and might be confusing on what value was set, i'e maxPageCount = limit / page_size; unless page_size is also modified. Feedbacks are welcome.