armink / FlashDB

An ultra-lightweight database that supports key-value and time series data | 一款支持 KV 数据和时序数据的超轻量级数据库
Apache License 2.0
1.92k stars 435 forks source link

建议:增加一个用户可配置的cache,用于加速遍历flash #331

Open ckforce opened 1 week ago

ckforce commented 1 week ago

某些芯片的片内flash读取需要清理cpu cache,开关中断等耗时操作,flashdb每次都是小数据量的读区flash内存,增加了读取次数,造成耗时变长,能否考虑增加一个读取cache,比如256字节,减少读取次数。

考虑内存占用的问题,可以由用户来配置这个cache的大小。

下面我自己测试的log,可以看到查找key的时候,有大量读取附近地址的操作,完全可以合并起来一次性读取 [004837] - D/fal_port: read: offset = 0x000E4CAC size = 4

[004843] - D/fal_port: read: offset = 0x000E492C size = 20

[004850] - D/fal_port: read: offset = 0x000E4CB0 size = 32

[004857] - D/fal_port: read: offset = 0x000E4CB0 size = 40

[004864] - D/fal_port: read: offset = 0x000E4CD8 size = 32

[004870] - D/fal_port: read: offset = 0x000E4CF8 size = 32

[004877] - D/fal_port: read: offset = 0x000E4D18 size = 32

[004884] - D/fal_port: read: offset = 0x000E4D38 size = 32

[004891] - D/fal_port: read: offset = 0x000E4D58 size = 32

[004898] - D/fal_port: read: offset = 0x000E4D78 size = 32

[004904] - D/fal_port: read: offset = 0x000E4D98 size = 32

[004911] - D/fal_port: read: offset = 0x000E4DB8 size = 32

[004918] - D/fal_port: read: offset = 0x000E4DD8 size = 24

[004925] - D/fal_port: read: offset = 0x000E4CD8 size = 8

[004932] - D/fal_port: read: offset = 0x000E4DF0 size = 32

[004938] - D/fal_port: read: offset = 0x000E4E0C size = 32

[004945] - D/fal_port: read: offset = 0x000E4E28 size = 32

[004952] - D/fal_port: read: offset = 0x000E4E44 size = 32

[004959] - D/fal_port: read: offset = 0x000E4E60 size = 32

[004966] - D/fal_port: read: offset = 0x000E4E7C size = 32

[004973] - D/fal_port: read: offset = 0x000E4E98 size = 32

[004979] - D/fal_port: read: offset = 0x000E4EB4 size = 32

[004986] - D/fal_port: read: offset = 0x000E4ED0 size = 32

armink commented 1 week ago

好想法呀,你可以来尝试试一下?

ckforce commented 1 week ago

昨天测试了一下,跑了一组固定的测试用例,有查找、写入、删除等操作,读取次数40w次左右的读取,用时20秒左右。 FlashDB配置: 100KB 分区,key大小从几十到几百字节不等,最大不超过1KB。

尝试在read(long offset, uint8_t *buf, size_t size) 里面加了512字节缓存,结果是用时差不多,没有明显改善,没有想通问题在哪里

no cache: flase_read time: 20012 ms

512 cache, <= 32 load cache:

cache hit: 364250/405314 = 89.87% time: 21311 ms

512 cache, <= 64 load cache:

cache hit: 339540/377003 = 90.06% time: 19571 ms