search

HWCloudEngine / storage-gateway

storage gateway to adapt cloud storage to replicate/snapshot/backup to other different cloud storage.
7 stars 10 forks source link

how to enhance cache initialization efficiency? #3

Closed yinweiishere closed 8 years ago

yinweiishere commented 8 years ago

When SG client gets initialized, we need first have journal replayer to load un-replayed journals to client's BCache and JCache; Until then, SG client could start serve request. The efficiency of the initialization is critical.

The technique is to load journal files on parallel, while parse journal entries in each journal file sequentially. Plus, we need maintain an update sequence per volume among log entries from different journal files. It's feasible since we maintain ordering inside journal files and among journal files.

It's possible to maintain the update seq with two num, first as the jfile seq and the second as the inside jfile seq. So the seq number belonging to each jfile won't overlap.

yinweiishere commented 8 years ago

this optimization has been done in current cache initialization