Closed kirillt closed 1 year ago
.ark/index
must be sorted in order to avoid redundant content modifications.The index is sorted:
modified
timestamp. The oldest resources first, the newest go at the end. Maybe in future we'll implement chunked storage, so older resources would go to separate file which would have low probability of being updated, so less conflicts in older "generation" would be expected.
Solves #15 by writing known ResourceIndex into
.ark/index
file. Consecutive scans re-use this file for quicker index loading.At this moment, format of
.ark/index
file is plain-text and is the following:Only relative paths must be stored there in order to read the index on any machine with the folder checked out by any path. But paths are kept only as absolute paths in memory, using
CanonicalPathBuf
type.There was a bug in
update
function making this optimization ineffective. Somehow, timestamps stored and retrieved do differ even without any modification of the resource. The timestamps are in nanoseconds. In my experiments, ~ 700ns was the difference, but not 1ms what would mean timestamps are just truncated. It should be investigated better, but basically, any call toupdate
was resulting in removing everything and adding it again into index. Threshold of 1ms is introduced now. This would also help with potential resources being modified by some process with high frequency.ResourceKind mocks are removed. Kinds should not be stored in
.ark/index
file for quicker and simpler indexing. Most likely, kinds should have their own table or be stored together with extra metadata (.ark/meta
). Latter approach would mean we write computed values into user-data though. Probably, we should create dedicated storage for generated metadata and call it "properties storage" (.ark/prop
). That would make separation between cache and user-data explicit, values like kinds and pages amount in documents would go to.ark/prop
and values like titles and descriptions of web bookmarks would go to.ark/meta
.