Data corruption hazard (make cache non-volatile)

[GoogleCodeExporter]

The current cache implementation introduces volatility into the system. While a 
filesystem backed by 
s3backer may be journaled there's still a high risk for data loss. For example, 
if there is a system failure with dirty blocks in the cache there is a 
likelihood that the filesystem journal will get out of sync with 
what's actually on S3. The journal can't help you in this case because as far 
as it's concerned the data has 
already been written [to s3backers cache]. The issue is compounded when the 
blocks are uploaded out of 
order. 

The easiest solution is probably to make the cache non-volatile so that the 
system can later recover.

Original issue reported on code.google.com by jonsview on 14 Aug 2009 at 1:49

[GoogleCodeExporter]

Thanks.. a couple of comments.

- s3backer will attempt to write blocks to S3 in the same order they are 
written.
However, if you have parallel writes enabled (on by default) then of course the 
order
in which they complete writing at Amazon is indeterminate. You can fix this 
problem
by disabling parallel writes.

- s3backer has a synchronous write mode that will not return from a write 
attempt
until the block is successfully written. Use this mode to avoid the "unwritten 
write"
problem.

- What do you mean by "make the cache non-volatile" (if not some combination of 
the
above)?

Original comment by archie.c...@gmail.com on 14 Aug 2009 at 1:56

[GoogleCodeExporter]

For example, lets say that the cache has 10 dirty blocks and the computer loses 
power. The cache is stored in memory 
(last I knew) so it would be lost and the filesystem would be left in a corrupt 
state with no possible recovery. By making 
the cache non-volatile I mean making it resistant to this kind of failure, so 
even after a power loss or s3backer crash the 
filesystem can be mounted and recovered.

> You can fix this problem by disabling parallel writes.
The unordered writes to S3 isn't really a problem and shouldn't be avoided. It 
just makes some possible solutions more 
difficult such as writing a journaling system just for s3backer blocks, but 
that's probably overkill.

Disclaimer: I'm not really knowledgeable with the inner workings of 
filesystems. I was just tinkering around with s3backer 
and this issue came to mind.

Original comment by jonsview on 14 Aug 2009 at 2:10

[GoogleCodeExporter]

I missed the second comment:

> s3backer has a synchronous write mode that will not return from a write 
attempt until the block is successfully 
written. Use this mode to avoid the "unwritten write" problem.
That's good to know! I haven't tested it yet, but I would assume that it would 
hamper performance especially on 
certain types of high-latency internet connections.

Ideally the caching layer would be resilient across restarts and crashes. I 
think I may have actually seen this on one 
of the wiki comments.

Original comment by jonsview on 14 Aug 2009 at 2:36

[GoogleCodeExporter]

It sounds like you are talking about the "persistent cache" idea, where the 
cached
blocks are stored not in memory but on the local disk. This would avoid the 
data loss
on power failure problem (assuming a successful subsequent restart).

Original comment by archie.c...@gmail.com on 14 Aug 2009 at 3:03

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Persistent cache has been implemented in version 1.3.0. However, only clean 
blocks
are cached across restarts.

Let me know if/how this addresses your questions.

Original comment by archie.c...@gmail.com on 1 Oct 2009 at 7:57

[GoogleCodeExporter]

Note also that the `--blockCacheMaxDirty` flag can be used to limit the amount 
of
inconsistency between the local cache and the S3 backing store. This flag was 
added
in r398 (after release 3.1.0).

Original comment by archie.c...@gmail.com on 13 Oct 2009 at 10:18

[GoogleCodeExporter]

Fixed using block cache and desired configuration to limit dirtiness.

Original comment by archie.c...@gmail.com on 11 May 2013 at 6:14

• Changed state: Fixed