phaistos-networks / TANK

A very high performance distributed log service
Apache License 2.0
938 stars 70 forks source link

Topic creation improvements #70

Open markpapadakis opened 5 years ago

markpapadakis commented 5 years ago

Now, when not running in cluster-aware mode, the directory name for the topic will be prepended with '.', and only when all partitions for that topic have been initialized (in that .directory), that .directory will be atomically renamed to the correct name(i.e the topic name).

This is important because it's possible that partition creation may fail(e.g system fault, FS fault, etc), and we don't want to leave a mess when that happens. That is to say, creating a topic should be an atomic operation.

markpapadakis commented 5 years ago

Turns out that deleting directories on startup may lead to data loss - because TANK may be misconfigured to use the wrong base directory. Will now generate a warning and will eventually accept a new command-line argument enabling GC on startup.