STEC EnhanceIO SSD Caching Software
25th December, 2012WHAT IS ENHANCEIO?
EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document).
EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device.
The source volume to SSD mapping is a set-associative mapping based on the source volume sector number with a default set size (aka associativity) of 512 blocks and a default block size of 4 KB. Partial cache blocks are not used. The default value of 4 KB is chosen because it is the common I/O block size of most storage systems. With these default values, each cache set is 2 MB (512 * 4 KB). Therefore, a 400 GB SSD will have a little less than 200,000 cache sets because a little space is used for storing the meta data on the SSD.
EnhanceIO supports three caching modes: read-only, write-through, and write-back and three cache replacement policies: random, FIFO, and LRU.
Read-only caching mode causes EnhanceIO to direct write IO requests only to HDD. Read IO requests are issued to HDD and the data read from HDD is stored on SSD. Subsequent Read requests for the same blocks are carried out from SSD, thus reducing their latency by a substantial amount.
In Write-through mode - reads are handled similar to Read-only mode. Write-through mode causes EnhanceIO to write application data to both HDD and SSD. Subsequent reads of the same data benefit because they can be served from SSD.
Write-back improves write latency by writing application requested data only to SSD. This data, referred to as dirty data, is copied later to HDD asynchronously. Reads are handled similar to Read-only and Write-through modes.
WHAT HAS ENHANCEIO CHANGED TO FLASHCACHE?
2.1. A new write-back engine
The write-back engine in EnhanceiO has been designed from scratch.
Several optimizations have been done. IO completion guarantees have
been improved. We have defined limits to let a user control the amount
of dirty data in a cache. Clean-up of dirty data is stopped by default
under a high load; this can be overridden if required. A user can
control the extent to which a single cache set can be filled with dirty
data. A background thread cleans-up dirty data at regular intervals.
Clean-up is also done at regular intevals by identifying cache sets
which have been written least recently.2.2. Transparent cache
EnhanceIO does not use device mapper. This enables creation and
deletion of caches while a source volume is being used. It's possible
to either create or delete cache while a partition is mounted.
EnhanceIO also supports creation of a cache for a device which contains
partitions. With this feature it's possible to create a cache without
worrying about having to create several SSD partitions and many
separate caches.2.3. Large I/O Support
Unlike Flashcache, EnhanceIO does not cause source volume I/O requests
to be split into cache block size pieces. For the typical SSD cache
block size of 4 KB, this means that a write I/O request size of, say,
64 KB to the source volume is not split into 16 individual requests of
4 KB each. This is a performance improvement over Flashcache. IO
codepaths have been substantially modified for this improvement.2.4. Small Memory Footprint
Through a special compression algorithm, the meta data RAM usage has
been reduced to only 4 bytes for each SSD cache block (versus 16 bytes
in Flashcache). Since the most typical SSD cache block size is 4 KB,
this means that RAM usage is 0.1% (1/1000) of SSD capacity.
For example, for a 400 GB SSD, EnhanceIO will need only 400 MB to keep
all meta data in RAM.
For an SSD cache block size of 8 KB, RAM usage is 0.05% (1/2000) of SSD
capacity.
The compression algorithm needs at least 32,768 cache sets
(i.e., 16 bits to encode the set number). If the SSD capacity is small
and there are not at least 32,768 cache sets, EnhanceIO uses 8 bytes of
RAM for each SSD cache block. In this case, RAM usage is 0.2% (2/1000)
of SSD capacity for a cache block size of 4K.2.5. Loadable Replacement Policies
Since the SSD cache size is typically 10%-20% of the source volume
size, the set-associative nature of EnhanceIO necessitates cache
block replacement.
The main EnhanceIO kernel module that implements the caching engine
uses a random (actually, almost like round-robin) replacement policy
that does not require any additional RAM and has the least CPU
overhead. However, there are two additional kernel modules that
implement FIFO and LRU replacement policies. FIFO is the default cache
replacement policy because it uses less RAM than LRU. The FIFO and LRU
kernel modules are independent of each other and do not have to be
loaded if they are not needed.
Since the replacement policy modules do not consume much RAM when not
used, both modules are typically loaded after the main caching engine
is loaded. RAM is used only after a cache has been instantiated to use
either the FIFO or the LRU replacement policy.
Please note that the RAM used for replacement policies is in addition
to the RAM used for meta data (mentioned in Section 2.1). The table
below shows how much RAM each cache replacement policy uses:
POLICY RAM USAGE
------ ---------
Random 0
FIFO 4 bytes per cache set
LRU 4 bytes per cache set + 4 bytes per cache block2.6. Optimal Alignment of Data Blocks on SSD
EnhanceIO writes all meta data and data blocks on 4K-aligned blocks
on the SSD. This minimizes write amplification and flash wear.
It also improves performance.2.7. Improved device failure handling
Failure of an SSD device in read-only and write-through modes is
handled gracefully by allowing I/O to continue to/from the
source volume. An application may notice a drop in performance but it
will not receive any I/O errors.
Failure of an SSD device in write-back mode obviously results in the
loss of dirty blocks in the cache. To guard against this data loss, two
SSD devices can be mirrored via RAID 1.
EnhanceIO identifies device failures based on error codes. Depending on
whether the failure is likely to be intermittent or permanent, it takes
the best suited action.2.8. Coding optimizations
Several coding optizations have been done to reduce CPU usage. These
include removing queues which are not required for write-through and
read-only cache modes, splitting of a single large spinlock, and more.
Most of the code paths in flashcache have been substantially
restructured.2.9 Sequential I/O bypass
EnhanceIO has removed the bypass of sequential IO available in flashcache.
The sequential detection logic has a limited use case, espescially in a
reasonably multithreaded scenario.3.1. Cache creation, deletion and editing properties
eio_cli utility is used for creating and deleting caches and editing
their properties. Manpage for this utility eio_cli(8) provides more
information.3.2. Making a cache configuration persistent It's essential that a cache be resumed before any applications or a filesystem use the source volume during a bootup. If a cache is enabled after a source volume is written to, stale data may be present in the cache. It may cause data corruption. The document Persistent.txt describes how to enable a cache during bootup using udev scripts.
In case an SSD does not come up during a bootup, it's ok to allow read
and write access to HDD only in the case of a Write-through or a
read-only cache. A cache should be created again when SSD becomes
available. If a previous cache configuration is resumed, it may cause
stale data to be read.3.3. Using a Write-back cache It's absolutely necessary to make a Write-back cache configuration persistent. This is required particularly in the case of an OS crash or a power failure. A Write-back cache may contain dirty blocks which haven't been written to HDD yet. Reading the source volume without enabling the cache will cause incorrect data to be read.
In case an SSD does not come up during a bootup, access to HDD should
stopped. It should be enabled only after SSD comes-up and a cache is
enabled.
Write-back cache needs to perform clean operation in order to flush the
dirty data to the source device(HDD). The clean can be either trigerred
by the user or automatically initiated, based on preconfigured
thresholds. These thresholds are described below. They can be set using
sysctl calls.
a) Dirty high threshold (%) : The upper limit on percentage of dirty
blocks in the entire cache.
b) Dirty low threshold (%) : The lower limit on percentage of dirty
blocks in the entire cache.
c) Dirty set high threshold (%) : The upper limit on percentage of dirty
blocks in a set.
d) Dirty set low threshold (%) : The lower limit on percentage of dirty
blocks in a set.
e) Automatic clean-up threshold : An automatic clean-up of the cache
will occur only if the number of outstanding I/O requests from the
HDD is below the threshold.
f) Time based clean-up interval (minutes) : This option allows you to
specify an interval between each clean-up process.
Clean is trigerred when one of the upper thresholds or time based clean
threshold is met and stops when all the lower thresholds are met. ACKNOWLEDGEMENTS
STEC acknowledges Facebook and in particular Mohan Srinivasan for the design, development, and release of Flashcache as an open source project.
Flashcache, in turn, is based on DM-Cache by Ming Zhao.