readingdb is a time-series database designed for efficiency and speed.

Time series data is any data which is a series of (time, sequence, value) points. ReadingDB buckets and compresses your data using a delta encoding and zlib, and then writes this into a bdb installation with a bdb index. It uses the bdb transaction manager for write-ahead logging so that the volume will not become corrupted.

To use it, first follow the instructions in INSTALL to build the database and the python interface module. The key objects in readingdb are "streams", which have an integer id. The readingdb python module lets you talk to the server process; traffic between the client and server is encoded using google protocol buffer definitions found in c6/pbuf.

A simple python script for inserting data would be:

-- import readingdb as rdb

specify default host/port

rdb.db_setup('localhost', 4242)

create a connection

db = rdb.db_open('localhost')

add data. the tuples are (timestamp, seqno, value)

rdb.db_add(db, 1, [(x, 0, x) for x in xrange(0, 100)])

read back the data we just wrote using the existing connection

the args are streamid, start_timestamp, end_timestamp

print rdb.db_query(1, 0, 100, conn=db)

close

rdb.db_close(db)

read back the data again using a connection pool this time. You can

specify a list of streamids to range-query multiple streams at once.

rdb.db_query([1], 0, 100)

ReadingDB supports efficient range querying and interation using the db_query, db_prev, and db_next operations; you can delete data with db_del.

As you can see, db_query can re-use an existing connection if desired. As of April 20, 2012, the result of querying the TSDB is returned as a list of numpy matrices. This is because this is a very memory-efficient data structure, and creating numpy matrices using the c API is a lot more efficient than later doing it in python.

If no connection is specified for db_query/db_prev/db_next, new connections will be opened to the host/port specified with db_setup. The most efficient way of downloading data from a large number of streams is to specify a list of streamids, which allows the client library to conduct multiple parallel downloads of the data. Using this approach we have observed readingdb easily saturating a 100Mb NIC.

Sketches

As of 0.7.0 (October, 2014), readingdb supports computing sketches over the data in order to allow for more interactive performance on very large data sets. As of this release, it supports precomputing min, max, mean, and count at 5-minute, 15-minute, and 1-hour resolutions.

By default, the behavor is the same as before. If sketches are enabled on reading-server (by starting it with the -r flag), it will stream a log of regions of streams which have new data to disk; these are the places where the sketches should be updated.

The sketches may be updated using the new reading-sketch program; when run, it reads in the log, computes new sketches for the regions of time which have changed, and exits. This should be called periodically (e.g., by cron); the debian package installes a disabled crontab for this purpose into /etc/cron.d/readingdb.

Clients may request sketches through the new sketch kwarg to rb_query. For instance:

--

load hourly minimia from stream number 2, over all time.

min_data = rdb.db_query([2], 0, int(time.time()), sketch=("min", 3600))[0]

The value should be a tuple of (sketchname, resolution (in seconds)). The server will raise an exception on the client if an invalid sketch is passed.

Note: older servers may instead return the underlying data. older client libraries will raise an exception since sketch is not a valid kwarg there.

Dependencies

For reading-server libdb4.8, libdb4.8-dev (berkeley database) libprotobuf, libprotobuf-dev (google protocol buffers) libprotoc6, libprotoc6-dev (c bindings for protobufs) zlib, zlib-dev (for compression) gcc, make, automake

For python bindings python, python-dev, python-numpy (python deps) libdb4.8-dev swig (interface generator)