Hyperluminal-DB is a high-performance, memory-mapped object database for Common Lisp.
It is based on Hyperluminal-MEM, a serialization/deserialization library, and STMX, a high-performance implementation of transactional memory.
Hyperluminal-DB is designed and optimized for the following scenarios:
mmap()
maximum size (on Linux 3.x, the limit is about 128 terabytes).usable on 32 bit systems, either retaining 64 bit file format (with some performance loss), or using native 32 bit file format - fast, but has the following limitations:
In any case, on 32 bit systems the dataset size is limited by mmap()
maximum size, usually around 1 gigabyte
Released version 0.5.1.
This is the last release packaging together Hyperluminal-MEM and Hyperluminal-DB in a single GPLv3 library.
Released version 0.5.0.
The serialization library is tested, documented and ready to use. It may still contain some rough edges and small bugs.
The serialization library works and is in BETA status.
The memory-mapped database (built on top of the serialization library) is in the early-implementation stage, not yet ready for general use.
Hyperluminal-DB is currently tested on the following Common Lisp implementations:
Note: on ABCL, memory buffers are implemented using java.nio.ByteBuffer instead of CFFI-SYS raw memory pointers due to currently limited compatibility between ABCL and CFFI/OSICAT libraries. Memory-mapped files are supported, and internally use java.nio.channels.FileChannel.map() instead of OSICAT-POSIX (mmap)
Note: CMUCL needs to be started with the option -fpu x87 to run Hyperluminal-DB reliably,
see STMX documentation
for details.
Hyperluminal-DB requires several libraries to work: LOG4CL, CLOSER-MOP, TRIVIAL-GARBAGE, BORDEAUX-THREADS, CFFI, OSICAT, STMX and HYPERLUMINAL-MEM. The last five, while reasonably portable, exploit features well beyond ANSI Common Lisp and their support for the various Common Lisp implementations varies widely.
For this reason no general guarantees can be given: Hyperluminal-DB may or may not work on other, untested Common Lisp implementations.
Hyperluminal-DB is available from GitHub. The simplest way to obtain it is to first install Quicklisp then download Hyperluminal-MEM and Hyperluminal-DB into your Quicklisp local-projects folder. Open a shell and run the commands:
$ cd ~/quicklisp/local-projects
$ git clone git://github.com/cosmos72/hyperluminal-mem.git
$ git clone git://github.com/cosmos72/hyperluminal-db.gitthen open a REPL and run:
CL-USER> (ql:quickload "hyperluminal-db")
;; lots of output...
CL-USER> (use-package :hlmem)
CL-USER> (use-package :hldb)If all goes well, this will load Hyperluminal-DB and its dependencies:
log4clcloser-moptrivial-garbagebordeaux-threadscffiosicatstmxhyperluminal-memIn case you get errors:
check that Quicklisp is installed correctly, for example by executing at REPL:
CL-USER> (ql:quickload "closer-mop")check that you downloaded Hyperluminal-DB creating an hyperluminal-db/ folder inside
your Quicklisp local-projects folder, usually ~/quicklisp/local-projects
After loading Hyperluminal-DB for the first time, it is recommended to run STMX, Hyperluminal-MEM and Hyperluminal-DB test suites to check that everything works as expected. From the REPL, run:
CL-USER> (asdf:test-system :stmx)
;; lots of output...
Did 7133 checks.
Pass: 7133 (100%)
Skip: 0 ( 0%)
Fail: 0 ( 0%)
CL-USER> (asdf:test-system :hyperluminal-mem)
;; lots of output...
Did 4092 checks.
Pass: 4092 (100%)
Skip: 0 ( 0%)
Fail: 0 ( 0%)
CL-USER> (asdf:test-system :hyperluminal-db)
;; lots of output...
Did 2505 checks.
Pass: 2505 (100%)
Skip: 0 ( 0%)
Fail: 0 ( 0%)Note: all the (asdf:test-system ...) intentionally work only after
the corresponding library has been loaded successfully with (ql:quickload ...).
All the test suites should report zero Skip and zero Fail; the number of Pass may vary. You are welcome to report any failure you get while running the test suites, please include in the report:
See "Contacts, help, discussion" below for the preferred method to send the report.
Hyperluminal-DB is loosely inspired by some techniques used by manardb but it is a completely separate and independent project.
It is based on STMX, a high-performance hybrid transactional memory library, and Hyperluminal-MEM, a high-performance serialization library, both from the same author.
Hyperluminal-DB uses a (supposedly) clever trick in order to overcome Intel claims that hardware memory transactions (specifically, Intel TSX) cannot perform input/output.
The result is that Hyperluminal-DB is able to perform transactional input/output while running hardware memory transactions - an apparent paradox - in an extremely specific but significant case: reading and writing memory-mapped files.
This allows reaching extremely high transaction speeds: the only hard limit is the hardware - an Intel Core i7 4770 peaks at 400 millions transactions per second when all the 4 cores and hyperthreading are exploited.
Quite clearly, the speed also strongly depends on the amount (and type) of data read and written during each transaction.
TODO
Hyperluminal-DB file format and ABI is inherited from Hyperluminal-MEM, which performs autodetection to match Lisp idea of CFFI-SYS pointers:
It is possible to override such autodetection by adding an appropriate entry
in the global variable *FEATURES* before compiling and loading Hyperluminal-MEM,
see Hyperluminal-MEM/README
for details.
As of February 2015, Hyperluminal-DB is being written by Massimiliano Ghilardi and it is considered by the author to be in ALPHA status.
As long as the traffic is low enough, GitHub Issues can be used to report test suite failures, bugs, suggestions, general discussion etc.
If the traffic becomes high, more appropriate discussion channels will be set-up.
The author will also try to answer support requests, but gives no guarantees.
Hyperluminal-DB is released under the terms of the GNU General Public License v3.0, known as the GPLv3.