search

npgall / cqengine

Ultra-fast SQL-like queries on Java collections
Apache License 2.0
1.72k stars 251 forks source link

readme

Build Status Maven Central

CQEngine - Collection Query Engine

CQEngine – Collection Query Engine – is a high-performance Java collection which can be searched with SQL-like queries, with extremely low latency.

Supports on-heap persistence, off-heap persistence, disk persistence, and supports MVCC transaction isolation.

Interesting reviews of CQEngine:

The Limits of Iteration

The classic way to retrieve objects matching some criteria from a collection, is to iterate through the collection and apply some tests to each object. If the object matches the criteria, then it is added to a result set. This is repeated for every object in the collection.

Conventional iteration is hugely inefficient, with time complexity O(n t). It can be optimized, but requires statistical knowledge of the makeup of the collection. Read more: The Limits of Iteration

Benchmark Sneak Peek

Even with optimizations applied to convention iteration, CQEngine can outperform conventional iteration by wide margins. Here is a graph for a test comparing CQEngine latency with iteration for a range-type query:

quantized-navigable-index-carid-between.png

See the Benchmark wiki page for details of this test, and other tests with various types of query.

CQEngine Overview

CQEngine solves the scalability and latency problems of iteration by making it possible to build indexes on the fields of the objects stored in a collection, and applying algorithms based on the rules of set theory to reduce the time complexity of accessing them.

Indexing and Query Plan Optimization

Several implementations of CQEngine's IndexedCollection are provided, supporting various concurrency and transaction isolation levels:

For more details see TransactionIsolation.

Complete Example

In CQEngine applications mostly interact with IndexedCollection, which is an implementation of java.util.Set, and it provides two additional methods:

Here is a complete example of how to build a collection, add indexes and perform queries. It does not discuss attributes, which are discussed below.

STEP 1: Create a new indexed collection

IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>();

STEP 2: Add some indexes to the collection

cars.addIndex(NavigableIndex.onAttribute(Car.CAR_ID));
cars.addIndex(ReversedRadixTreeIndex.onAttribute(Car.NAME));
cars.addIndex(SuffixTreeIndex.onAttribute(Car.DESCRIPTION));
cars.addIndex(HashIndex.onAttribute(Car.FEATURES));

STEP 3: Add some objects to the collection

cars.add(new Car(1, "ford focus", "great condition, low mileage", Arrays.asList("spare tyre", "sunroof")));
cars.add(new Car(2, "ford taurus", "dirty and unreliable, flat tyre", Arrays.asList("spare tyre", "radio")));
cars.add(new Car(3, "honda civic", "has a flat tyre and high mileage", Arrays.asList("radio")));

STEP 4: Run some queries

Note: add import statement to your class: import static com.googlecode.cqengine.query.QueryFactory.*

Complete source code for these examples can be found here.

String-based queries: SQL and CQN dialects

As an alternative to programmatic queries, CQEngine also has support for running string-based queries on the collection, in either SQL or CQN (CQEngine Native) format.

Example of running an SQL query on a collection (full source here):

public static void main(String[] args) {
    SQLParser<Car> parser = SQLParser.forPojoWithAttributes(Car.class, createAttributes(Car.class));
    IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>();
    cars.addAll(CarFactory.createCollectionOfCars(10));

    ResultSet<Car> results = parser.retrieve(cars, "SELECT * FROM cars WHERE (" +
                                    "(manufacturer = 'Ford' OR manufacturer = 'Honda') " +
                                    "AND price <= 5000.0 " +
                                    "AND color NOT IN ('GREEN', 'WHITE')) " +
                                    "ORDER BY manufacturer DESC, price ASC");

    results.forEach(System.out::println); // Prints: Honda Accord, Ford Fusion, Ford Focus
}

Example of running a CQN query on a collection (full source here):

public static void main(String[] args) {
    CQNParser<Car> parser = CQNParser.forPojoWithAttributes(Car.class, createAttributes(Car.class));
    IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>();
    cars.addAll(CarFactory.createCollectionOfCars(10));

    ResultSet<Car> results = parser.retrieve(cars,
                                    "and(" +
                                        "or(equal(\"manufacturer\", \"Ford\"), equal(\"manufacturer\", \"Honda\")), " +
                                        "lessThanOrEqualTo(\"price\", 5000.0), " +
                                        "not(in(\"color\", GREEN, WHITE))" +
                                    ")");

    results.forEach(System.out::println); // Prints: Ford Focus, Ford Fusion, Honda Accord
}

Feature Matrix for Included Indexes

Legend for the feature matrix

Abbreviation Meaning Example
EQ Equality equal(Car.DOORS, 4)
IN Equality, multiple values in(Car.DOORS, 3, 4, 5)
LT Less Than (numerical range / Comparable) lessThan(Car.PRICE, 5000.0)
GT Greater Than (numerical range / Comparable) greaterThan(Car.PRICE, 2000.0)
BT Between (numerical range / Comparable) between(Car.PRICE, 2000.0, 5000.0)
SW String Starts With startsWith(Car.NAME, "For")
EW String Ends With endsWith(Car.NAME, "ord")
SC String Contains contains(Car.NAME, "or")
CI String Is Contained In isContainedIn(Car.NAME, "I am shopping for a Ford Focus car")
RX String Matches Regular Expression matchesRegex(Car.MODEL, "Ford.*")
HS Has (aka IS NOT NULL) has(Car.DESCRIPTION) / not(has(Car.DESCRIPTION))
SQ Standing Query Can the index accelerate a query (as opposed to an attribute) to provide constant time complexity for any simple query, complex query, or fragment
QZ Quantization Does the index accept a quantizer to control granularity
LP LongestPrefix longestPrefix(Car.NAME, "Ford")

Note: CQEngine also supports complex queries via and, or, not, and combinations thereof, across all indexes.

Index Feature Matrix

Index Type EQ IN LT GT BT SW EW SC CI HS RX SQ QZ LP
Hash ✓ ✓ ✓
Unique ✓ ✓
Compound ✓ ✓ ✓
Navigable ✓ ✓ ✓ ✓ ✓ ✓
PartialNavigable ✓ ✓ ✓ ✓ ✓ ✓
RadixTree ✓ ✓ ✓
ReversedRadixTree ✓ ✓ ✓
InvertedRadixTree ✓ ✓ ✓ ✓
SuffixTree ✓ ✓ ✓ ✓
StandingQuery ✓
Fallback ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
OffHeap ✓ ✓ ✓ ✓ ✓ ✓ ✓[1]
PartialOffHeap ✓ ✓ ✓ ✓ ✓ ✓ ✓
Disk ✓ ✓ ✓ ✓ ✓ ✓ ✓[1]
PartialDisk ✓ ✓ ✓ ✓ ✓ ✓ ✓

[1] See: forStandingQuery()

The Benchmark page contains examples of how to add these indexes to a collection, and measures their impact on latency.

Attributes

Read Fields

CQEngine needs to access fields inside objects, so that it can build indexes on fields, and retrieve the value of a certain field from any given object.

CQEngine does not use reflection to do this; instead it uses attributes, which is a more powerful concept. An attribute is an accessor object which can read the value of a certain field in a POJO.

Here's how to define an attribute for a Car object (a POJO), which reads the Car.carId field:

public static final Attribute<Car, Integer> CAR_ID = new SimpleAttribute<Car, Integer>("carId") {
    public Integer getValue(Car car, QueryOptions queryOptions) { return car.carId; }
};

...or alternatively, from a lambda expression or method reference:

public static final Attribute<Car, Integer> Car_ID = attribute("carId", Car::getCarId);

(For some caveats on using lambdas, please read LambdaAttributes)

Usually attributes are defined as anonymous static final objects like this. Supplying the "carId" string parameter to the constructor is actually optional, but it is recommended as it will appear in query toStrings.

Since this attribute reads a field from a Car object, the usual place to put the attribute is inside the Car class - and this makes queries more readable. However it could really be defined in any class, such as in a CarAttributes class or similar. The example above is for a SimpleAttribute, which is designed for fields containing only one value.

CQEngine also supports MultiValueAttribute which can read the values of fields which themselves are collections. And so it supports building indexes on objects based on things like keywords associated with those objects.

Here's how to define a MultiValueAttribute for a Car object which reads the values from Car.features where that field is a List<String>:

public static final Attribute<Car, String> FEATURES = new MultiValueAttribute<Car, String>("features") {
    public Iterable<String> getValues(Car car, QueryOptions queryOptions) { return car.features; }
};

...or alternatively, from a lambda expression or method reference:

public static final Attribute<Car, String> FEATURES = attribute(String.class, "features", Car::getFeatures);

Null values

Note if your data contains null values, you should use SimpleNullableAttribute or MultiValueNullableAttribute instead.

In particular, note that SimpleAttribute and MultiValueAttribute do not perform any null checking on your data, and so if your data inadvertently contains null values, you may get obscure NullPointerExceptions. This is because null checking does not come for free. Attributes are accessed heavily, and the non-nullable versions of these attributes are designed to minimize latency by skipping explicit null checks. They defer to the JVM to do the null checking implicitly.

As a rule of thumb, if you get a NullPointerException, it's probably because you used the wrong type of attribute. The problem will usually go away if you switch your code to use a nullable attribute instead. If you don't know if your data may contain null values, just use the nullable attributes. They contain the logic to check for and handle null values automatically.

The nullable attributes also allow CQEngine to work with object inheritance, where some objects in the collection have certain optional fields (e.g. in subclasses) while others might not.

Creating queries dynamically

Dynamic queries can be composed at runtime by instantiating and combining Query objects directly; see this package and this package. For advanced cases, it is also possible to define attributes at runtime, using ReflectiveAttribute or AttributeBytecodeGenerator.

Generate attributes automatically

CQEngine also provides several ways to generate attributes automatically.

Note these are an alternative to using ReflectiveAttribute, which was discussed above. Whereas ReflectiveAttribute is a special type of attribute which reads values at runtime using reflection, AttributeSourceGenerator and AttributeBytecodeGenerator generate code for attributes which is compiled and so does not use reflection at runtime, which can be more efficient.

See AutoGenerateAttributes for more details.

Attributes as Functions

It can be noted that attributes are only required to return a value given an object. Although most will do so, there is no requirement that an attribute must provide a value by reading a field in the object. As such attributes can be virtual, implemented as functions.

Calculated Attributes

An attribute can calculate an appropriate value for an object, based on a function applied to data contained in other fields or from external data sources.

Here's how to define a calculated (or virtual) attribute by applying a function over the Car's other fields:

public static final Attribute<Car, Boolean> IS_DIRTY = new SimpleAttribute<Car, Boolean>("is_dirty") {
    public Boolean getValue(Car car, QueryOptions queryOptions) { return car.description.contains("dirty"); }
};

...or, the same thing using a lambda:

public static final Attribute<Car, Boolean> IS_DIRTY = attribute("is_dirty", car -> car.description.contains("dirty"));

A HashIndex could be built on the virtual attribute above, enabling fast retrievals of cars which are either dirty or not dirty, without needing to scan the collection.

Associations with other IndexedCollections or External Data Sources

Here is an example for a virtual attribute which associates with each Car a list of locations which can service it, from an external data source:

public static final Attribute<Car, String> SERVICE_LOCATIONS = new MultiValueAttribute<Car, String>() {
    public List<String> getValues(Car car, QueryOptions queryOptions) {
        return CarServiceManager.getServiceLocationsForCar(car);
    }
};

The attribute above would allow the IndexedCollection of cars to be searched for cars which have servicing options in a particular location.

The locations which service a car, could alternatively be retrieved from another IndexedCollection, of Garages, for example. Care should be taken if building indexes on virtual attributes however, if referenced data might change leaving obsolete information in indexes. A strategy to accommodate this is: if no index exists for a virtual attribute referenced in a query, and other attributes are also referenced in the query for which indexes exist, CQEngine will automatically reduce the candidate set of objects to the minimum using other indexes before querying the virtual attribute. In turn if virtual attributes perform retrievals from other IndexedCollections, then those collections could be indexed appropriately without a risk of stale data.

Joins

The examples above define attributes on a primary IndexedCollection which read data from secondary collections or external data sources.

It is also possible to perform SQL EXISTS-type queries and JOINs between IndexedCollections on the query side (as opposed to on the attribute side). See Joins for examples.

Persistence on-heap, off-heap, disk

CQEngine's IndexedCollections can be configured to store objects added to them on-heap (the default), or off-heap, or on disk.

On-heap

Store the collection on the Java heap:

IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>();

Off-heap

Store the collection in native memory, within the JVM process but outside the Java heap:

IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>(OffHeapPersistence.onPrimaryKey(Car.CAR_ID));

Note that the off-heap persistence will automatically create an index on the specified primary key attribute, so there is no need to add an index on that attribute later.

Disk

Store the collection in a temp file on disk (then see DiskPersistence.getFile()):

IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>(DiskPersistence.onPrimaryKey(Car.CAR_ID));

Or, store the collection in a particular file on disk:

IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>(DiskPersistence.onPrimaryKeyInFile(Car.CAR_ID, new File("cars.dat")));

Note that the disk persistence will automatically create an index on the specified primary key attribute, so there is no need to add an index on that attribute later.

Wrapping

Wrap any Java collection, in a CQEngine IndexedCollection without any copying of objects.

Collection<Car> collection = // obtain any Java collection

IndexedCollection<Car> indexedCollection = new ConcurrentIndexedCollection<Car>(
        WrappingPersistence.aroundCollection(collection)
);

Composite

CompositePersistence configures a combination of persistence types for use within the same collection. The collection itself will be persisted in the first persistence provided (the primary persistence), and the additional persistences provided will be used by off-heap or disk indexes added to the collection subsequently.

Store the collection on-heap, and also configure DiskPersistence for use by DiskIndexes added to the collection subsequently:

IndexedCollection<Car> cars = new ConcurrentIndexedCollection<Car>(CompositePersistence.of(
    OnHeapPersistence.onPrimaryKey(Car.CAR_ID),
    DiskPersistence.onPrimaryKeyInFile(Car.CAR_ID, new File("cars.dat"))
));

Index persistence

Indexes can similarly be stored on-heap, off-heap, or on disk. Each index requires a certain type of persistence. It is necessary to configure the collection in advance with an appropriate combination of persistences for use by whichever indexes are added.

It is possible to store the collection on-heap, but to store some indexes off-heap. Similarly it is possible to have a variety of index types on the same collection, each using a different type of persistence. On-heap persistence is by far the fastest, followed by off-heap persistence, and then by disk persistence.

If both the collection and all of its indexes are stored off-heap or on disk, then it is possible to have extremely large collections which don't use any heap memory or RAM at all.

CQEngine has been tested using off-heap persistence with collections of 10 million objects, and using disk persistence with collections of 100 million objects.

On-heap

Add an on-heap index on "manufacturer":

cars.addIndex(NavigableIndex.onAttribute(Car.MANUFACTURER));

Off-heap

Add an off-heap index on "manufacturer":

cars.addIndex(OffHeapIndex.onAttribute(Car.MANUFACTURER));

Disk

Add a disk index on "manufacturer":

cars.addIndex(DiskIndex.onAttribute(Car.MANUFACTURER));

Querying with persistence

When either the IndexedCollection, or one or more indexes are located off-heap or on disk, take care to close the ResultSet when finished reading. You can use a try-with-resources block to achieve this:

try (ResultSet<Car> results = cars.retrieve(equal(Car.MANUFACTURER, "Ford"))) {
    results.forEach(System.out::println);
}

Result Sets

CQEngine ResultSets provide the following methods:

Deduplicating Results

It is possible that a query would result in the same object being returned more than once.

For example if an object matches several attribute values specified in an or-type query, then the object will be returned multiple times, one time for each attribute matched. Intersections (and-type queries) and negations (not-type queries) do not produce duplicates.

By default, CQEngine does not perform de-duplication of results; however it can be instructed to do so, using various strategies such as Logical Elimination and Materialize. Read more: DeduplicationStrategies

Ordering Results

By default, CQEngine does not order results; it simply returns objects in the order it finds them in the collection or in indexes.

CQEngine can be instructed to order results via query options as follows.

Order by price descending

ResultSet<Car> results = cars.retrieve(query, queryOptions(orderBy(descending(Car.PRICE))));

Order by price descending, then number of doors ascending

ResultSet<Car> results = cars.retrieve(query, queryOptions(orderBy(descending(Car.PRICE), ascending(Car.DOORS))));

Note that ordering results as above uses the default materialize ordering strategy. This is relatively expensive, dependent on the number of objects matching the query, and can cause latency in accessing the first object. It requires all results to be materialized into a sorted set up-front before iteration can begin.

Index-accelerated ordering

CQEngine also has support to use an index to accelerate, or eliminate, the overhead of ordering results. This strategy reduces the latency to access the first object in the sorted results, at the expense of adding more total overhead if the entire ResultSet was iterated. Read more: OrderingStrategies

Merge Strategies

Merge strategies are the algorithms CQEngine uses to evaluate queries which have multiple branches.

By default CQEngine will use strategies which should suit most applications, however these strategies can be overridden to tune performance. Read more: MergeStrategies

Index Quantization, Granularity, and tuning index size

Quantization involves converting fine-grained or continuous values, to discrete or coarse-grained values. A Quantizer is a function which takes fine-grained values as input, and maps those values to coarse-grained counterparts as its output, by discarding some precision.

Quantization can be a useful tool to tune the size of indexes, trading a reduction in index size, for increases in CPU overhead and vice-versa. Read more: Quantization and included Quantizers

Grouping and Aggregation (GROUP BY, SUM...)

CQEngine has been designed with support for grouping and aggregation in mind, but note that this is not built into the CQEngine library itself, because CQEngine is designed to integrate with Java 8+ Streams. This allows CQEngine results to be grouped, aggregated, and transformed in flexible ways using lambda expressions.

CQEngine ResultSet can be converted into a Java 8 Stream by calling ResultSet.stream().

Note that Streams are evaluated via filtering and they do not avail of CQEngine indexes. So for best performance, as much of the overall query as possible should be encapsulated in the CQEngine query, as opposed to in lambda expressions in the stream. This combination would dramatically outperform a stream and lambda expression alone, which simply filtered the collection.

Here's how to transform a ResultSet into a Stream, to compute the distinct set of Colors of cars which match a CQEngine query.

public static void main(String[] args) {
    IndexedCollection<Car> cars = new ConcurrentIndexedCollection<>();
    cars.addAll(CarFactory.createCollectionOfCars(10));
    cars.addIndex(NavigableIndex.onAttribute(Car.MANUFACTURER));

    Set<Car.Color> distinctColorsOfFordCars = cars.retrieve(equal(Car.MANUFACTURER, "Ford"))
            .stream()
            .map(Car::getColor)
            .collect(Collectors.toSet());

    System.out.println(distinctColorsOfFordCars); // prints: [GREEN, RED]
}

Accessing Index Metadata and Statistics from MetadataEngine

The MetadataEngine, is a high-level API which can retrieve metatadata and statistics from indexes which have been added to the collection.

It provides access to the following:

For more information, see JavaDocs for: MetadataEngine, AttributeMetadata, SortedAttributeMetadata

Using CQEngine with Hibernate / JPA / ORM Frameworks

CQEngine has seamless integration with JPA/ORM frameworks such as Hibernate or EclipseLink.

Simply put, CQEngine can build indexes on, and query, any type of Java collection or arbitrary data source. ORM frameworks return entity objects loaded from database tables in Java collections, therefore CQEngine can act as a very fast in-memory query engine on top of such data.

Usage in Maven and Non-Maven Projects

CQEngine is in Maven Central, and can be added to a Maven project as follows:

<dependency>
    <groupId>com.googlecode.cqengine</groupId>
    <artifactId>cqengine</artifactId>
    <version>x.x.x</version>
</dependency>

See ReleaseNotes for the latest version number.

For non-Maven projects, a version built with maven-shade-plugin is also provided, which contains CQEngine and all of its own dependencies packaged in a single jar file (ending "-all"). It can be downloaded from Maven central as "-all.jar" here.

Using CQEngine in Scala, Kotlin, or other JVM languages

CQEngine should generally be compatible with other JVM languages besides Java too, however it can be necessary to apply a few tricks to make it work. See OtherJVMLanguages.md for some tips.

Related Projects

Project Status

Report any bugs/feature requests in the Issues tab. For support please use the Discussion Forum, not direct email to the developers.

Many thanks to JetBrains for supporting CQEngine with free IntelliJ licenses!