objectbox / objectbox-java

Android Database - first and fast, lightweight on-device vector database
https://objectbox.io
Apache License 2.0
4.42k stars 304 forks source link

Make "between" query conditions use index data #1186

Open eric-nextsense opened 3 months ago

eric-nextsense commented 3 months ago

Build info

Steps to reproduce

I have a database that contains say 100,000 rows. I try to query 20,000 rows using an id + a between query on an indexed long. It takes ~1.5 seconds in average, sometime up to 5-10s.

Data is frequently written to the database, about 40 times per second. Is that an issue? About 2400 rows are added per second in total between these writes.

The relativeSamplingTimestamp that is used as index has 2 values for the same timestamp in all cases (2 different data streams with same time). Like:

1 1 2 2 3 3 ...

Expected behavior

I would expect it to be under a 100ms?

Actual behavior

Takes 1-10 seconds. Average 1.5s.

Code

Code ```java Schema: @Id public long id public ToOne localSession; @Convert(converter = Converters.SerializableConverter.class, dbType = byte[].class) private HashMap samples; @Convert(converter = Converters.InstantConverter.class, dbType = Long.class) private Instant receptionTimestamp; @Nullable @Index private Integer relativeSamplingTimestamp; @Convert(converter = Converters.InstantConverter.class, dbType = Long.class) @Index @Nullable private Instant absoluteSamplingTimestamp; Query: samplesRelativeTimestampBetweenQuery = sampleBox.query().equal(Sample_.localSessionId, 0) .between(Sample_.relativeSamplingTimestamp, 0, 0).build(); Which i then call with parameters like: samplesRelativeTimestampBetweenQuery.setParameter( Sample_.localSessionId, localSessionId) .setParameters(Sample_.relativeSamplingTimestamp, startTimestamp, endTimestamp) .find()) ```
greenrobot commented 3 months ago

Seems like between doesn't use the index. In your case "time series" would make sense I guess. It's a special feature that's better than a secondary index and supports between.

github-actions[bot] commented 2 months ago

Without additional information, we are unfortunately not sure how to resolve this issue. Therefore this issue has been automatically closed. Feel free to comment with additional details and we can re-open this issue.

eric-nextsense commented 2 months ago

You are planning to add this to standard ObjectBox? If so, do you have an approximate ETA?

greenrobot commented 2 months ago

We have a lot of things that we want to do, so there's no ETA on specific features. We track feature requests so we can at least estimate the community interest. (And if you have some business agreement with us, it much more likely things are getting prioritized.)