ostafen / clover

A lightweight document-oriented NoSQL database written in pure Golang.
MIT License
633 stars 54 forks source link

Question: Searching for all documents within time range #113

Closed jcsco closed 1 year ago

jcsco commented 1 year ago

I'm searching for a field in all documents within a given time range. Currently, my approach is to use MatchFunc to search in all fields of the document and added a where clause before it to specify the time range. I realize I could check the timestamp range within the MatchFunc but that wouldn't benefit from the indexing. Any suggestions to improve query performance?

My dataset is one million documents and the time field is an integer and is indexed.

    db.CreateIndex(eventCollection, "eventTimestamp")
    db.CreateIndex(eventCollection, "_id")

I'm trying the following query but it still takes 30 secs.

    log.Printf("Searching for %s within timestamp %d and %d", value, start, end)

    docs, err := db.FindAll(
        clover.NewQuery(eventCollection).Where(clover.Field("eventTimestamp").GtEq(start).
            And(clover.Field("eventTimestamp").LtEq(end))).MatchFunc(func(doc *clover.Document) bool {
            for _, v := range doc.ToMap() {

                val, ok := v.(string)
                if !ok {
                    continue
                }

                if val == value {
                    return true
                }
            }
            return false
        }).Sort(clover.SortOption{Field: "eventTimestamp", Direction: 1}))
    if err != nil {
        log.Printf("FindAll Error: %s\n", err.Error())
        return nil, errors.New("error finding documents")
    }
jcsco commented 1 year ago

Nevermind. I flipped the order of MatchFunc and Where clause and it is quick now. Maybe this behavior needs to be documented.

    docs, err := db.FindAll(
        clover.NewQuery(eventCollection).MatchFunc(func(doc *clover.Document) bool {
            for _, v := range doc.ToMap() {

                val, ok := v.(string)
                if !ok {
                    continue
                }

                if val == value {
                    return true
                }
            }
            return false
        }).Where(clover.Field("eventTimestamp").GtEq(start).And(clover.Field("eventTimestamp").LtEq(end))).Sort(clover.SortOption{Field: "eventTimestamp", Direction: 1}))