how do you handle full text search?

[jpmonettas]

Hi! first of all thanks for the library!

I'm thinking about using it for a project and was trying to figure out how to implement things like search all todos with titles like "%t%".

I know datascript doesn't support full text search but you can filter db with any predicate, and that can do the trick.

Looking at the filters implemented in posh I don't see a filter by any predicate, is there any other way to implement this functionality?

Thanks!

[metasoarous]

Posh filters are an entirely different beast. They are more or less a performance mechanism that prevent queries from having to be rerun when a tx hits the db which didn't change any attributes showing up in the query. This has nothing to do with filter clauses that you might find in a query. For that, you just need to use a function in your query relations, just like you would without Posh. In other words, if you get this to work in DataScript it should work in Posh.

[seantempesta]

Hey @jpmonettas. I just saw your message now. I had a similar need and I solved it by using Lunr.js. I ended up using Posh's filters to listen to specific attributes that I cared about indexing and would then update the Lunr index. Which worked really well.

https://lunrjs.com

Here's some of the code I wrote in case you want to go down the same path:

;; Pure Functions
;;

(defn search [idx query]
  (js->clj (ocall @idx "search" query) :keywordize-keys true))

(defn to-json [idx]
  (debug "lunr: to-json!")
  (trace idx)
  (ocall @idx "toJSON"))

;; Side Effecting functions
;;

(defn datoms->lunr-index!
  "Updates a lunr index
  from datascript datoms [eid attr val]
  to lunr docs {[eid (str attr)] val}"
  [datoms Index]
  (let [eid->attr->val (reduce (fn [accum [eid attr val]]
                                 (assoc-in accum [eid (str attr)] val)) {} datoms)]
    (trace "datoms->index! Updating index" eid->attr->val)
    (doseq [[k v] eid->attr->val]
      (ocall Index "update" (clj->js (merge {:id k} v))))))

(defn load! [Index idx lunr-json]
  (debug "lunr: load!")
  (trace Index idx lunr-json)
  (let [restored-idx (.load Index lunr-json)]
    (do (debug "lunr: Loaded lunr index")
        (trace restored-idx)
        (reset! idx restored-idx))))

(defn init! [lunr-config]
  (let [{:keys [schema]} lunr-config
        Index (oget rr/Lunr "Index")
        idx (atom (Index.))]

    (info "Lunr: starting")
    (trace "lunr-init" lunr-config Index idx)

    ;; configure the fields to be indexed
    (doseq [[field boost] schema]
      (ocall @idx "field" (str field) (clj->js boost)))

    {:idx                 idx
     :Index               Index
     :search              #(search idx %)
     :datoms->lunr-index! #(datoms->lunr-index! Index %)
     :load!               #(load! Index idx %)
     :to-json             #(to-json idx)}))

[jpmonettas]

@seantempesta hey thanks a lot for that. It can be useful in the project I'm working on.

[seantempesta]

@jpmonettas You're welcome! I was super impressed with the capabilities of Lunr.js. You can create different indexes for different parts of your app too. This really helped us make a fulltext search of a notes field much more powerful.