Closed sujit-baniya closed 1 year ago
I cannot reproduce this error, please provide input data.
@micpst Code i tried:
package main
import (
"encoding/json"
"fmt"
"os"
"github.com/micpst/minisearch/pkg/store"
"github.com/micpst/minisearch/pkg/tokenizer"
)
type ICD struct {
Code string `json:"code"`
Desc string `json:"desc"`
}
func readData() (icds []ICD) {
jsonData, err := os.ReadFile("icd10_codes.json")
if err != nil {
fmt.Printf("failed to read json file, error: %v", err)
return
}
if err := json.Unmarshal(jsonData, &icds); err != nil {
fmt.Printf("failed to unmarshal json file, error: %v", err)
return
}
return
}
func main() {
data := readData()
db := store.New[ICD](&store.Config{
DefaultLanguage: tokenizer.ENGLISH,
TokenizerConfig: &tokenizer.Config{
EnableStemming: true,
EnableStopWords: true,
},
})
p := store.InsertBatchParams[ICD]{
Documents: data,
BatchSize: 100,
}
errs := db.InsertBatch(&p)
if len(errs) > 0 {
panic(errs)
}
}
You have an error in your store schema definition, you need to add index
tag to the fields you want to query.
type ICD struct {
Code string `json:"code" index:"code"`
Desc string `json:"desc" index:"desc"`
}
I still cannot reproduce the error, even without this modification db.InsertBatch
works correctly.
@micpst Weird. Maybe it's because of map data. I've modified this repo to support for map data. Maybe you add support for map?
If you have modified the lib on your own, I can't really help. Currently there is no need to add map support, you can easily nest properties using the current implementation.
I've been working on small library on the top of this package. At the moment, I've millions of data to be indexed.
Error occurs in
Tokenizer
Can you please suggest some solution?