simjanos-dev / LinguaCafe

LinguaCafe is a self-hosted software that helps language learners read foreign languages.
https://simjanos-dev.github.io/LinguaCafeHome/
GNU General Public License v3.0
896 stars 32 forks source link

JMDict japanese dictionary too slow search. #67

Closed simjanos-dev closed 10 months ago

simjanos-dev commented 10 months ago

JMDict search uses a separate dictionary search function, that has 4 queries for different kind of matches, and it is too slow.

app/Http/Controllers/DictionaryController.php

// exact word matches
$search = VocabularyJmdict::select('id')->whereRelation('words', 'word', 'like', $term)->get()->toArray();
foreach ($search as $result) {
    if (!in_array($result, $ids, true)) {
        array_push($ids, $result);
    }
}

// exact reading matches
$search = VocabularyJmdict::select('id')->whereRelation('readings', 'reading', 'like', $term)->get()->toArray();
foreach ($search as $result) {
    if (!in_array($result, $ids, true)) {
        array_push($ids, $result);
    }
}

// partial word matches, max 10
$search = VocabularyJmdict::select('id')->whereRelation('words', 'word', 'like', $term . '%')->get()->toArray();
foreach ($search as $result) {
    if (!in_array($result, $ids, true) && count($ids) < 10) {
        array_push($ids, $result);
    }
}

// partial reading matches, max 10
$search = VocabularyJmdict::select('id')->whereRelation('readings', 'reading', 'like', $term . '%')->get()->toArray();
foreach ($search as $result) {
    if (!in_array($result, $ids, true) && count($ids) < 10) {
        array_push($ids, $result);
    }
}
simjanos-dev commented 10 months ago

I've made some tests, and it seems to be working reasonably well. All search results came back below 0.5s. However I've found some problems with DeepL, I'll create a separate issue for it.