Umlaut handling

[nilshoerrmann]

Describe the bug

We've got a collection of streets with umlauts in their names. One of them is named "Görgesstraße", the id is orte/goergesstraße. If I search for "Gö", the street ist displayed in the results. If I type any more letters, the correct match vanishes from the results. Even if I write the full name or if I try the id the seaked street will not be displayed anymore.

This happens with other umlauts as well. When searching for "Höfenstraße" it display results for "Hafenstraße" most prominently, ignoring the direct match.

Context

This is a huge dataset: there are 2000 sibling pages with street names which all contain additional pages (excluded from the search though).

Plugin options

    'search' => [
        'provider' => 'sqlite',
        'fields' => [
            'pages' => ['title'],
            'files' => false,
            'users' => ['email', 'name']
        ],
        'templates' => [
            'pages' => function ($model) {
                return $model->intendedTemplate()->name() !== 'location';
            }
        ]
    ]

The location template is used by child pages of the streets.
The index has been generated using the provided shell script.

Expected behavior

The street should not be excluded when typing additional letters after the umlaut.

Additional context

Specifications

• Plugin version: latest master as submodule
• Kirby version: 3.4.1
• Browser: latest Firefox

[nilshoerrmann]

The database seems to have an encoding issue:

[distantnative]

I could not reproduce the encoding issue in the database itself (might depend on your viewer?). But I can definitely reproduce the issue. Unfortunately it seems not just to be one bug.

What I have found so far:

• Using str_word_count wasn't a good idea. It's splitting at a bunch of characters where it shouldn't.
• Seems I can replace it with preg_match_all('/[\pL\pN\pPd]+/u', $query, $tokens) though.
• There still seem to be issues with inserting fields with Umlaute etc. - they appear empty in my tests
• Running manual SQL queries with Umlaute also leads to 0 results.

Sorry that I can't yet report to have found the perfect solution. But at least I feel like progressing in baby steps here.

[distantnative]

Ha, I think I found it. I should think more of @afbora's mb_ bug fixes in the core. Seems the same thing bit me here.

[distantnative]

@nilshoerrmann Could you try if https://github.com/distantnative/search-for-kirby/commit/9378f4aab56f49eaa5ecf1296eed6c43f73d32ee fixes it for you? At least in conjunction with https://github.com/distantnative/search-for-kirby/commit/a609599a7124cdc90b8397ec00eecb6ff9259777 it seems to work for me. So basically the current state of the develop branch.

[distantnative]

Have done some more testing with the develop branch and it seems to be fine now. Please let me know if you're still experiencing issues. Closing this for now.

[nilshoerrmann]

Thanks! I'll check this out next week.

[hannesherold]

In our project it seems to work just fine (it's a catalogue of a library with mainly german books).