search

pbek / QOwnNotes

QOwnNotes is a plain-text file notepad and todo-list manager with Markdown support and Nextcloud / ownCloud integration.
https://www.qownnotes.org/
GNU General Public License v2.0
4.67k stars 413 forks source link

[I] Note list panel issue when searching cyrillic words in different case #1590

Open ysaroka opened 4 years ago

ysaroka commented 4 years ago

Expected behaviour

Cyrillic words must be searched in the same mode as Latin, in case-insensitive mode.

Actual behaviour

Cyrillic words are searched only in case-sensitive mode, while Latin words are searched case-insensitively (this is correct behavior)

I did not find the ability to configure the search engine in any way. This is probably a issue.

Steps to reproduce

Video (Tested words: "установка", "усТанОвка"): https://www.youtube.com/watch?v=jP_tyzJ1pkc

Information about the application, settings and environment

Expand QOwnNotes Debug Information =========================== ## General Info **Current Date**: `Чт янв 16 06:03:01 2020` **Version**: `20.1.9` **Build date**: `Jan 15 2020` **Build number**: `5275` **Platform**: `windows` **Operating System**: `Windows 10 (10.0)` **Build architecture**: `i386` **Current architecture**: `x86_64` **Release**: `AppVeyor` **Qt Version (build)**: `5.13.2` **Qt Version (runtime)**: `5.13.2` **Portable mode**: `yes` **Settings path / key**: `D:/apps/qownnotes/Data/PBE/QOwnNotes.ini` **Application database path**: `D:\apps\qownnotes\Data\QOwnNotes.sqlite` **Application arguments**: `D:\apps\qownnotes\QOwnNotes.exe`, `--portable` **Qt Debug**: `0` **Locale (system)**: `ru_RU` **Locale (interface)**: `en` **Primary screen resolution**: `1366x768` **Icon theme**: `breeze-qownnotes` **Notes in current note folder**: `2` **Calendar items**: `0` **Enabled scripts**: `0` ## Server Info **serverUrl**: *empty* **appIsValid**: `no` **notesPathExists**: *empty* **connectionErrorMessage**: *empty* ## Spellchecking **Enabled**: `false` **Selected language**: *empty* **Language codes**: *empty* **Language names**: *empty* **Application dictionaries path**: `D:\apps\qownnotes\Data\dicts` ## Note folders **currentNoteFolderId**: `1` ### Note folder `по умолчанию` **id**: `1` **isCurrent**: `yes` **activeTagId**: `0` **localPath**: `D:\apps\qownnotes\Data\..\..\..\Dropbox\docs\!notes\qownnotes-notes` **remotePath**: `Notes` **cloudConnectionId**: `1` **isShowSubfolders**: `yes` **isUseGit**: `no` **activeNoteSubFolder name**: `Temp` **database file**: `D:\apps\qownnotes\Data\..\..\..\Dropbox\docs\!notes\qownnotes-notes\notes.sqlite` ## Cloud connections ### Cloud connection `По умолчанию` **id**: `1` **isCurrent**: `yes` **serverUrl**: *empty* **username**: *empty* ## Enabled scripts ## Settings **AboutDialog/geometry** (QByteArray): `` **ActiveNoteHistoryItem** (NoteHistoryItem): `` **Debug/fakeOldVersionNumber** (bool): `false` **Debug/fileLogging** (bool): `false` **DistractionFreeMode/editorWidthMode** (QString): `1` **Editor/CurrentSchemaKey** (QString): `EditorColorSchema-6033d61b-cb96-46d5-a3a8-20d5172017eb` **Editor/autoBracketClosing** (bool): `true` **Editor/autoBracketRemoval** (bool): `true` **Editor/centerCursor** (QString): `false` **Editor/disableCursorBlinking** (bool): `false` **Editor/editorWidthInDFMOnly** (bool): `true` **Editor/highlightCurrentLine** (bool): `true` **Editor/indentSize** (int): `4` **Editor/useTabIndent** (bool): `false` **Editor/vimMode** (bool): `false` **FileDialog/LastPath** (QString): `D:/apps/qownnotes/!settings` **FileDialog/LastPath-ExternalEditor** (QString): `D:/apps/SublimeTextPortable` **FileDialog/LastPath-SettingsExport** (QString): `D:/apps/qownnotes/!settings` **IssueAssistantDialog/geometry** (QByteArray): `` **LastUpdateCheck** (QDateTime): `2020-01-16T05:44:45.912` **LinkDialog/geometry** (QByteArray): `` **LocalTrashDialog/geometry** (QByteArray): `` **LogWidget/criticalLog** (bool): `true` **LogWidget/debugLog** (bool): `true` **LogWidget/fatalLog** (bool): `true` **LogWidget/infoLog** (bool): `true` **LogWidget/scriptingLog** (bool): `true` **LogWidget/statusLog** (bool): `true` **LogWidget/warningLog** (bool): `true` **MainWindow/geometry** (QByteArray): `` **MainWindow/mainToolBar.iconSize** (int): `18` **MainWindow/menuBarGeometry** (QByteArray): `` **MainWindow/noteSubFolderTreeWidgetExpandState-1** (QStringList): `dsv, Programming, Other, Programming/PHP` **MainWindow/noteTextEdit.code.font** (QString): `Consolas,10,-1,5,50,0,0,0,0,0,Regular` **MainWindow/noteTextEdit.font** (QString): `Consolas,10,-1,5,50,0,0,0,0,0,Regular` **MainWindow/noteTextView.code.font** (QString): `Courier New,9,-1,2,50,0,0,0,0,0` **MainWindow/noteTextView.font** (QString): `MS Shell Dlg 2,8.25,-1,5,50,0,0,0,0,0` **MainWindow/noteTextView.ignoreCodeFontSize** (bool): `true` **MainWindow/noteTextView.rtl** (bool): `false` **MainWindow/noteTextView.underline** (bool): `true` **MainWindow/noteTextView.useEditorStyles** (bool): `true` **MainWindow/noteTextView.useInternalExportStyling** (bool): `true` **MainWindow/showNotesFromAllNoteSubFolders** (bool): `false` **NoteDialog/geometry** (QByteArray): `` **NoteDiffDialog/geometry** (QByteArray): `` **NoteHistory-1** (QVariantList): `` **NoteHistoryCurrentIndex-1** (int): `34` **PiwikClientId** (QString): `` **SearchEngineId** (int): `2` **SettingsDialog/geometry** (QByteArray): `` **SettingsDialog/mainSplitterState** (QByteArray): `` **SettingsExport/platform** (QString): `windows` **ShareDialog/geometry** (QByteArray): `` **Shortcuts/MainWindow-actionReplace_in_current_note** (QKeySequence): `Ctrl+H` **ShowSystemTray** (bool): `false` **StartHidden** (bool): `false` **TagAddDialog/geometry** (QByteArray): `` **UpdateDialog/geometry** (QByteArray): `` **WelcomeDialog/geometry** (QByteArray): `` **acceptAllExternalModifications** (bool): `false` **allowDifferentNoteFileName** (bool): `false` **allowNoteEditing** (bool): `true` **allowOnlyOneAppInstance** (bool): `true` **appMetrics/disableAppHeartbeat** (bool): `false` **appMetrics/disableTracking** (bool): `true` **appMetrics/notificationShown** (QString): `true` **automaticNoteFolderDatabaseClosing** (bool): `false` **checkSpelling** (bool): `false` **closeTodoListAfterSave** (bool): `false` **cryptoKey** (QString): `` **currentNoteFolderId** (QString): `1` **currentWorkspace** (QString): `ae47cf02-04be-42d5-8ffd-f0434f2b0215` **cursorWidth** (int): `1` **customNoteFileExtensionList** (QStringList): *empty* **darkMode** (bool): `false` **darkModeColors** (bool): `false` **darkModeIconTheme** (bool): `false` **darkModeTrayIcon** (bool): `false` **defaultNoteFileExtension** (QString): `md` **demoNotesCreated** (QString): `true` **disableAutomaticUpdateDialog** (bool): `true` **disableSavedSearchesAutoCompletion** (bool): `true` **dockWasInitializedOnce** (QString): `true` **enableNoteTree** (bool): `false` **enableSocketServer** (bool): `false` **externalEditorPath** (QString): *empty* **fullyHighlightedBlockquotes** (bool): `true` **gitCommitInterval** (int): `30` **gitExecutablePath** (QString): *empty* **gitLogCommand** (QString): *empty* **guiFirstRunInit** (bool): `true` **ignoreAllExternalModifications** (bool): `false` **ignoreAllExternalNoteFolderChanges** (bool): `false` **ignoreNoteSubFolders** (QString): `^\.` **imageScaleDown** (bool): `false` **imageScaleDownMaximumHeight** (int): `1024` **imageScaleDownMaximumWidth** (int): `1024` **initialLayoutIdentifier** (QString): `minimal` **insertTimeFormat** (QString): *empty* **interfaceFontSize** (int): `11` **interfaceLanguage** (QString): `en` **internalIconTheme** (bool): `false` **itemHeight** (int): `15` **legacyLinking** (bool): `false` **localTrash/autoCleanupDays** (int): `30` **localTrash/autoCleanupEnabled** (bool): `true` **localTrash/supportEnabled** (bool): `true` **localTrashSplitterSizes** (QByteArray): `` **markdownHighlightingEnabled** (bool): `true` **navigationPanelHideSearch** (bool): `false` **networking/ignoreSSLErrors** (bool): `true` **networking/proxyType** (int): `2` **newNoteAskHeadline** (bool): `false` **noteEditIsCentralWidget** (bool): `false` **noteListPreview** (bool): `false` **noteSaveIntervalTime** (int): `60` **noteSubfoldersPanelDisplayAsFullTree** (bool): `true` **noteSubfoldersPanelHideSearch** (bool): `false` **noteSubfoldersPanelOrder** (int): `0` **noteSubfoldersPanelShowFullPath** (bool): `false` **noteSubfoldersPanelShowNotesRecursively** (bool): `true` **noteSubfoldersPanelShowRootFolderName** (bool): `true` **noteSubfoldersPanelSort** (int): `0` **notesPanelOrder** (int): `0` **notesPanelSort** (int): `1` **notesPath** (QString): `../../../Dropbox/docs/!notes/qownnotes-notes` **notifyAllExternalModifications** (bool): `false` **overrideInterfaceFontSize** (bool): `false` **ownCloud/supportEnabled** (bool): `false` **ownCloud/todoCalendarBackend** (int): `3` **ownCloud/todoCalendarCalDAVPassword** (QString): `` **ownCloud/todoCalendarCalDAVServerUrl** (QString): *empty* **ownCloud/todoCalendarCalDAVUsername** (QString): *empty* **ownCloud/todoCalendarCloudConnectionId** (int): `1` **ownCloud/todoCalendarDisplayNameList** (QStringList): *empty* **ownCloud/todoCalendarEnabledList** (QStringList): *empty* **ownCloud/todoCalendarEnabledUrlList** (QStringList): *empty* **ownCloud/todoCalendarUrlList** (QStringList): *empty* **ownCloudInfo/appIsValid** (bool): `false` **ownCloudInfo/connectionErrorMessage** (QString): *empty* **ownCloudInfo/notesPathExistsText** (QString): *empty* **ownCloudInfo/serverVersion** (QString): *empty* **previousWorkspace** (QString): `ae47cf02-04be-42d5-8ffd-f0434f2b0215` **recentNoteFolders** (QString): `D:/!/test` **restoreCursorPosition** (bool): `true` **savedSearches/noteFolder-1** (QStringList): `lo, Note 2020-01-15T23.23.48, Note 2020-01-15T23.23.44, Note 2020-01-15T23.23.40, Note 2020-01-15T23.22.47, Note 2020-01-15T23.22.46, Note 2020-01-15T23.22.34, Note 2020-01-15T22.49.53, Note 2020-01-15T22.49.49, Note 2020-01-15T22.49.45, Note 2020-01-15T22.49.31, * Li, Note 2020-01-15T21.54.12` **showMatches** (bool): `true` **showMenuBar** (bool): `true` **showStatusBar** (bool): `false` **systemIconTheme** (bool): `false` **tagWasAddedToNote** (QString): `true` **taggingShowNotesRecursively** (bool): `false` **tagsPanelHideSearch** (bool): `false` **tagsPanelOrder** (int): `0` **tagsPanelSort** (int): `0` **todoCalendarSupport** (bool): `false` **toolbar/1/items** (QStringList): `action_Back_in_note_history, action_Forward_in_note_history, , actionShow_local_trash, actionAllow_note_editing` **toolbar/1/name** (QString): `mainToolBar` **toolbar/1/title** (QString): `main toolbar` **toolbar/2/items** (QStringList): `actionFormat_text_bold, actionFormat_text_italic, actionStrike_out_text, actionInsert_code_block, actionInsert_block_quote` **toolbar/2/name** (QString): `formattingToolbar` **toolbar/2/title** (QString): `formatting toolbar` **toolbar/3/items** (QStringList): `actionInsert_Link_to_note, actionInsert_image, actionInsert_current_time` **toolbar/3/name** (QString): `insertingToolbar` **toolbar/3/title** (QString): `inserting toolbar` **toolbar/4/items** (QStringList): `action_Encrypt_note, actionEdit_encrypted_note, actionDecrypt_note` **toolbar/4/name** (QString): `encryptionToolbar` **toolbar/4/title** (QString): `encryption toolbar` **toolbar/5/items** (QStringList): `actionWorkspaceComboBox, actionStore_as_new_workspace, actionRemove_current_workspace, actionRename_current_workspace, actionSwitch_to_previous_workspace, actionUnlock_panels, , actionToggle_distraction_free_mode, action_Increase_note_text_size, action_Decrease_note_text_size, action_Reset_note_text_size` **toolbar/5/name** (QString): `windowToolbar` **toolbar/5/title** (QString): `window toolbar` **toolbar/6/items** (QStringList): `action_Quit` **toolbar/6/name** (QString): `quitToolbar` **toolbar/6/title** (QString): `quit toolbar` **toolbar/size** (int): `6` **useSoftWrapInNoteEditor** (QString): `true` **useUNIXNewline** (bool): `true` **webSocketServerService/bookmarksNoteName** (QString): `Bookmarks` **webSocketServerService/bookmarksTag** (QString): `bookmarks` **webSocketServerService/port** (int): `22222` **workspace-ae47cf02-04be-42d5-8ffd-f0434f2b0215/name** (QString): `John Kind` **workspace-ae47cf02-04be-42d5-8ffd-f0434f2b0215/noteSubFolderDockWidgetVisible** (QString): `true` **workspace-ae47cf02-04be-42d5-8ffd-f0434f2b0215/windowState** (QByteArray): `` **workspace-initial/name** (QString): `Минимальный` **workspace-initial/noteSubFolderDockWidgetVisible** (QString): `true` **workspace-initial/windowState** (QByteArray): `` **workspaces** (QStringList): `initial, ae47cf02-04be-42d5-8ffd-f0434f2b0215` ## System environment **ALLUSERSPROFILE**: `C:\ProgramData` **APPDATA**: `C:\Users\johnkind\AppData\Roaming` **CommonProgramFiles**: `C:\Program Files (x86)\Common Files` **CommonProgramFiles(x86)**: `C:\Program Files (x86)\Common Files` **CommonProgramW6432**: `C:\Program Files\Common Files` **COMPOSE_CONVERT_WINDOWS_PATHS**: `true` **COMPUTERNAME**: `COMPUTER` **ComSpec**: `C:\Windows\system32\cmd.exe` **DOCKER_CERT_PATH**: `D:\Docker Toolbox\machine\machines\default` **DOCKER_HOST**: `tcp://192.168.99.101:2376` **DOCKER_MACHINE_NAME**: `default` **DOCKER_TLS_VERIFY**: `1` **DOCKER_TOOLBOX_INSTALL_PATH**: `D:\Docker Toolbox` **DriverData**: `C:\Windows\System32\Drivers\DriverData` **HOMEDRIVE**: `C:` **HOMEPATH**: `\Users\johnkind` **LOCALAPPDATA**: `C:\Users\johnkind\AppData\Local` **LOGONSERVER**: `\\COMPUTER` **MAGICK_HOME**: `D:\apps\imagick\modules\coders\` **NO_PROXY**: `192.168.99.100,192.168.99.101` **NUMBER_OF_PROCESSORS**: `8` **NVM_HOME**: `C:\Users\johnkind\AppData\Roaming\nvm` **NVM_SYMLINK**: `C:\Program Files\nodejs` **OS**: `Windows_NT` **Path**: `C:\Ruby26-x64\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Users\johnkind\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\ProgramData\ComposerSetup\bin;C:\Program Files\TortoiseGit\bin;D:\apps\php\;D:\apps\imagick\;D:\Docker Toolbox;C:\Ruby26-x64\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Users\johnkind\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\ProgramData\ComposerSetup\bin;C:\Program Files\TortoiseGit\bin;C:\Users\johnkind\AppData\Local\Microsoft\WindowsApp;D:\apps\php;` **PATHEXT**: `.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.RB;.RBW` **PROCESSOR_ARCHITECTURE**: `x86` **PROCESSOR_ARCHITEW6432**: `AMD64` **PROCESSOR_IDENTIFIER**: `Intel64 Family 6 Model 158 Stepping 10, GenuineIntel` **PROCESSOR_LEVEL**: `6` **PROCESSOR_REVISION**: `9e0a` **ProgramData**: `C:\ProgramData` **ProgramFiles**: `C:\Program Files (x86)` **ProgramFiles(x86)**: `C:\Program Files (x86)` **ProgramW6432**: `C:\Program Files` **PSModulePath**: `C:\Program Files (x86)\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules;C:\Program Files (x86)\AutoIt3\AutoItX` **PUBLIC**: `C:\Users\Public` **SESSIONNAME**: `Console` **SystemDrive**: `C:` **SystemRoot**: `C:\Windows` **TEMP**: `C:\Users\johnkind\AppData\Local\Temp` **TMP**: `C:\Users\johnkind\AppData\Local\Temp` **USERDOMAIN**: `COMPUTER` **USERDOMAIN_ROAMINGPROFILE**: `COMPUTER` **USERNAME**: `johnkind` **USERPROFILE**: `C:\Users\johnkind` **VBOX_MSI_INSTALL_PATH**: `C:\Program Files\Oracle\VirtualBox\` **windir**: `C:\Windows` **QML_DISABLE_DISK_CACHE**: `true`
pbek commented 4 years ago

Thank you for reporting, I can reproduce your issue.

pbek commented 4 years ago

Searching is done directly in a in-memory sqlite database.

https://github.com/pbek/QOwnNotes/blob/a7d8e0ed33ee58888ad910b08df5b6e828f4293a/src/entities/note.cpp#L813-L876

Seems like that's a limitation there.

pbek commented 4 years ago

Haven't found a way around it yet.

pbek commented 4 years ago

As far as I found out sqlite only supports case insensitive search with ASCII characters.

ysaroka commented 4 years ago

As far as I found out sqlite only supports case insensitive search with ASCII characters.

Yes it is.

The SQLite API has a sqlite3_create_function(): https://www.sqlite.org/c3ref/create_function.html Probably, you can override or create a new the LOWER (or LIKE) method to search in UTF-8 case-insensitively: ... UTF8_LOWER(note_text) LIKE ? ... or ... note_text UTF8_LIKE ? ...

Found the same problem here: https://forum.lazarus.freepascal.org/index.php?topic=34259.0 This is Pascal, but in this case it does not matter.

ysaroka commented 4 years ago

Another solution is to store additionally "note_text_lowercase" and "file_name_lowercase" fields for the corresponding values in lower case and compare them with the query string reduced to lower case. But i think this is not a good solution)

pbek commented 4 years ago

The SQLite API has a sqlite3_create_function():

I played around with that a bit. The only way to create custom functions in Qt seems to be to depend on (and include) the sqlite source directly. I don't really want to introduce an external dependency to QOwnNotes.

pbek commented 4 years ago

sqlite3.c has 223k lines of code :laughing:

ysaroka commented 4 years ago

I don't really want to introduce an external dependency to QOwnNotes.

I agree, bad idea. Sorry, I thought that Qt SQLite driver has a native ability to define custom functions.

pbek commented 4 years ago

Sorry, I thought that Qt SQLite driver has a native ability to define custom functions.

It's all abstracted behind QSqlDatabase

macsunmood commented 2 years ago

I second that. Are there viable approaches to solving this problem?

pbek commented 2 years ago

Not that I found out any...

macsunmood commented 2 years ago

sqlite only supports case insensitive search

Here's a relatively fresh article "5 ways to implement case-insensitive search in SQLite with full Unicode support": https://dev.to/shallowdepth/5-ways-to-implement-case-insensitive-search-in-sqlite-with-full-unicode-support-53p2

could this help?

also this: https://sudonull.com/post/64260-Another-implementation-of-case-insensitive-search-for-Cyrillic-characters-in-SQLite

pbek commented 2 years ago

We would need to find something that works with Qt5...