Handle malformed Strings gracefully

arianneorpilla commented 1 year ago

Steps to Reproduce

Make an indexed attribute for a class (not case sensitive) and attempt to find 𠮟咤.

The following fatal crash will occur.

I/flutter ( 5273): RangeError (index): Invalid value: Only valid value is 0: 1
I/flutter ( 5273): #0      _TwoByteString.codeUnitAt (dart:core-patch/string_patch.dart:1355)
I/flutter ( 5273): #1      encodeString (package:isar/src/native/encode_string.dart:23)
I/flutter ( 5273): #2      CString.toCString (package:isar/src/native/encode_string.dart:50)
I/flutter ( 5273): #3      _strToNative (package:isar/src/native/index_key.dart:201)
I/flutter ( 5273): #4      _addKeyValue (package:isar/src/native/index_key.dart:99)
I/flutter ( 5273): #5      buildIndexKey (package:isar/src/native/index_key.dart:28)
I/flutter ( 5273): #6      _buildLowerIndexBound (package:isar/src/native/query_build.dart:139)
I/flutter ( 5273): #7      _addIndexWhereClause (package:isar/src/native/query_build.dart:199)
I/flutter ( 5273): #8      buildNativeQuery (package:isar/src/native/query_build.dart:35)
I/flutter ( 5273): #9      IsarCollectionImpl.buildQuery (package:isar/src/native/isar_collection_impl.dart:598)
I/flutter ( 5273): #10     QueryBuilderInternal.build (package:isar/src/query_builder.dart:281)
I/flutter ( 5273): #11     QueryExecute.build (package:isar/src/query_builder_extensions.dart:194)
I/flutter ( 5273): #12     QueryExecute.findAllSync (package:isar/src/query_builder_extensions.dart:206)
I/flutter ( 5273): #13     prepareSearchResultsJapaneseLanguage (package:yuuna/src/language/implementations/japanese_language.dart:429)
I/flutter ( 5273): <asynchronous suspension>

Resources that may be related: https://core.trac.wordpress.org/ticket/51097 https://japanese.stackexchange.com/questions/6872/are-the-4-byte-utf-8-kanji-rare-enough-that-i-can-ignore-them

Code sample

termMatch = database.dictionaryHeadings
              .where()
              .termEqualTo('𠮟咤')
              .findAllSync();

Details

Platform: Google Pixel 6 Pro (also reproduced on an Android emulator)
Flutter version: 3.7.8
Isar version: 3.0.6-dev.0

[X] I searched for similar issues already
[X] I filled the details section with the exact device model and version
[X] I am able to provide a reproducible example

arianneorpilla commented 1 year ago

Extra logs:

signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
Abort message: 'called `Result::unwrap()` on an `Err` value: IllegalArg { message: "The provided String is not valid." }'
    rax 0000000000000000  rbx 0000000000001951  rcx 00007ac9e10765cf  rdx 0000000000000006
    r8  00007ac9e10fd740  r9  00007ac9e10fd740  r10 00007ac6d327a200  r11 0000000000000207
    r12 00007ac6d6b29b10  r13 0000000000000001  r14 00007ac6d327a1f8  r15 0000000000001c3c
    rdi 0000000000001951  rsi 0000000000001c3c
    rbp 00007ac787ce4918  rsp 00007ac6d327a1f0  rip 00007ac9e10765cf
backtrace:
      #00 pc 000000000005e5cf  /apex/com.android.runtime/lib64/bionic/libc.so (abort+191) (BuildId: 5db8d317d3741b337ef046540bbdd0f7)
      #01 pc 00000000000c4de6  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #02 pc 00000000000c4d0a  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #03 pc 00000000000ed4f7  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #04 pc 00000000000ef373  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #05 pc 00000000000eeeb1  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #06 pc 00000000000eee25  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #07 pc 00000000000eede1  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #08 pc 00000000000701e1  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #09 pc 0000000000073221  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk
      #10 pc 0000000000067c5f  /data/app/~~xDxaJjnf-m0NEsUPDDeVKA==/app.lrorpilla.yuuna-9VKD9vvw15IKGUWOauKOQQ==/base.apk (isar_key_add_string+207)
      #11 pc 00000000000063ea  [anon:dart-code]

arianneorpilla commented 1 year ago

Looks like this error occurs when there is a lone surrogate in the text. You can match and sanitize lone surrogates with the following expression:

RegExp loneSurrogate = RegExp(
      '[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?:[^\uD800-\uDBFF]|^)[\uDC00-\uDFFF]',
);

I think it would be good if the package checked for malformed strings and prevented fatal crashes. I'll leave it up to what the maintainer thinks -- otherwise, this issue should be good to close for me.

simc commented 1 year ago

Thanks for reporting this. I'll leave it open to fix in a future version 🙂

isar / isar