When using punctuation characters in queries, they need to be properly quoted before sent to sqlite. This is now done through keeping a manually curated list of punctuation characters which still leads to some characters not handled properly causing SQL errors.
The solution is:
Before the query is sent to sqlite, it is tokenized depending on the tokenizer defined in the schema meta data.
All tokens are quoted - unless a quote is already given spanning multiple tokens
Punctuation characters are automatically removed from the tokens - exceptions apply, e.g. for prefix and wildcard queries depending on context.
When using punctuation characters in queries, they need to be properly quoted before sent to sqlite. This is now done through keeping a manually curated list of punctuation characters which still leads to some characters not handled properly causing SQL errors.
The solution is: