Sorting the column by clicking on its header blanks the browser pane.

[Yang2Gui3zi]

I ran the first test twice on the 0.1alpha2 versions of advanced_browser.py and advanced_browser_custom_fields.py.

First I selected the deck "Developing Chinese 2nd ed". Then I clicked the Browse button. The Browser window opened. I depressed the [Enter] key to display all cards in the current deck. A single column, labelled "Hanzi" displayed for all cards, apparently correctly.

I right-clicked on the column header. The drop-down menu displayed. From this I selected first the choice "Fields". A sub-drop-down menu displayed, listing the fields in the note type and, if I remember correctly, fields in other note types. By the way, this sub-drop-down menu disappears almost instantly if the cursor is not moved onto one of the choices.

I selected the field "Note ID". The drop-down menus disappeared, and a new column appeared, with its header labelled "Note ID". The values of Note ID appeared to be correct for the cards in the current deck. I clicked on the Whole Collection selection filter in the lefthand pane, and it showed cards from all decks, including ones in which the Note ID field was blank. I returned to the view of the Current Deck.

Then I clicked the column header labelled Note ID, in order to sort the cards by their Note IDs in ascending order. The column header bar and all contents of cards within the browser pane disappeared! No horizontal scroll bar was visible. I switched the selection filter to Whole Collection. Here, too, no column header bar nor card contents displayed.

By closing the browser window and selecting a different deck, then clicking the Browser button and selecting Current Deck, I was able to see the cards of that different deck in the browser. However, even with a different deck, the Whole Collection filter selection would not display the column header bar nor any cards.

When I chose the deck "Developing Chinese 2nd ed" again, the browser pane would still not display its cards in the browser pane. Shutting down Anki and restarting it did not fix the display of the browser. To fix the browser display, I restored the collection from a file named backup-.apkg.

[hssm]

Even with the collection you sent me, I can't reproduce this. You might have to send it to me again since something might have changed.

The sub-menu disappearing could be a toolkit problem on OS X. It works fine on my Linux machine at least. I'm not sure what I can do about it besides not use sub-menus.

[Yang2Gui3zi]

Hi, Houssam,

Some things in the collection have definitely changed since I first sent the collection to you.  For one thing, I added some notes to the collection.  For another, to restore the collection after the browser contents disappeared, I clicked on a backup file called "backup-XX.apkg".  For a third thing, I changed the name of the user directory from "User 1" to "Lincoln 01".

Here is a hyperlink from which you can download my whole Anki directory:  https://www.dropbox.com/sh/vwt9e3jly67k7ju/S1piCm-OWv.  This directory includes the collection in the state it was before I tried a third test.  (Most of the directory's size is due to the add-on Chinese Support.)  If you have any trouble on your linux machine handling the directory which was produced under Mac, please let me know and I can export the collection, at least, as an *.apkg file for you.

Because you were not able to reproduce the disappearance of the browser contents, I decided to rerun the test before sending you my collection.  As you will see below, I was able to reproduce the disappearance of the browser contents for the deck "Developing Chinese 2nd ed".  However, I also saw that the disappearance does not happen to at least one of the other decks when I sort on the contents of a field added to the browser for that deck.  So I hypothesize that my deck has somehow been corrupted.

So here are the steps I went through in this third test:

1. I opened Anki and selected the deck "Developing Chinese 2nd ed". Then I clicked the Browse button. The Browser window opened. I depressed the [Enter] key to display all cards in the current deck. Two columns were displayed, labelled Card andHanzi, showing the contents of the cards in this deck.  These were the columns that had been displaying when I last closed the browser.
2. I clicked on the Fields… button.  The Fields window opened.  I navigated to highlight the line for the field Hanzi.  It showed that the radio button for "Sort by this field in browser" had been displayed.  I closed the Fields window.
3. Back in the browser pane, I positioned the cursor on the column header for the field Hanzi.  I clicked the mouse to sort by this column.  All the cards in the browser pane disappeared, and the title in the window's title bar changed to "Browser (0 cards shown, 0 selected)".
4. Nothing I could do would restore the browser pane to show any cards in the deck "Developing Chinese 2nd ed" or in the "Whole Collection".  Among the things I tried were closing Anki and reopening it.
5. However, I could still see, in the browser pane, the contents of the decks NPCR and "Pinyin Toolkit Sample Deck (Advanced Mode)".  I could sort the browser pane for the deck NPCR by clicking on the column header for the field Hanzi.  This was the same field name whose sorting had caused the browser contents to disappear in the deck "Developing Chinese 2nd ed", although the two decks had different note types.  In the case of the deck NPCR, I could add a column for the field Pinyinand sort by the contents of that column, too, without causing the contents of the browser pane to disappear.
6. So I conclude that the deck "Developing Chinese 2nd ed" has somehow become corrupt.  I do not see any evidence that there is a bug in the add-on advanced-browser.

I wonder how this deck got corrupted?

I exported the deck as a text file, with the fields separated by tabs.  I can import the text file into Excel, which will separate the fields into columns, so that I can look at them more closely and try to find out how it is corrupt.  Also, I believe I might be able to import the text file, after removing the corruption, into Anki to rebuild the deck. Thanks, Yang2Gui3zi Yáng Guǐzi 洋鬼子

[hssm]

It's unlikely that your deck is corrupt. For future reference, if you hold shift while Anki loads, it will start without add-ons. I expect cards will load normally in that case. I haven't looked at this yet, but it's almost certainly caused by this add-on.

[Yang2Gui3zi]

You are right. I held down [Shift] while loading Anki. Then the browser pane did show the cards in the deck "Developing Chinese 2nd ed" and Whole Collection. I have not yet investigated it any further.

I am not sure whether, in the clause, "... but it's almost certainly caused by this add-on", the add-on to which you referred was Chinese Support or advanced-browser.

I suppose I could find out which of the two (or both) causes the display problem by deleting one of them each time and then opening Anki with the other add-on installed. So that would tell me which add-on causes the display problem. But I would need a somewhat different procedure to discover which of the two add-ons triggers the problem when I try to sort the cards by clicking on a field's header column.

[aleksejrs]

I couldn't figure out how to add an arbitrary sorting function (for forecast), and anything I do that does not error out verbally just makes the browser blank when sorted by my column.

[hssm]

If you can send or link me to what you're trying, I can see why it might be failing. I know I still need to document how this is meant to work, but I need to sort out other things first.

If you call an external function in your sort and it throws an error, your entire query will return nothing, hence the empty browser. (It might be a good place to enclose in a try/except and print any errors). That's what I think is happening here as well, but I still can't reproduce it with what I have. I'm hoping it's not some discrepancy in SQLite versions.

[aleksejrs]

hssm helped me find two typo-grade problems which would have been easier to find with the then-current version of advanced-browser.

[Yang2Gui3zi]

I retested with the latest versions of the add-ons advanced_browser.py and advanced_browser_custom_fields.py. The browser pane still blanks if I attempt to sort by clicking on the header of custom fields.

11 hours ago, hssm wrote, "...I'm hoping it's not some discrepancy in SQLite versions."

My main PC is a Mac running OS X Lion. However, I also have a (virtual) machine which runs the latest 64-bit Windows 7. Would you like me to test these two add-ons with my collection under Anki 2 on the Windows 7 machine? If they work under Windows 7, it would reinforce the hypothesis that there's a discrepancy in SQLite versions -- or in something else that differs between the implementations of Anki.

[hssm]

I tried it on a Windows 7 machine earlier and it still worked normally for me. I've also got the Chinese Support add-on installed and it makes no difference. You are free to try it there, if it's not too much trouble.

But first, are you able to run Anki from the console with the latest version of the advanced-browser (alpha3)? It should print out an error there when it encounters one. That should give us a clue.

[Yang2Gui3zi]

I imagine you are talking about running the debug console (http://ankisrs.net/docs/manual.html#debug-console) under OS X.

I have never done this before, but am certainly willing to try. So I will get ready to do that, but check back here first, in about five minutes, to see if you have written a reply telling me that you meant something other than the debug console under OS X.

(I should mention that my deck Developing Chinese 2nd ed and the columns now showing in the browser pane are slightly different from the version last deposited in the Dropbox for you. I didn't add any cards, but did fill in some empty Simplified fields.)

[hssm]

Sorry, it's been too long since I've used OS X... let's see. This looks like a decent guide: http://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line

You need to start Anki from the Terminal application in much the same way as the Running a Command example in the link shows you. I'm not too sure how Anki is distributed for OS X, but I imagine you just need to run a single anki command from the directory where Anki is installed. (I'm short on time right now, so if you can't figure it out, I'll have to dig up the details some other time).

[Yang2Gui3zi]

Oh, okay, when I wrote that last note, I wondered if you might be referring to running Anki from the Terminal utility application. Okay, I'll do a bit of reading on how to do that and give it a go.

[Yang2Gui3zi]

Sorry, I don't know how I closed the issue. I must have clicked on the wrong button. Anyway, I've reopened it now.

[Yang2Gui3zi]

Well, I ran Anki from a Terminal window and tested the sorting of a custom field in the browser by clicking on the header of its column. Terminal did not display any error messages in its window.

First, here's all the text that appeared in the Terminal window:

Last login: Tue Jun 25 08:21:41 on console
mbp8-4:~ lincoln$ pwd
/Users/lincoln
mbp8-4:~ lincoln$ cd /
mbp8-4:/ lincoln$ cd Applications
mbp8-4:Applications lincoln$ ls Anki*
Anki 2-0-11.app:
Contents

Anki 2-0-8.app:
Contents
mbp8-4:Applications lincoln$ open "Anki 2-0-11.app"
mbp8-4:Applications lincoln$ exit
logout

After I typed the command open "Anki 2-0-11.app" [Enter], Anki's main window opened:

I gather that the faint gray shadowing, in the image above, behind the deck name Developing Chinese 2nd ed indicates that that deck has been selected as the current deck.

Then I editted both add-ons to make sure they had the comment # Version: 0.1alpha3 in their preambles.

Then I clicked on the Browse hyperlink and the browser window displayed:

Note in the image above that the contents of the Sort Field and the Hanzi field are the same. This is because the radio button Sort by this field in the browser had been chosen for Hanzi in the Fields window.

Then I clicked on the header of the column for the field Pinyin, in order to test sorting by that custom field. The browser pane promptly blanked:

At that point I quit the Anki application windows and typed exit [Enter] in Terminal.

[Yang2Gui3zi]

Wait a minute: I believe I found what you were after. You wanted me to start Anki from Terminal, then look in Console.app for error messages. Well, I found this one in Console.app: 25/06/13 11:44:30.532 [0x0-0x56056].net.ichi2.anki: sql : select c.id from cards c where ((c.did in (1365911862218) or c.odid in (1365911862218))) order by (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) is null, (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) is '', (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) 25/06/13 11:44:30.532 [0x0-0x56056].net.ichi2.anki: Error finding cards: user-defined function raised exception

The last line talks about an error and an exception being raised. You, Houssam, had written this:

If you call an external function in your sort and it throws an error, your entire query will return nothing, hence the empty browser.

Now I can't read Python, but that SQL code looks to me as if it is trying to 'order' cards on a value that is selected in a field named 'Pinyin'. Seems to be exactly what I was doing by clicking on the header of the column for the Pinyin field.

[Yang2Gui3zi]

Okay, now I know how I closed the issue by mistake. I had typed a comment, and then I clicked on the button labelled Close & Comment . I believed that meant to close the comment window and post the comment, rather than to close the issue.

[hssm]

That's exactly what I was after, thanks. The valueForField function is a python function in the add-on that is called from SQLite. That's the "external function" that it is referring to. And for some reason, it is throwing an error on your end.

Can you please replace the file advanced_browser_custom_fields.py with this version(right-click and Save As) and try to produce those errors again? This version prints more verbose output on the error, so it should hopefully reveal the cause of the problem.

[Yang2Gui3zi]

I did as you asked, rerunning the test with version alpha3.1 of the add-on advanced_browser_custom_fields.py.

Within Console, this output was found in the section SYSTEM LOG QUERIES | All Messages:

26/06/13 6:20:53.729  [0x0-0x51051].net.ichi2.anki: sql : select c.id from cards c where ((c.did in (1365911862218) or c.odid in (1365911862218))) order by (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) is null, (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) is '', (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) 
26/06/13 6:20:53.729  [0x0-0x51051].net.ichi2.anki: Failed to get value for field.
26/06/13 6:20:53.729  [0x0-0x51051].net.ichi2.anki: Mid: 109495274
26/06/13 6:20:53.729  [0x0-0x51051].net.ichi2.anki: flds Error finding cards: user-defined function raised exception

That gives you some verbose output, but you wanted more. I found this try/except block within your add-on:

    try:
        index = _modelFieldPos.get(mid).get(fldName, None)
        if index is not None:
            fieldsList = flds.split("\x1f", index+1)
            #return anki.utils.stripHTML(fieldsList[index])
            return stripHTML(fieldsList[index])
    except Exception, e:
        print "Failed to get value for field."
        print "Mid:",mid
        print "flds", flds
        print "fldName", fldName
        print "_modelFieldPos", _modelFieldPos
        print "Error was: " + e.message
        raise e

So the Console only displayed output from the first three of six print statements. And it didn't display the value of variable (or attribute? object?) flds.

By the way, Console seems to display the same amount of info whether I run Anki by clicking on its icon in the dock or by opening /Applications/Anki 2-0-11.app from within Terminal.

[hssm]

Okay, so flds is coming back with nothing, which is odd. Can you comment out the lines that have print "fldName", fldName and raise e and try it again? I want to see if it happens for every row or just one or two.

You don't need to post the console output (it will likely end up with lots of these). You'll also get your results in the browser unsorted.

[Yang2Gui3zi]

Yes, I will comment out those two lines and run the test again. However, I have to leave soon for some meetings, so I won't be able to run the test until tonight or tomorrow.

I am unclear about your sentence, "You'll also get your results in the browser unsorted". Do you mean that the results of the print statements

        print "Failed to get value for field."
        print "Mid:",mid
        print "flds", flds
        print "_modelFieldPos", _modelFieldPos
        print "Error was: " + e.message

will appear in the browser pane even before I click on the header of the Pinyin column to sort it? But I thought that the print statements were only executed when an Exception occurred and that an Exception only occurred when I clicked on the column header to perform a sort.

[hssm]

The results will appear in the console output as usual, but it may be flooded with many copies of the error. Since we're catching the exception this time (and not throwing a new one by removing those bad lines), the query won't halt prematurely and return some results. So what I meant was that the browser won't be blank; it will have results, but the sorted order will be wrong.

This is to find out if it's one (or several) lines that are coming back with no flds or if it's happening for all of them. There is no need to post the entire output here if it's too large. I only need to know if it's one or many.

While you're at it, run this in the Anki Debug Console so I can know which version of SQLite you are running:

import sqlite3
print sqlite3.version
print sqlite3.sqlite_version

Also, the Anki docs say to use Command+: to load the debug console. If it doesn't work, try also holding shift. I'm not sure if the documentation is outdated or if the extra shift is required on Linux.

[Yang2Gui3zi]

I ran the test seven times. In the first test, Console displayed messages from Anki six times. Each test thereafter, Console displayed messages only once. Here are the messages displayed after the last test:

27/06/13 7:45:01.975  [0x0-0x88088].net.ichi2.anki: sql : select c.id from cards c where ((c.did in (1365911862218) or c.odid in (1365911862218))) order by (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) is null, (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) is '', (select valueForField(mid, flds, 'Pinyin') from notes where id = c.nid) 
27/06/13 7:45:01.975  [0x0-0x88088].net.ichi2.anki: Failed to get value for field.
27/06/13 7:45:01.975  [0x0-0x88088].net.ichi2.anki: Mid: 109495274
27/06/13 7:45:01.975  [0x0-0x88088].net.ichi2.anki: flds Error finding cards: user-defined function raised exception

The messages were basically the same in each test, except for the first test, when the six SQL statements had significant differences among them. There were no messages generated by the statements

        print "_modelFieldPos", _modelFieldPos
        print "Error was: " + e.message

If you would like me to send you the complete set of messages from the first test, or from all six tests, I can do so.

Note that, after I clicked on the Pinyin column header, in no test did any cards or anything except blank space display in the browser pane. No scroll bar was displayed on the right of the browser pane, so I could not scroll down below what showed on my monitor. I could click again on the Pinyin header, reversing the direction of the sorted (nonexistent) cards, but no cards were to be seen. The message in the title bar of the browser window read Browser (0 cards shown; 0 selected). The search field continued to read deck:current, as it had after I hit the [Enter] key when the browser window had opened.

I am at a loss to explain why the first test displayed six sets of messages, while each of the following tests displayed only one set of messages. The test preconditions were the same before each test, or at least I intended them to be the same. What I did to make them the same was this:

1. Before running the first test, I copied the complete folder /Users/lincoln/Documents/Anki/as folder /Users/lincoln/Documents/Anki full backups/Anki 2013-06-26 05h43m/. The name of this copy came from the date and time when I performed this copy. This folder thus contained the collection, the collection media, and the add-ons that were installed at the time.
2. Also before running the first test, I downloaded the raw verbose-error version of advanced_browser_custom_fields.py. I changed the version number comment and commented out the two lines previously mentioned, so that those three lines then read:
   # Version: 0.1alpha3.15
...
# print "fldName", fldName
...
# raise e
   After changing those lines, I then saved the add-on as /Applications/Installers/Anki add-ons & plugins/advanced_browser_custom_fields_0.1alpha3.15.py
3. Before running each of the tests, I deleted the folder /Users/lincoln/Documents/Anki/. In its place, I copied the folder Anki 2013-06-26 05h43m/. Then I changed its name to simple Anki/.
4. The first time I opened Anki in each test, I clicked Tools » Add-ons » advanced_browser_custom_fields » Delete... » Yes » OK and closed down Anki. Then I copied the add-on advanced_browser_custom_fields_0.1alpha3.15.py into the folder /Users/lincoln/Documents/Anki/addons/ and changed the copy's name back to simple advanced_browser_custom_fields.py. Then I performed the probably unnecessary deletion of advanced_browser_custom_fields.pyo, which I assume is the compiled version of the add-on.

In this way, I made the preconditions before each test the same. So I do not know why the first test produced six sets of messages, rather than only one set. However, it is always possible that I made some mistake in my procedure the first time, which would have created different preconditions.

Before writing the next post, I will run the Anki Debug Console (for the first time ever) to check what version of SQLite is loaded on my Mac. Also I will search through the Anki 2.0 Manual to see if it explains how to update SQLite to the latest version.

[Yang2Gui3zi]

Yesterday, 27th June, Houssam wrote by e-mail:

... Since you're probably still testing the add-on, I thought I'd save you some time from my mistake. I had actually meant to comment out the line that prints "flds", not "fldName". Sorry about that.

Sending me all of the output would be useful as well.

---

Hi, Houssam,

I read your e-mail only after writing the previous post.

Today I reran the test after commenting out the printing of field flds.

After I sorted by clicking on the Pinyin column header, the browser this time selected and displayed all 109 cards in the deck. Again, they were not sorted in the proper order. Rather than clutter up this post with an image of the browser, I have saved that image here, from where you can download it.

Console displayed messages at two points in time. The first point was when I sorted the Pinyin column. The second point was when I exitted Anki. You can download all the messages from here. I discerned some patterns in the messages, but you will undoubtedly see more than I do.

[hssm]

This is certainly odd. It's trying to get values for a model with ID 109495274, but your collection doesn't have a model with that ID. At least, not the one that I have. Is it possible you've changed it since the last version you uploaded to DropBox?

[hssm]

Ooooh I see what is happening now. You have a model ID with the value 1365909095402. This is very big number. When cast as an integer, it becomes 109495274. Something, somewhere, in the chain of libraries used by Anki on OS X, is turning that long into an int. It could be python or SQLite or some lower level system library, I'm not certain yet. (I'm explicitly casting it into an int in python, but they are meant to be equivalent. Maybe not?)

What a wonderful bug. At least I know what it is now!

[Yang2Gui3zi]

I'm glad you have isolated the bug, apparently, and can proceed to resolving it or finding a work-around.

(This post diverts away from the issue of sorting in advanced_browser_custom_fields, so if you'd like to continue via direct e-mail or maybe on some other Anki forum, please let me know.)

Meanwhile, I have found out what version of SQLite is in my Mac's libraries. Terminal displays

Last login: Fri Jun 28 18:18:09 on console
mbp8-4:~ lincoln$ sqlite3
SQLite version 3.7.7 2011-06-25 16:35:41

This agrees with the output of the Anki Debug Console

>>> import sqlite3
... print sqlite3.version
... print sqlite3.sqlite_version
2.6.0
3.7.7

According to the SQLite download page today, the latest version is 3.7.17. So the libraries on my Mac are ten dot-dot-versions behind, assuming the libraries on my Mac are the sqlite3.sqlite_version.

Would you please explain to me what the difference is between the sqlite3.version and the sqlite3.sqlite_version? and why the former is a whole version behind the latter?

Updating the SQLite libraries is unlikely to resolve the bug, but I'd like to update them, anyway.

SQLite's Webpages do not tell me how to update SQLite on a Mac, but googling turned up two reasonably straightforward approaches.

One of these is Homebrew, a generalized package manager for Mac OS X. This has "formulae", sets of parameters that drive installation and, it seems, updates, as well. For example, the formula for SQLite.

There are three "howevers" with the Homebrew approach to updating SQLite:

• I'm not certain that Homebrew will handle an update to an SQLite version that was not installed by Homebrew. I am hoping that there's a forum somewhere on the WorldWide Web that will answer this question.
• It seems having either Xcode or Command Line Tools for Xcode installed is a prerequisite. Possibly I already have Xcode installed, but I am not sure.
• Homebrew apparently has a Python engine and recommends that I first install Python through Homebrew itself. This adds another layer of complication.

The other approach is a relatively simple new installation of SQLite. However, I guess that this will not perform an update, so I have to figure out first how to uninstall the current version of SQLite.

[hssm]

sqlite3.version is the version of the python module that serves as the wrapper to sqlite3. sqlite3.sqlite_version is the version of the actual sqlite3 library installed on the system.

The windows machine that I tested it on has versions 2.6.0 and 3.6.21, so even with an older sqlite3, I don't get this problem. Another possibility could be the compile-time options chosen to build the binary, or maybe you have the 32-bit version installed. You can check the latter if you put this in a python interpreter (the anki debug console will work just a well):

import struct
print struct.calcsize("P") * 8

I've never had the occasion to fiddle with packages on OS X, so I can't really offer you any advice there (other than to not do it, since it's usually perilous).

[hssm]

I got to try it on a 32-bit Windows machine, and even on that I don't end up with a problem getting the right value. In any case, I've put in a workaround which you can try. It's in 0.1alpha5 which you can grab off the master branch in this repository.

[Yang2Gui3zi]

I installed the 0.1alpha5 versions of both advanced_browser.py and advanced_browser_custom_fields.py. They work to sort on a custom field by clicking on the header! (I haven't tested any other functions.)

The sort puts the cards in an order other than I am used to, but one that I can easily get used to. For example, all uppercase characters come before any lowercase characters. I have a vague memory that that order actually follows the ASCII table: I haven't checked to see if my memory is correct. However, I am used to seeing ordered lists which follow, in the alphabetic portion, "A a B b C c...". Probably the sort algorithms I am used to convert all lowercase characters to uppercase (or vice versa), then sort, then restore the original character string before displaying. ... But, as I said, I can easily get used to the sort order implemented in the 0.1alpha5 version.

Thank you very much for seeing this issue through to a workaround.

Do you want to find out where the long gets truncated into an int? Or shall we just leave it at the workaround?

[hssm]

Good, I'm glad it's working now. I'm aware of the sorting behaviour and plan to add some way to configure the ordering style for each column eventually (e.g., sort numerically). But maybe the default behaviour can be improved if we use a built-in collation order. I will try that next.

I don't want to invest any more time on the long/int issue, especially since there's likely nothing I can do about it beyond just accounting for its existence like I'm doing now.

I'll let you close this issue if you're satisfied with the fix.

[hssm]

While we're here, the change to ignore case while sorting is simple enough, so I've checked in a copy that does just that.

[Yang2Gui3zi]

Thanks, the version that ignores case works fine for me.

If you do add some way to specify the collation order for each column, be aware that many Anki language learners will be using character sets that include glyphs outside of alphabets, such as CJK (Chinese Japanese Korean) regions of Unicode. I guess that the default sort order in Anki is based on UTF-8.

[hssm]

Yes, I'm aware of that (and happen to be among those users). SQLite has few built-in collation sequences. It can sort by the binary value and optionally fold upper-case ASCII characters (which I just added), but that's it, the latter of which is the one that Anki uses.

I can defer collation to a python library or SQLite extension, but then it becomes a problem of efficiency. We already have to split the fields and strip HTML, so it's already kind of slow as it is.