Sentences are missing for archived texts, which messes up references.

[jzohrab]

While reading today, I had a word showing up as "known", but I couldn't find a reference for it (when I clicked "sentences"). Turns out a book had been archived, and the sentences had been removed. This is probably a relic of prior code that used to wipe sentences.

This will likely be an expensive operation (will have to re-parse any texts that don't have sentences), so maybe have a special section for one-time jobs, and only show this if there are texts that don't have sentences.

Will this blow up the size of the DB?

[jzohrab]

Yes, this will make the DB large. A few options:

• store the processed texts (with zero-width-space word delimiters) on disk in the file system, and read from there. Don't know what speed implications this would have. Could store an index in the db (wordrefs -> WoID, TxID), that table would have lots of records (same as book wordcount) but would be fine.
• have a separate "library" db just with the processed texts? Really, if I store things on disk, who cares whether it's in separate files, or in a single sqlite db? It's the same thing in the end. So the dbs would be: library with all texts, and lute, just with words and various other things for current active texts.
• have some clever ??? way to remove sentences that don't add any value (e.g., sentences where all words are well known, or sentences whose words are already contained by other more recent sentences).

[jzohrab]

Not fixing this -- it's a bug due to old code, life moves on, and things get better. We hope. Anyway, the way that text is stored/parsed needs to be revamped a bit, I think, as the db currently contains stuff that it really doesn't need (eg. texts.TxText seems redundant, when we have the Sentences table).