Enhancing the indexer package

[Omikhleia]

Main goal: I need multiple index support for a potential project. Stretch goal: As #1194 progresses, it could be nice for the SILE manual to include an index of all mentioned settings, commands, key concepts, etc.

The indexer package is a nice start.

I however noted a few things to fix and improve, from the easiest to the more challenging. Let's track them here.

• [ ] docs: No, \printindex does not "deposit (entries) in a list". That's vague and obscure. It actually renders the index.
• [ ] docs: index:item is passed the indexed content as argument, and an option pageno (which is a list of comma-separated pages, so it could be a plural here). Could be nice to document.
• [ ] fix: if nothing was indexed, \printindex without option aborts with a low-level error (pairs() being called on nil...). Let's play fair, it could just be a warning that the "main" index is empty, and let the document be processed to end.
• [ ] fix: If index "myindex" does not exist, \printindex[index=myindex] aborts with that same low-level error. Let's play fair again, it could just be a clear error g that the "myindex" index is unknown (it could be a typo, whatever). If it exists, but is empty - this shouldn't happen in the current implementation - a warning would be ok.
• [ ] feat: The book class should support the indexer by default, so we have one potentially class using it...

That's the simple bits. Call that "Stage 0" tasks.

Let's now go for real things....

• [ ] feat: PDF link annotations (page linking to the index call), for the index to be useful in a PDF reader. Would be easy a first glance, but see next item below...
• [ ] feat: Collecting subsequent pages into ranges, so one does not get "someentry 62, 63, 64, 65, 70, 71, 72" but rather "someentry 62-65, 70-72" (in which case linking could be on the first occurrence of a range).
• [ ] feat: Explicit ranges. The raw entry might be "someentry 62, 65", but the author knows 62-65 would be more appropriate. (Several LaTeX packages use delimiters such as |( and |), for the record).
• [ ] feat: Distinguishing the index key (used for sorting) from actual the content. This would allow the later to contain commands. Currently there is no way, e.g. to index "John \smallcaps{Doe}" at "John Doe" indeed but with the formatting kept. (For the record again, LaTeX uses delimiters '@' and/or '|' for formatting. I don't like their syntax much, but it also allows index "John Doe" and e.g. "Doe, John" to be indexed at the same entry, especially if one want to do it in a macro (so options are not the solution)
• [ ] feat: Multi-level indexes, a.k.a. sub-entries or nested entries. (For the record, LaTeX has a ! delimiter). It's the last feature in my list, but I would need it eventually. Example use cases:
  • having all members of a family listed together. Doe (family) + John 35, Peter 70, Sarah 71-72.
  • in geographical or linguistic works, e.g. to list some sub-places, variants, etc. within the same main entry.

Some things that are less important to me...

• [ ] Cross-referenced entries. E.g. "Aaaa, 32 (...), Dddd, see Aaaa.". (LaTeX |see or |seealso, IIRC.)

And there are the troubleshooting tasks anyway...

• [ ] fix: Lost indexes (or rather "stuff seen in passing, to reproduce and perhaps investigate")

  • Edge-case 1: Since info nodes are moved the class endPage, any entry that would be on the same page as the \printindex will not be present. That's for the case where the index doesn't stand on its own starting page. But still.

  • Edge-case 2: On one case I had an index alone in a paragraph, i.e.

    Some text
    
    \indexentry[label=titi]
    
    Some text

    This was not indexed. Maybe that's a deeper issue with the special types of info nodes and that was intended for some reason. The info node was inserted in SILE.typesetter.state.nodes (checked), but its outputYourself was never called, so likely the fake "special" hbox was eaten by something. Would need a check because I am pretty sure I had met this already with my cross-reference package (and while a standalone indexentry is a bit dumb, a standalone label entry is not so so weird). At least the condition(s) that make(s) those boxes disappear should be checked/documented...

That's already a long list, for sure.

[alerque]

Main goal: I need multiple index support for a potential project.

I have skimmed your message and probably a lot of the detail points are great ideas. At first blush however I'd just like to mention that SILE already provides most of the tooling needed for indexing as many ways from Sunday as you want.

For example one of the things I added to the cabook class (the default book class in CaSILE) was a verse index that keeps track of all verse references mentioned in the text and allows appendixes to be build with links pack to pages that referenced them. This was implemented starting with a copy of SILE's tableofcontents package, but all the tooling is there.

That caveat aside, yes making it more ergonomic to add your own indexes would be a welcome improvement.

[Omikhleia]

SILE already provides most of the tooling needed for indexing as many ways from Sunday as you want.

I didn't daresay at the time, but it made my day laughing ;)

[Omikhleia]

And, as if that long list of ideas ever made any sense...

https://github.com/sile-typesetter/sile/blob/1a86520d198d7709966e1fd1bfd8dfaceda86393/packages/indexer/init.lua#L61

Language-dependent sorting? Collation? Gimme a hint.

[alerque]

table.sort() should just stay as the Lua default function, and as you started doing we can provide a utility function that does locale-aware sorting for typesetting specific purposes.