allocation of user defined whatsits

[Udi-Fogiel]

The LuaTeX engine let you define new custom whatsits and identifying them by setting them an ID. Currently there is no built in way (that I'm aware of) in OpTeX to allocate these IDs so if several macro writers use this feature they can possibly clash by using the same ID. Allocating these IDs can prevent that.

[olsak]

What do you suggest to implement exactly?

[Udi-Fogiel]

Probably something like (I don't think user_id the same limit as counts)

-- Allocator for user_defined whatsit nodes. There is
-- is interface only at the lua end. 
local whatsits = {}
function alloc.new_whatsit(name)
    whatsits[#whatsits+1] = name
    return #whatsits
end

I don't see any point for a support from the TeX end.

[Udi-Fogiel]

@vlasakm any thoughts?

[vlasakm]

Sorry, I must have missed this discussion, @Udi-Fogiel thanks for pinging me.

Addition of alloc.new_whatsit as you propose sounds reasonable. Though I had to freshen up on the topic, because it's been a long time since I needed whatsits for anything :). TeX nodes have types and subtypes, these are fixed by TeX. One of the node types is whatsit, and it has subtypes like write, special or pdf_literal. One special whatsit subtype is user_defined, which has a user_id field to be arbitrarily set by the user. Now, everybody should be using different user_ids, so just like with counters and every other register, the best idea is to introduce an allocator into the format that would allow each user to just ask for a unique user_id.

I looked briefly into my older clone of TeX Live, and I didn't find that many uses of the allocator. Personally I thought whatsits are problematic in multiple ways - e.g. interaction with various TeX algorithms which although nominally ignore whatsits, still see them as nodes in the node lists, which can cause some problems. Hence the better approach for e.g. implementing color has been LuaTeX attributes. For other traditional uses of whatsits like PDF literals or images, I agree with the LuaTeX team that a rule is a much better way to represent those. TeX algorithms already know that rule is something abstract, but with a height, width and depth that it has to take into account for calculations, unlike the "dimensionless" whatsits. There is a user subtype of rule, and I guess it could be used, and distinguished by attributes, and we have a attribute allocator already.

Just out of interest @Udi-Fogiel what is your use for user defiend whatsits?

So here are a few questions:

Do we want to implement this just because it is missing, or do we actually have a need for it? I think being futureproof might make sense even if nobody has any use for it, but in this particular case, whatsits sound like the past, not the future :).

I like your simple definition with just a local counter. But other allocators are based on TeX counters with name _<resource>alloc (like _attributealloc). On one hand, this is unnecessary since this is a Lua only resource, that is not interesting on the TeX side, but on the other hand we use the regularity to implement the compatibility with minim allocators (https://github.com/olsak/OpTeX/blob/1096beffa4cafb9f6ae063abf841c0a9bd3f97fb/optex/pkg/minim.opm#L155). But we actually already kind of do have an exception: something like our define_lua_command is in LaTeX / minim a two step process: allocate a lua function id, define a token to Lua function. My idea was, that because the functions are stored in a table anyways, there is no need for allocator, since the size of the table itself is a generator of "fresh numbers".

I would have to also freshen up on LuaMetaTeX, but I think it also simplified things by removing whatsits, though I don't remember what exactly is the offered alternative (probably the user rules and attributes on nodes), so introducing a whatsit allocator now, may not prove futureproof with regards to LuaMetaTeX (if ever considered by OpTeX).

With user whatsit ids, I am not sure what is the better approach to take. I think it would be nice to have it integrate with minim, and that either requires a TeX counter or a special case. (I personally have no problem with either.).