confusing points about token types

[pete-]

A couplefew things:

• If users are expected to use the TT* names everywhere else, it makes sense for them to use the TT* names when calling H_MAKE. Currently, the HMAKE macro adds the TT prefix on its own. I expect this is because the implementation of the underlying h_make function expects itself to be used only for user-defined token types, thus shielding the user. Given the nascent token-type registry, however, I believe this is destined to (and should) be changed.
• I understand that folding the TT_ prefix into the macro body is intended to make the H_MAKESEQ, etc macros look more or less the same, but I still think it's confusing to use the TT some places but not others.
• Also given the nascent token-type registry, I think it makes sense to provide an another wrapper for h_make that interacts with it. Perhaps an "H_MAKE_USER" macro would do the trick, taking a string and doing the registry lookup within. Though perhaps a full-on function might be better.
• That said, were I allowed to be super-greedy, I'd love to see exactly one token-creation entry point that expects an int, and exactly one wrapper that expects a string and converts it to an int via a registry lookup. That said, I totally see how it's more efficient to do it this way (though I think it might be worth at least investigating how much prettier the code is vs. how much slower it performs). What I wouldn't give for pattern matching in C!

I'm happy to write code for any/all of this, but I don't want to start digging in without some discussion/approval from the powers-that-be. =)

[pete-]

Also, the existence of all the macros that shield users from having to worry about the arena allocator makes it difficult to see how one would create user-defined tokens while still keeping p->arena duly hidden. Dems sum inconsistencies and I likes me sum consistency.

[thequux]

1. The macros were never intended to hide functionality; they were always intended to handle common cases with fewer tokens. I.e., we found that code was easier to read with fewer references to p->arena.
2. Token creation happens in a critical path (parse tree creation), so performance is somewhat more critical than it would otherwise be. Having to stash token IDs somewhere else doesn't seem that bad, particularly considering that canonical token names will probably be fairly long. In fact, even if we did have an H_MAKE_USER macro that takes a string, I would want there to be some facility for adding aliases, at which point we're just reimplementing the host language.
3. Point noted regarding TT_. Do you want to send that PR or should I?

[abiggerhammer]

In general I'm in favour of a consistency-oriented rewrite. Pete, how about you do one like you're envisioning, and send along a PR?

[pete-]

@thequux

re: 2. Precisely. That's what I was trying to get at with my fourth bullet point, though it almost certainly could have been phrased better. I'll see if I can come up with something both intuitive and efficient.

re: 3. I'll take care of it.

@abiggerhammer

On it.