Space terminated dimen assignment doesn't work

[Skillmon]

There seems to be an error regarding tokens following a dimension assignment:

\makeatletter
\newdimen\ifdimen@gobbledimen
\long\def\ifdimen@test\ifdimen@false#1#2{#1}
\def\ifdimen@false{\z@\ifdimen@false\@firstofone}
\protected\long\def\ifdimen#1%
  {%
    \afterassignment\ifdimen@test
    \ifdimen@gobbledimen #1 \ifdimen@false
  }
\makeatother

\ifdimen{1}{true}{false}
\ifdimen{1pt}{true}{false}

\unravel{\ifdimen{1}{true}{false}}
\unravel{\ifdimen{1pt}{true}{false}}

This definition will work outside of \unravel but inside \unravel the space after #1 will not be gobbled if #1 is a valid dimension and therefore \ifdimen@test will throw an error as it doesn't match its definition.

On the other hand

\makeatletter
\newdimen\ifdimen@gobbledimen
\long\def\ifdimen@test\ifdimen@false#1#2{#1}
\def\ifdimen@false{\z@\ifdimen@false\@firstofone}
\protected\long\def\ifdimen#1%
  {%
    \afterassignment\ifdimen@test
    \ifdimen@gobbledimen #1\ifdimen@false
  }
\makeatother

\ifdimen{1}{true}{false}
\ifdimen{1pt}{true}{false}

\unravel{\ifdimen{1}{true}{false}}
\unravel{\ifdimen{1pt}{true}{false}}

will throw an error outside of \ifdimen if #1 is a valid dimension as \ifdimen@false will be expanded, but work inside of \unravel

[blefloch]

Thanks a lot for finding this bug. I think it's a one-line fix, but I want to think about it a bit more. I might send an update to CTAN this weekend.

[blefloch]

Your tests are very nice and caught a stupid logic mistake I made initially, thanks.

I should note though that your approach for searching for floats fails for \ifdimen{1pt }{}{}.

[Skillmon]

@blefloch you're right, haven't thought of malevolent input, but the test isn't meant to be rock-stable, but to be just fast (and that it is, haven't found or thought of anything faster yet, and I gave it some thought). But that would be easy to harden by changing to \long\def\ifdimen@test#1#2#3{#2}, but that's slower, as TeX has to pass in an additional argument :) Not sure which route I should choose here.

[jfbu]

viz your "I'm not sure why" commit message https://github.com/blefloch/latex-unravel/commit/bb4640ac0c555080934ed2ec0fe508bc9e5da54d, possibly it may have had to do with allowing assignments <dimen>=3pt<EOL> without having to worry about end-of-line space token.

This elimination of optional space can aslo impact \if tests

\dimen255=3pt
\edef\foo{\ifdim\dimen255>2pt\else\fi}
\meaning\foo
\bye

(or without \else also).

("unraveling" the above does give the \relax with bb4640ac0c as TeX does)

[blefloch]

Thanks, I've just added a test along those lines.

While writing tests with fill units, I realized that while "p t" is not a valid unit, "fil \space l" is valid. Curious side-effect of the implementation.

[jfbu]

About the "fil unit" there is a comment in TeX by Topic 36.2 Keywords:

A keyword is sequence of characters (or character tokens) of any category code but 13 (active)....TeX does not distinguish between lowercase and uppercase characters in keywords. ... any keyword can be preceded by optional spaces... ...By far the strangest example, however, is provided by the grammar rule ⟨fil unit⟩ → fil | ⟨fil unit⟩l which implies that fil L l is also a legal ⟨fil dimen⟩. Strange errors can ensue from this;

The notion of "keyword" arises in the grammar of TeX:

Here is the full list of all keywords: at, bp, by, cc, cm, dd, depth, em, ex, fil, height, in, l, minus, mm, mu, pc, plus, pt, scaled, sp, spread, to, true, width.

(I am sure of course you have implemented already the above rules)

I have always admired Victor's in depth and concise description of TeX internals (I should test what happens if letter 'p' is assigned catcode 14 and if above description is really ok) but it looks so clear that I never looked at TeX source code except on one exceptional case, as the need did not arise. I think you are reading closely TeX source code and are probably the best knowledgeable person currently (or second best? let's not forget D.K.)

[jfbu]

keyword is sequence of characters (or character tokens) of any category code but 13

I should test what happens if letter 'p' is assigned catcode 14 and if above description is really ok

Well a catcode 14 p will behave as comment early on and no it does not work.

\catcode`p 14
\dimen255=3pt
\catcode`\p 11
\the\dimen255

Perhaps Victor refers to later stage (in fact he says "character token" and probably a catcode 14 p never becomes a "character token"). Strangely the above triggers nothing in log, no error message.

[jfbu]

ah no silly me of course no error because the 3 multiplies the 14 so it becomes 42sp... and p does not recover its catcode letter...

\catcode`p 14
\dimen255=3pt
\catcode`\p 11
\the\dimen255
\catcode`\p 11
\the\dimexpr42sp\relax
\bye

not tested with unravel...

[blefloch]

I'm only knowledgeable about some aspects of tex.web; for instance I haven't had to look at boxes very much. On the other hand, I've had to look at how TeX scans units and so on quite carefully. In fact, my code for that follows TeX's quite closely, to the point where I use the same magic numbers and almost the same function names.

Since unravel isn't able to deal with catcode changes, it's not possible to test these things with unravel.