Handle color using attributes

[vlasakm]

This is an attempt to get rid of \pdfcolorstack + \aftergroup mix along with global / local colors concept. The implementation handles color using on LuaTeX's attributes. For some motivation and information about differences of attribtes vs colorstacks/literals see the documentation added to optex.lua in this pull request.

This is sadly a big and breaking change. Not that the old behaviour couldn't somehow be emulated, but perhaps it would be beneficial to do it "the right way". This means that we should first define what is the right and expected behaviour when color is involved.

This pull request colors content (text, rules, etc.) based on its attributes. Attributes stay frozen, like the font property of char/glyph nodes. No boxing and unboxing changes that. Nodes (what becomes out of characters, boxes, etc) get assigned attributes at their creation, which means that there are edge cases like the horizontal lists created by linebreaking, which get attributes that are active at the end of paragraph. This is no problem for this implementation, because we only care about nodes which have relation to colors (glyphs, rules, pdf literals).

One can think of it like this:

• Attributes: color for content is decided when the content is created and doesn't ever change.
• Colorstacks: color is determined by what happens to surround the content at shipout time and we somehow try to achieve the above.

Is it for example expected, that an insert created when "\Green" is set, will be green? I don't think so, that is why this implementation still forces color reset at some places (instead of \_ensureblack this is called \_resetcolor).

I chose to not implement coloring of leaders. I tried, but I don't think its worth the complexity. Without handling leaders the whole coloring is so consise and easy to understand. Note that despite that perhaps all reasonable uses of \leaders work (for example \.tecky from CTUstyle3).

Like with colorstacks, one can also set a "default" color. So the main text color doesn't have to black. This is very easy to support, but is maybe non-obvious and maybe we would be better without it. (But I think OpTeX currently supports it with \_pdfblackcolor). If we keep this we need to somewhere draw line between what is \Black and what is "default color".

Do we need to support \_currentcolor? I currently didn't implement it. It would be kind of painful, because it is the TeX/Lua interfacing again. I think that with the right grouping mindset, it isn't needed, but maybe I am wrong.

Should \_setcolor be implemented in Lua, or be perhaps more like:

\def\_setcolor#1{\colorattribute=\translatecolor{#1}}

Handling of general inserts and footnotes has to be decided. I currently reset the color to black at the start of each insert.

Perhaps it would be worth to devise better mechanism for Lua scripts in OpTeX. define_lua_command is how we let TeX call into Lua, because it allows us to create "pseudoprimitives". Each feature could then be implemented in separate file and exposed to Lua only using this. The Lua code could even be inlined to .opm files, and still have no cost at run time (only at format creation time). I don't know a nice way to make \public variants of these Lua primtives.

Currently the implementation hooks itself into output routine by redefining \shipout. Onether possibility is to define our preshipout callback, which could be used by coloring (this is how it is done in LaTeX and essentially also in ConTeXt).

PDF form XObjects are problematic because of \immediate. Currently everyone has to first \colorizebox and then use \pdfxform.

I still have to check handling of discretionary nodes (I think that they don't participate much in this stage).

Maybe we can save some color switches/checks -- for example images are(?) not influenced by color setting. However, this increases complexity and probably doesn't save anything in real scenarios.

Surprisingly for documents I tried in the end everything seemed to work correctly. But that is because usual documents don't use nonlocal colors, colored inserts and a lot of boxing/unboxing.

All in all, these are the advantages:

• more natural than colorstacks
• no catches with aftergroup
• smaller PDF output [1]
• should have no spacing issues regardles of use of color
• some OpTeX macros can be simplified

Disadvantages:

• has to be implemented in Lua (That means we have to draw border between TeX and Lua, and the interaction is not that nice. My idea is implemented, but I am up for debate for any of that.)
• incompatible with old behaviour
• arguably more unnatural - whatsits are the way we are used to
• is slower [1] and eats more memory (I didn't measure, but shouldn't be much)
• doesn't fully handle leaders

[1]: I tested my thesis (mediocre use of colors) and optex-doc.tex (more substantial use of colors):

Thesis:

Benchmark 1: mmoptex vlasami6-bp.tex (with attributes)

  Time (mean ± σ):      1.606 s ±  0.008 s    [User: 1.516 s, System: 0.087 s]
  Range (min … max):    1.594 s …  1.622 s    10 runs

File size: 463782 (compressed), 1604511 (uncompressed)

Benchmark 2: mmoptex vlasami6-bp.tex (with colorstacks)

  Time (mean ± σ):      1.592 s ±  0.013 s    [User: 1.500 s, System: 0.090 s]
  Range (min … max):    1.573 s …  1.613 s    10 runs

File size: 469759 (compressed), 1643950 (uncompressed)

OpTeX documentation:

Benchmark 3: mmoptex optex-doc.tex (with attributes)

  Time (mean ± σ):      4.251 s ±  0.122 s    [User: 4.124 s, System: 0.121 s]
  Range (min … max):    4.132 s …  4.566 s    10 runs

File size: 1374568 (compressed), 8072243 (uncompressed)

Benchmark 4: mmoptex optex-doc.tex (without attributes)

  Time (mean ± σ):      4.188 s ±  0.022 s    [User: 4.059 s, System: 0.117 s]
  Range (min … max):    4.156 s …  4.223 s    10 runs

File size: 1383204 (compressed), 8163743 (uncompressed)

The time loss seems to stay under 2 % (0.8 % for thesis, 1.5 % for documentation). What surprised me was the size shrink, even for compressed files (1.3 % for thesis, 0.7 % for documentation). I think that this trade-off alone may be worth.

This diff snippet shows how we decrease the file size:

@@ -55680,16 +54605,7 @@ BT
 1 0 0 1 76.883 112.783 Tm [<006A00390033>]TJ
 1 1 0 0 k 1 1 0 0 K
 /F35 7.97011 Tf
-1 0 0 1 90.791 112.783 Tm [<0024006E0054006D002300480042002B>]TJ
-0 g 0 G
-1 1 0 0 k 1 1 0 0 K
-1 0 0 1 128.881 112.783 Tm [<0024002300420023006900320074003F005100510046>]TJ
-0 g 0 G
-1 1 0 0 k 1 1 0 0 K
-1 0 0 1 179.666 112.783 Tm [<002400230042002300510054006900420051004D0062>]TJ
-0 g 0 G
-1 1 0 0 k 1 1 0 0 K
-1 0 0 1 230.452 112.783 Tm [<00240023004200230054001C00600069>]TJ
+1 0 0 1 90.791 112.783 Tm [<0024006E0054006D002300480042002B>-531<0024002300420023006900320074003F005100510046>-531<002400230042002300510054006900420051004D0062>-531<00240023004200230054001C00600069>]TJ
 0 g 0 G
 1 0 0 1 268.541 112.783 Tm [<0063>]TJ
 0 1 1 0 k 0 1 1 0 K
@@ -55705,9 +54621,7 @@ BT
 /F15 9.96264 Tf
 1 0 0 1 426.068 93.232 Tm [<00420062>-367<006D00620032002F>-367<0023>-28<00320037005100600032>-368<005400600042004D>28<00690042004D003B>]TJ
 1 0 0 1 70.866 81.277 Tm [<002B001C0054006900420051004D>-333<0042004D>-333<0069001C00230048003200620058>]TJ
-0 g 0 G
 1 0 0 1 292.656 55.46 Tm [<00390033>]TJ
-0 g 0 G
 ET

 endstream

Redundant settings of color are eliminated and this allows LuaTeX to sometimes merge following texts into one text object with kernings, instead of having to set new text matrix.

Things to sort out before merging:

• [x] Do we even want this considering the advantages/disadvantages?
• [x] Is there any breakage of existing documents?
• [x] How to handle inserts?
• [x] More thorough implementation of leaders?
• [x] Concept of \Black vs default color?
• [x] Move implementation to seperate file? Inline into colors.opm? Better Lua mechanism?
• [x] Preshipout callback?
• [x] What about PDF xforms?
• [x] Check discretionary handling.
• [x] Smarter color checks (for e.g. images)?
• [x] \_setcolor in Lua or TeX?
• [x] What about the old local/global colors concept?

More to do:

• [x] Add documentation to TeX side.
• [x] Allow typsetting the documentation inoptex.lua.
  • I have not looked deeply into it yet, but inline code with ` seems to be the problem.

I am surely missing something, but why is the nested \_hbox to0pt needed in \draftbox?

I am certain that this brain dump is somewhere wrong and I may have not explained well something that now seems obvious to me. Please comment with anything that comes to mind.

@olsak I would be grateful for your review of the TeX parts and suggestions on above raised points. I am available to you here or privately. I thought that for code review/updates we could use Github with its review features and add more commits to pull requests (either by me or you). This way we could enhance this branch with small changes and then squash the commits for nicer git history. But only if you are really interested in this feature at all, after really looking into it I can see reasoning for both approaches (colorstacks and attributes). Let me know.

[vlasakm]

Example how this works:

\fontfam[lm]
\onlyrgb

black {\Red \hbox{m\Brown ixe}d {\Green\hbox{green}} red} black

\bye

Before colorize:

└─VLIST width: 455.24pt, height: 718.25pt
  ╚═head:
    ├─VLIST width: 455.24pt, height: 694.25pt
    │ ╚═head:
    │   ├─GLUE subtype: topskip, width: 3.06pt
    │   ├─HLIST subtype: line, width: 455.24pt, depth: 2.06pt, height: 6.94pt
    │   │ ╚═head:
    │   │   ├─LOCAL_PAR
    │   │   ├─HLIST subtype: indent, width: 20pt
    │   │   ├─GLYPH subtype: 256, char: b, width: 5.56pt, height: 6.94pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: l, width: 2.78pt, height: 6.94pt
    │   │   ├─GLYPH subtype: 256, char: a, width: 5pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: c, width: 4.44pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─KERN kern: -0.28pt
    │   │   ├─GLYPH subtype: 256, char: k, width: 5.28pt, height: 6.94pt
    │   │   │   properties: {['injections'] = {['leftkern'] = -18350.08}}
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt
    │   │   ├─HLIST subtype: box, width: 20.83pt, depth: 0.11pt, height: 6.57pt, attr: 1=1
    │   │   │ ╚═head:
    │   │   │   ├─GLYPH subtype: 256, char: m, width: 8.33pt, height: 4.42pt, attr: 1=1
    │   │   │   ├─GLYPH subtype: 256, char: i, width: 2.78pt, height: 6.57pt, attr: 1=2
    │   │   │   ├─GLYPH subtype: 256, char: x, width: 5.28pt, height: 4.31pt, attr: 1=2
    │   │   │   └─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=2
    │   │   ├─GLYPH subtype: 256, char: d, width: 5.56pt, height: 6.94pt, depth: 0.11pt, attr: 1=1
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt, attr: 1=1
    │   │   ├─HLIST subtype: box, width: 23.36pt, depth: 2.06pt, height: 4.53pt, attr: 1=3
    │   │   │ ╚═head:
    │   │   │   ├─GLYPH subtype: 256, char: g, width: 5pt, height: 4.53pt, depth: 2.06pt, attr: 1=3
    │   │   │   ├─GLYPH subtype: 256, char: r, width: 3.92pt, height: 4.42pt, attr: 1=3
    │   │   │   ├─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=3
    │   │   │   ├─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=3
    │   │   │   └─GLYPH subtype: 256, char: n, width: 5.56pt, height: 4.42pt, attr: 1=3
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt, attr: 1=1
    │   │   ├─GLYPH subtype: 256, char: r, width: 3.92pt, height: 4.42pt, attr: 1=1
    │   │   ├─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=1
    │   │   ├─GLYPH subtype: 256, char: d, width: 5.56pt, height: 6.94pt, depth: 0.11pt, attr: 1=1
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt
    │   │   ├─GLYPH subtype: 256, char: b, width: 5.56pt, height: 6.94pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: l, width: 2.78pt, height: 6.94pt
    │   │   ├─GLYPH subtype: 256, char: a, width: 5pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: c, width: 4.44pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─KERN kern: -0.28pt
    │   │   ├─GLYPH subtype: 256, char: k, width: 5.28pt, height: 6.94pt
    │   │   │ ╚═  properties: {['injections'] = {['leftkern'] = -18350.08}}
    │   │   ├─PENALTY subtype: linepenalty, penalty: 10000
    │   │   ├─GLUE subtype: parfillskip, stretch: +1fil
    │   │   └─GLUE subtype: rightskip
    │   ├─GLUE stretch: +1fill
    │   ├─KERN subtype: userkern
    │   └─GLUE
    ├─GLUE subtype: baselineskip, width: 17.34pt
    └─HLIST subtype: box, width: 455.24pt, height: 6.66pt
      ╚═head:
        ├─GLUE stretch: +1fil, shrink: -1fil, shrink_order: 2
        ├─GLYPH subtype: 256, char: 1, width: 5pt, height: 6.66pt
        └─GLUE stretch: +1fil, shrink: -1fil, shrink_order: 2

After:

└─VLIST width: 455.24pt, height: 718.25pt
  ╚═head:
    ├─VLIST width: 455.24pt, height: 694.25pt
    │ ╚═head:
    │   ├─GLUE subtype: topskip, width: 3.06pt
    │   ├─HLIST subtype: line, width: 455.24pt, depth: 2.06pt, height: 6.94pt
    │   │ ╚═head:
    │   │   ├─LOCAL_PAR
    │   │   ├─HLIST subtype: indent, width: 20pt
    │   │   ├─GLYPH subtype: 256, char: b, width: 5.56pt, height: 6.94pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: l, width: 2.78pt, height: 6.94pt
    │   │   ├─GLYPH subtype: 256, char: a, width: 5pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: c, width: 4.44pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─KERN kern: -0.28pt
    │   │   ├─GLYPH subtype: 256, char: k, width: 5.28pt, height: 6.94pt
    │   │   │ ╚═  properties: {['injections'] = {['leftkern'] = -18350.08}}
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt
    │   │   ├─HLIST subtype: box, width: 20.83pt, depth: 0.11pt, height: 6.57pt, attr: 1=1
    │   │   │ ╚═head:
    │   │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: 1 0 0 rg 1 0 0 RG
    │   │   │   ├─GLYPH subtype: 256, char: m, width: 8.33pt, height: 4.42pt, attr: 1=1
    │   │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: .5 .165 .165 rg .5 .165 .165 RG
    │   │   │   ├─GLYPH subtype: 256, char: i, width: 2.78pt, height: 6.57pt, attr: 1=2
    │   │   │   ├─GLYPH subtype: 256, char: x, width: 5.28pt, height: 4.31pt, attr: 1=2
    │   │   │   └─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=2
    │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: 1 0 0 rg 1 0 0 RG
    │   │   ├─GLYPH subtype: 256, char: d, width: 5.56pt, height: 6.94pt, depth: 0.11pt, attr: 1=1
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt, attr: 1=1
    │   │   ├─HLIST subtype: box, width: 23.36pt, depth: 2.06pt, height: 4.53pt, attr: 1=3
    │   │   │ ╚═head:
    │   │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: 0 1 0 rg 0 1 0 RG
    │   │   │   ├─GLYPH subtype: 256, char: g, width: 5pt, height: 4.53pt, depth: 2.06pt, attr: 1=3
    │   │   │   ├─GLYPH subtype: 256, char: r, width: 3.92pt, height: 4.42pt, attr: 1=3
    │   │   │   ├─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=3
    │   │   │   ├─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=3
    │   │   │   └─GLYPH subtype: 256, char: n, width: 5.56pt, height: 4.42pt, attr: 1=3
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt, attr: 1=1
    │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: 1 0 0 rg 1 0 0 RG
    │   │   ├─GLYPH subtype: 256, char: r, width: 3.92pt, height: 4.42pt, attr: 1=1
    │   │   ├─GLYPH subtype: 256, char: e, width: 4.44pt, height: 4.48pt, depth: 0.11pt, attr: 1=1
    │   │   ├─GLYPH subtype: 256, char: d, width: 5.56pt, height: 6.94pt, depth: 0.11pt, attr: 1=1
    │   │   ├─GLUE subtype: spaceskip, width: 3.33pt, stretch: 1.66pt, shrink: 1.11pt
    │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: 0 g 0 G
    │   │   ├─GLYPH subtype: 256, char: b, width: 5.56pt, height: 6.94pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: l, width: 2.78pt, height: 6.94pt
    │   │   ├─GLYPH subtype: 256, char: a, width: 5pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─GLYPH subtype: 256, char: c, width: 4.44pt, height: 4.48pt, depth: 0.11pt
    │   │   ├─KERN kern: -0.28pt
    │   │   ├─GLYPH subtype: 256, char: k, width: 5.28pt, height: 6.94pt
    │   │   │ ╚═  properties: {['injections'] = {['leftkern'] = -18350.08}}
    │   │   ├─PENALTY subtype: linepenalty, penalty: 10000
    │   │   ├─GLUE subtype: parfillskip, stretch: +1fil
    │   │   └─GLUE subtype: rightskip
    │   ├─GLUE stretch: +1fill
    │   ├─KERN subtype: userkern
    │   └─GLUE
    ├─GLUE subtype: baselineskip, width: 17.34pt
    └─HLIST subtype: box, width: 455.24pt, height: 6.66pt
      ╚═head:
        ├─GLUE stretch: +1fil, shrink: -1fil, shrink_order: 2
        ├─GLYPH subtype: 256, char: 1, width: 5pt, height: 6.66pt
        └─GLUE stretch: +1fil, shrink: -1fil, shrink_order: 2

[olsak]

Thank you for very interesting request. I need more time to thing and test it, because it in not fully backward compatible. But it seem that it brings more advantages than disadvantages. Maybe, several weeks (months?) we will have it as alternative branch...

[vlasakm]

In 8c40acd5dc5c070a515e33dc85a8dca3e98baec2 I moved a lot of things to TeX side. There seems to be no measurable performance penalty. And we can have nice interaction with \global.

This for example works as expected:

{\global\Blue} blue

\_colorprefix could be used to implement "nonlocal/global colors":

\def\_colorprefix{\_global}
{\Red} red

Although I would much rather see \localcolor \nonlocalcolor gone.

I chose to break compatibility with \_setcolor, which wasn't public. This allowed other change in 750a557e37def8487ca533d5f83e3405d8c2a672 - split setting of nonstroke/stroke color.

LuaTeX itself only uses stroke color for rules that have width or height+depth smaller than 1 bp. So in theory, if we change stroke color only for those rules and literals that may use stroke colors, we save a lot of color switching. In practice the calculated "width" and "height" dimensions of rule, used by shipout takes into consideration also text direction and running dimensions (-2^30). I didn't implement the same calculation, therefore there may be unnecessary color switches (a lot of rules have running dimensions). Rule dimensions are often 0.4 pt, which also requires color checks. Still there are a lot of savings, because color changes don't happen that often.

For example for optex-doc.pdf:

Version	optex-doc.pdf size (compressed)	optex-doc.pdf size (uncompressed)
colorstack	1383227	8147005
attributes	1374596	8055603
attributes (lazy stroke)	1366765	7783857

EDIT: Forgot about \onlyrgb / \onlycmyk. What is described above is not usable, yet! EDIT2: Implemented \onlyrgb and onlycmyk in terms of the new \_setcolor. I also added a small macro optimization, so that \_cmyktorgb / \_rgbtocmyk aren't run 4 times at first use of the color.

[vlasakm]

Still haven't implemented \_currentcolor. In my opinion it falls in the same category as \nonlocalcolor. I think that with the right use of grouping neither is necessary.

[vlasakm]

Until you have the time to get back to this (no need to hurry!), this is what I now think of the raised points:

• Do we even want this considering the advantages/disadvantages?
  • Color using attributes has importanat advantages. Not only is it more natural, it also simplifies a few other macros, that had to be specially rigged for \aftergroup. The speed penalty is small. Size improvements are also not insignificant.
• Is there any breakage of existing documents?
  • Any reasonable documents should work. Grey area is the interaction with \pdfliterals and \pdfsave's. Currently they are taken into consideration, i.e. color will change if pdfliteral/pdfsave is encountered and they don't have the same attribute value. This is because PDF literals may contain something colorable, and in situations like \Green\pdfsave\hrule we have to ensure that color is changed before the q operator, else we might have a bad idea about what the current color really is.
• How to handle inserts?
  • Inserts should locally have the \_resetcolor, i.e. they should be by default black, unless requested otherwise. They should not inherit the color active at their creation time. E.g. {\Red footnote in red\fnote{footnote in red}} should produce red text "footnote in red" in the text area, but black "footnote in red" text at footnote area.
• More thorough implementation of leaders?
  • Special handling of leaders will not be supported. Leaders with more colors have to be implemented manually by the user.
• Concept of \Black vs default color?
  • I am still torn about this. One one hand I can write \Grey at the start of the document and have everything colored grey (of course with the exception of page numbers and inserts), so full support of setting custom default color would be nice (and very easy to implement). But the distinction between \Black and default color is still weird. E.g. what is the right behaviour when default color is grey and \_hyperlinks \Black \Black is set? I think not supporting default color other than black is the way to go. (It can still be an OpTeX trick or something user can "easily" do manually, because they can make their own decision on handling black vs default.)
• Move implementation to seperate file? Inline into colors.opm? Better Lua mechanism?
  • At present I am thinking about having something like .opl files for Lua and having colors.opl. But still not sure. In either case, single optex.lua is infeasible.
• \_setcolor in Lua or TeX? What about the old local/global colors concept?
  • Having \_setcolor in TeX allows nice usage of \global\Color. Using something like \let\_colorprefix=\global we can have the old \_nonlocalcolors. Personally I would like to see the concept of \_nonlocalcolors go away, along with all uses of \_localcolor in OpTeX macros (i.e. local colors being unchangable default, with possible more granular \global settings of \global\Color).

[vlasakm]

Still haven't implemented \_currentcolor. In my opinion it falls in the same category as \nonlocalcolor. I think that with the right use of grouping neither is necessary.

I did it in the end, but I still think it may not be necessary along with as with \_currentcolor.

[vlasakm]

We can also hook into luaotfload's handling of colors for the color font feature (made a available with \setfontcolor in OpTeX). Instead of the default behaviour of inserting colorstacks - which would have lower priority than our (later inserted) literals, we can set our own attribute. This has the advantage that there aren't two mechanisms fighting together, but instead working together.

E.g. the following works as expected (sometimes, see below):

{\Blue\setfontcolor{00FF00FF}\currvar green text}

The disadvantage is, that to set this callback luaotfload has to be loaded. First problem is that in OpTeX luaotfload isn't always loaded. And when it is, it has to follow optex.lua, which defines the luatexbase functions needed by luaotfload. For the moment I put the code in fonts-select.opm.

Another problem is, that luaotfload passes RGB string as a request. Currently I map it directly to a corresponding number or 0. But often the color wouldn't have a corresponding number (e.g. the previous example works only if \onlyrgb is set and \Green is used before). Therefore a color allocator in Lua would be needed, and it should cooperate with the TeX one.

Also it may be desirable to handle transparency using attributes as well (or we can let luaotfload do it, but that is just for fonts).

By the way, \_mfontfeatures doesn't include \_ffcolor, which means that this color font features isn't ever applied to math fonts. Even with it, I didn't get colored superscripts though.

[vlasakm]

@olsak Please mark conversations you deem resolved with Resolve conversation.

[vlasakm]

Just to comment on 8365aa4:

While require("something") does the hard job of finding the module and loading it only once (because the table is cached into packe.loaded) it is nevertheless good to save the returned table in a local variable.

Even though for "quick 'n dirty" debugging it would be fine the old way (less typing if one does it manually), I think code in OpTeX tricks should respect "good style" .

[vlasakm]

I finally looked into discretionary handling. At the point of (pre)shipout only 'no-break' (replace) part of discretionary has any meaning. It is essentially unwrapped and inserted instead of the discretionary node by hlist_out. This means that it has to be also colorized. An example that works correctly now, but didn't before:

\begtt \adef!{\Red} \adef?{\Green}
\_def\_topglue {\_nointerlineskip!\_vglue?-\_topskip\_vglue} % for top of page
\endtt

This is simplified example of what actually happens in optex-doc.tex. Because - ends up as a discretionary node with pre = - and replace = -, the color of - nested in replace has to be checked. If it isn't then the minus sign will have the previous (wrong) color.

Or in nodetree diff:

     │   │   ├─WHATSIT subtype: pdf_end_link
     │   │   ├─DISC subtype: automatic, penalty: 50
     │   │   │ ╠═pre:
     │   │   │ ║ └─GLYPH subtype: 256, char: -, width: 4.25pt, height: 2.75pt
     │   │   │ ╚═replace:
+    │   │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: 0 g
     │   │   │   └─GLYPH subtype: 256, char: -, width: 4.25pt, height: 2.75pt
-    │   │   ├─WHATSIT subtype: pdf_literal, mode: 2, data: 0 g
     │   │   ├─GLYPH subtype: 256, char: \, width: 4.25pt, height: 5.55pt, depth: 0.66pt
     │   │   ├─GLYPH subtype: 256, char: _, width: 4.25pt, depth: 1.06pt
     │   │   ├─GLYPH subtype: 256, char: t, width: 4.25pt, height: 4.43pt, depth: 0.04pt

This would have been tought to spot in the PDF. I hope there are no more "bugs" like this one, although discretionaries were always on my TODO list.

@olsak I am going to test color using OpTeX tricks and if they are fine I will mark this Pull Requset as ready. At that point we can also discuss how to merge - squash all commits into one, or something more granular? My first idea is to squash into two commits - implementation of color using attributes and new OpTeX trick.

Also is the functionality of colored fonts from luaotfload to be kept? If we decide to remove it, we can simplify a few things.

[olsak]

is the functionality of colored fonts from luaotfload to be kept?

There is transparency feature in the mentioned functionality. So, it cannot be simply replaced by our colors+arrtibutes. I can imagine that we allocate a next attribute and its value will be transparency*256, for example. But the implementation of transparency at shipout level (and PDF primitive commands) seems to be more complicated. See https://www.cstug.cz/bulletin/pdf/bul_051.pdf page 22 and 23, for example.

[vlasakm]

Yes, with transparencies we are getting into the /ExtGState territory. This brings the problem of managing PDF resources / page attributes (\pdfpageresources, \pdfpageattr). While separate "registers" are available to Lua (which solves clashes with others accessing them, like OpTeX itself or TikZ), it is like the registers well not very well structured (i.e. still one "string") very much like \pdfliteral - opaque text that can (sometimes) mess things up. I don't like that very much and I think bringing more structure would be better here. But that is not a short term solution.

Anothere problem is that with the separation of fill/stroke colors, the colorize functions is now too specialized to support transparency or other things.

I will experiment in the area of more structure, in the meantime luaotfload hook it is.

[vlasakm]

OPmac trick 0085 will cease to work after this PR is merged.

The trick is not itself part of OpTeX tricks, but is linked from OpTeX trick 0044.

This is the diff required to make it work, it boils down to the philosophical difference between colorstacks and attributes - color active at content location vs color active at content creation.

    \def\uline##1{\skip0=##1\advance\skip0 by.05em
       \Bcolor\leaders \vrule\coltextstrut\hskip\skip0 \hskip-.05em\relax}%
    \def\uspace{\fontdimen2\font plus\fontdimen3\font minus\fontdimen4\font}%
-   \def~{\egroup\hbox{\uline{\wd0}\llap{\Tcolor\copy0}}\nobreak{\uline\uspace}\relax \setbox0=\hbox\bgroup}%
+   \def~{\egroup\hbox{\uline{\wd0}\llap{\copy0}}\nobreak{\uline\uspace}\relax \setbox0=\hbox\bgroup\Tcolor}%
    \leavevmode\coltextA #3 {} }}
 \def\coltextA#1 {\ifx^#1^\unskip\unskip\else
    \hyphenprocess{#1}\expandafter\coltextB\listwparts\-\end
 \expandafter\coltextA\fi}
 \def\coltextB#1\-#2\end{\ifx^#2^\coltextC{#1}\else
    \coltextD{#1}\def\next{\coltextB#2\end}\expandafter\next\fi}
-\def\coltextC#1{\setbox0=\hbox{#1}\hbox{\uline{\wd0}\hbox{\llap{\Tcolor\copy0}}}\uline\uspace\relax}
-\def\coltextD#1{\setbox0=\hbox{#1}\hbox{\uline{\wd0}\llap{\Tcolor\copy0}}\-}
+\def\coltextC#1{\setbox0=\hbox{\Tcolor #1}\hbox{\uline{\wd0}\hbox{\llap{\copy0}}}\uline\uspace\relax}
+\def\coltextD#1{\setbox0=\hbox{\Tcolor #1}\hbox{\uline{\wd0}\llap{\copy0}}\-}

I am not sure about the right approach here. Should the relevent OPmac tricks (0085 and its dependency 0065) be copied to OpTeX tricks, now that they are different? What do you think @olsak?

[vlasakm]

I think all relevant OpTeX tricks (with the above exception) and other documents I tried worked fine (at least from what I could tell). I am marking this as "Ready for review" and awaiting your instructions.

[vlasakm]

In the last three commits (409c2d972608e7d76a70ddad007f52a6744b2407, 0314851c3248edc33b95b61a125352915a953752 and bc6482be2a32b0814a8ced669237165a38dd8e58) I added a mechanism for generic pre-shipout injectors. They work similarly to a previous colorize - go through the [hv]lists and insert a PDF literal if attribute changes. Color could be handled by the same mechanism, but then we couldn't use the stroke/nonstroke optimization.

I added also handling of transparency and font outlines. Both use the mechanism and require a straightforward initialization. This is what works now:

\fontfam[lm]

a{\_transparency{.5}b{\_transparency{.2}c}b}a

\Red a{A\_outlinefont{.1}B\pdfliteral{0 1 0 RG}\_outlinefont{.3}B}a

\bye

Of course this is just proof of concept. There is code duplication and other dubious design decisions.

There are a few things to sort out, before this is usable:

• What about page resources? Currently for transparency I expect sole access to \pdfpageresources, which is not true (some parts of OpTeX use it, as well as other packages like TikZ)
• I only use one PDF object for /ExtGState of all pages. This may be be fine, but also depends on previous point.
• Color has a forced reset on every page, that means that initial color change is always inserted (even if color isn't used in the whole document). This is just to support other "default" colors than black. The cost is a few bytes of every pages. This is already questionable, but has been kept for now, because one could achieve the same thing with colorstacks (and they were even more wasteful). But for transparency and font outlines I did not force the initial setting (i.e. default is 0 not -1). Is this the right choice? It means that one can't choose e.g., another transparency as default, but also no wasteful bytes are insersted for each page.
• Another wasted resource is processing power. Most documents won't use these features, but they would still take time to process (iterating over millions of nodes). Should all these pre-shipouts be opt-in? What about color? Opt-out?
• The current \_resetcolor should probabably be \_resetattributes or something like that, and also reset transparency and font outlines.
• The TeX code for this is essentially duplicated. While each case is a little bit different, maybe we should devise a common "allocating function" for these attributes.
• What should the transparency number represent? Right now I am using the direct mapping of \_transparency argument to /ca graphics state value, but it should be the inverse.
• I should check if the used operators can be inserted everywhere in the content stream ("direct literals") like colors, or they require special treatment.
• What is the interaction with images?
• What about leaders?

@olsak I am available if you would like to discuss.

[olsak]

I am ready to discuss about it using an interactive tool (virtual meeting; we will agree on the coordinates by email). The results of such discussion will be put here. My first idea: this extra-color features (transparency, outlines) should be separated from standard color support, so they cannot be in the same commit, in the same files etc.

[vlasakm]

We agreed with @olsak to instead transform the last three commits (generic pre-shipout injector, transparency, font outlines) into OpTeX tricks. And also decided against more thorough management of "ExtGStates".

Playing with it more, I think that OpTeX tricks is a good place for "generic" attribute handling and its applications (transparency and font outlines). They are just examples of what is possible with the new hooks (the tricks would require no change in OpTeX).

But I realized that the ExtGState management I devised is pretty much what PGF/TikZ already does. I think its worth to do this right. (See #60, #61.)

Will transform to OpTeX tricks (in this pull request) according to how #60 and #61 go.

For reference this is the code I used for testing (both cases - with and without TikZ - are of course interesting for testing).

\fontfam[lm]
\load[tikz]
\_addto\_byehook{\_the\_cs{pgfutil@everybye}}

a{\transparency{.5}b{\transparency{.2}c}b}a

{\Red a{A\outlinefont{.1}B\pdfliteral{0 1 0 RG}\outlinefont{.3}B}a}

\inoval[\shadow=3]{abc}

\tikzpicture[line width=1ex]
\draw (0,0) -- (3,1);
\filldraw [fill=yellow!80!black,draw opacity=0.5] (1,0) rectangle (2,1);
\endtikzpicture

\vfil\break
\slides
\slideshow

\sec A

\layers 3
{\pshow2 Second text.} {\pshow3 Third text.} {\pshow1 First text.}
\endlayers

* a \pg+
* b \pg+
* c \pg+
* d

\bye

[olsak]

Will transform to OpTeX tricks

OK. Note that #60 was accepted to the master branch.

[vlasakm]

Sorry, for not yet converting the above into OpTeX trick(s).

Apart from needing to check the PDF spec, there is a problem. The code I previously pushed to this branch benefited from the already present local declarations in optex.lua. Without them the Lua part of the OpTeX trick is a bit lengthy:

local node_id = node.id
local glyph_id = node_id("glyph")
local rule_id = node_id("rule")
local glue_id = node_id("glue")
local hlist_id = node_id("hlist")
local vlist_id = node_id("vlist")
local disc_id = node_id("disc")
local token_getmacro = token.get_macro

local direct = node.direct
local todirect = direct.todirect
local tonode = direct.tonode
local getfield = direct.getfield
local setfield = direct.setfield
local getlist = direct.getlist
local setlist = direct.setlist
local getleader = direct.getleader
local getattribute = direct.get_attribute
local insertbefore = direct.insert_before
local copy = direct.copy
local traverse = direct.traverse

local pdf_base_literal = direct.new("whatsit", "pdf_literal")
setfield(pdf_base_literal, "mode", 2) -- direct mode
local function pdfliteral(str)
    local literal = copy(pdf_base_literal)
    setfield(literal, "data", str)
    return literal
end

local function create_pre_shipout_injector(attribute, default, namespace)
    local current
    local function injector(head)
        for n, id, subtype in traverse(head) do
            if id == hlist_id or id == vlist_id then
                -- nested list, just recurse
                setlist(n, injector(getlist(n)))
            elseif id == disc_id then
                -- only replace part is interesting at this point
                local replace = getfield(n, "replace")
                if replace then
                    setfield(n, "replace", injector(replace))
                end
            elseif id == glyph_id or id == rule_id
                    or (id == glue_id and getleader(n)) then
                local new = getattribute(n, attribute) or 0
                if new ~= current then
                    local literal = token_getmacro(namespace..new)
                    head = insertbefore(head, n, pdfliteral(literal))
                    current = new
                end
            end
        end
        return head
    end

    return function(list)
        current = default
        return tonode(injector(todirect(list)))
    end
end

callback.add_to_callback(
    "pre_shipout_filter",
    create_pre_shipout_injector(registernumber("_transpattr"), 0, "_transp:"),
    "_transp"
)

callback.add_to_callback(
    "pre_shipout_filter",
    create_pre_shipout_injector(registernumber("_fntoutattr"), 0, "_fntout:"),
    "_fntout"
)

While the local assignments can be merged into one line, that doesn't help the readability much.. Still the pdfliteral function is duplicated (or has to be made available from optex.lua).

@olsak, how should I proceed?

[olsak]

I hope that it is possible to create a global function optex.pdfliteral in optex.lua file. This function can be used in the tricks and we need not to copy whole function again. We can create three OpTeX tricks: one with the idea of a general pre-shipout injector (doc+code}. OK, the local declarations will be repeated here. The links to following two tricks can be here as examples of usage of this lua code. The lua code can be presented "as-is" here and the notice can be added: user can create his own file "mycode.lua" and do \directlua{require "mycode"} or all code can be put to the \directlua argument. The next two tricks can show outlines and transparency implementation on the TeX level and using \direclua{callbalck...("pre_shipout_filter",...)}.

[vlasakm]

In the end I changed the interface a little bit. @olsak I am open to your suggestions or improvements. Feel free to delete documentation if excessive or notify me about missing explanations.

As far as I know this Pull request should as a whole more or less work as expected. The exceptions are:

• whatsits aren't colored (we settled on this, but maybe it will not work out well in practice)
• https://github.com/olsak/OpTeX/pull/57#issuecomment-875017540

[vlasakm]

I added rebased (ready to be merged without conflicts) variant of this branch to my fork:

https://github.com/olsak/OpTeX/compare/master...vlasakm:attributecolor-rebased

It is squashed into just 7 functionally distinct and hopefully atomic commits.

@olsak I suggest that instead of eventually merging this Pull request (which would require me force pushing this branch and as far as I can tell deleting the visual and correct history seen above), you merge/rebase manually instead with something like:

git switch master # or git checkout master
git fetch vlasak attributecolor-rebased
git rebase vlasak/attributecolor-rebased

If you merge https://github.com/olsak/OpTeX/pull/62 first, then I will force-push the attributecolor-rebased branch again, so that the above commands would still work.

[vlasakm]

For OpTeX trick 64 I have the following image, but it needs adjustments for your page. You may be better off creating the image from the source below.

\fontfam[lm]

\directlua{
local node_id  = node.id
local glyph_id = node_id("glyph")
local rule_id  = node_id("rule")
local glue_id  = node_id("glue")
local hlist_id = node_id("hlist")
local vlist_id = node_id("vlist")
local disc_id  = node_id("disc")

local direct       = node.direct
local todirect     = direct.todirect
local tonode       = direct.tonode
local getfield     = direct.getfield
local setfield     = direct.setfield
local getlist      = direct.getlist
local setlist      = direct.setlist
local getleader    = direct.getleader
local getattribute = direct.get_attribute
local insertbefore = direct.insert_before
local copy         = direct.copy
local traverse     = direct.traverse

local token_getmacro = token.get_macro

local pdfliteral = optex.directpdfliteral

function register_pre_shipout_injector(name, attribute, namespace, default)
    local current
    local default = default or 0
    local function injector(head)
        for n, id, subtype in traverse(head) do
            if id == hlist_id or id == vlist_id then
                % nested list, just recurse
                setlist(n, injector(getlist(n)))
            elseif id == disc_id then
                % only replace part is interesting at this point
                local replace = getfield(n, "replace")
                if replace then
                    setfield(n, "replace", injector(replace))
                end
            elseif id == glyph_id or id == rule_id
                    or (id == glue_id and getleader(n)) then
                local new = getattribute(n, attribute) or 0
                if new ~= current then
                    local literal = token_getmacro(namespace..new)
                    head = insertbefore(head, n, pdfliteral(literal))
                    current = new
                end
            end
        end
        return head
    end

    callback.add_to_callback("pre_shipout_filter", function(list)
        current = default
        return tonode(injector(todirect(list)))
    end, name)
end
}

\newattribute \transpattr
\newcount \transpcnt \transpcnt=1 % allocations start at 1
\def\transparency#1{\inittransparency \transpattr=
   \ifcsname transp::#1\endcsname \lastnamedcs\relax \else
      \transpcnt
      \sxdef{transp::#1}{\the\transpcnt}%
      \sxdef{transp:\the\transpcnt}{/Tr\the\transpcnt\space gs}%
      \addextgstate{/Tr\the\transpcnt <</ca #1 /CA #1>>}%
      \incr \transpcnt
   \fi
}
\addto\_resetcolor{\transpattr=-"7FFFFFFF }
% Transparency of "1" is the default
\sdef{transp::1}{0}
\sdef{transp:0}{/Tr0 gs}
\def\inittransparency{%
   \addextgstate{/Tr0 <</ca 1 /CA 1>>}%
   \glet\inittransparency=\relax
}
\directlua{
register_pre_shipout_injector("transp", registernumber("transpattr"), "transp:")
}

\newattribute \fntoutattr
\newcount \fntoutcnt \fntoutcnt=1 % allocations start at 1
\def\outlinefont#1{\fntoutattr=
   \ifcsname fntout::#1\endcsname \lastnamedcs\relax \else
      \fntoutcnt
      \sxdef{fntout::#1}{\the\fntoutcnt}%
      \sxdef{fntout:\the\fntoutcnt}{#1 w 1 Tr}%
      \incr \fntoutcnt
   \fi
}
\addto\_resetcolor{\fntoutattr=-"7FFFFFFF }
\sdef{fntout:0}{0 w 0 Tr}
\directlua{
register_pre_shipout_injector("fntout", registernumber("fntoutattr"), "fntout:")
}

\parindent=0mm
\parskip=\bigskipamount
\hsize=80mm

Normal, {\transparency{.5} half transparent, \Red{\transparency{.2}red
and more transparent,} back to half,} back to normal.

Normal, {\outlinefont{.15}outlined}, {\outlinefont{.3}more outlined}.

\bye

[olsak]

If you merge #62 first, then I will force-push...

OK, we start with merging this issue and after that the #62 will be merged.

[olsak]

Please, don't make any new changes in this issue. I will try to merge it using attributecolor-rebased in my computer, then I'll do little changes in a "technical" commit and push it to the github as a new OpTeX/master. This will be done tomorrow.

[vlasakm]

Please, don't make any new changes in this issue. I will try to merge it using attributecolor-rebased in my computer, then I'll do little changes in a "technical" commit and push it to the github as a new OpTeX/master. This will be done tomorrow.

Ok, if this is something that can be incorporated into the 7 remaining commits I can still edit it retroactively.

Note that because you will do a "manual merge/rebase"' then Github won't know that this Pull request should be closed. One of us will have to do it manually (with "Close" not "Merge").

[vlasakm]

Please, don't make any new changes in this issue. I will try to merge it using attributecolor-rebased in my computer, then I'll do little changes in a "technical" commit and push it to the github as a new OpTeX/master. This will be done tomorrow.

@olsak Sorry, but I had to push a bug fix. It shouldn't hurt in this branch, because it won't be merged. I don't know whether you already pulled attributecolor-rebased, for simplicity I created attributecolor-rebased2 that should serve the same purpose. The following should work exactly the same like the above. (git merge may be familiar than rebase, and --ff-only ensures that you don't accidentally start resolving a merge conflict that, in fact, shouldn't be there)

git checkout master
git fetch vlasak attributecolor-rebased2
git merge --ff-only vlasak/attributecolor-rebased2

[olsak]

I used the same code from your attributecolor-rebased and merged it to the master. Thank you for your excelent code.