Multiple columns (or other layout variables) in Contents section

[gwyndaf]

I'm not sure if this is an enhancement, or my own ignorance!

Is there (or could there be) a way to create Contents pages with multiple columns, even if the default layout for (most) songs is a single column?

If it doesn't exist, a basic approach might be a variable in the relevant contents entry that takes a numeric value.

Or, more flexible, might be something like a preamble value for each Contents section, which could include multiple directives separated by \n. That might allow more subtle adjustment of Contents layout:

"contents" : [
  { "fields"   : [ "sorttitle" ],
    "label"    : "Contents at a Glance",
    "preamble" : "{columns: 2}\n{textsize: 70%}\n{+pdf.spacing.toc 0.9}"
  }
]

That would allow one Contents section to have two columns, smaller type and tighter spacing, while still allowing other Contents sections to use defaults, or their own custom layout/type settings.

I don't know if that's a workable approach, but it seems quite intuitive to follow similar syntax to adjusting the layout in individual song files.

This is mostly a 'nice to have' as my songbook gets larger. If it was essential, I could probably set my default config to have two columns, then manually add {columns: 1} to nearly every song! :)

[sciurius]

I love your ideas!

You can configure the size (pdf.fonts.toc) and spacing (pdf.spacing.toc) already in the config. These are toc only and won't hurt other songs.

As for the other settings... If you can set columns, you can do all kinds of other naughty things as well☺.

You can see that this is not a succesful approach without some other tweaks. E.g. a separator between the columns (vertical line?) and leading between the song title and the page number.

I give you 6.042_008 to play with.

{   "contents" : [
        { "fields"   : [ "songindex" ],
          "label"    : "Table of Contents",
          "line"     : "%{title}",
          "pageno"   : "%{page}",
          "fold"     : false,
          "omit"     : false,
          "preamble" : [
          "{title: Fancy ToC}",
          "{columns:3}",
          "{tocsize:90%}",
          "{+pdf.spacing.toc 1}",
          "{image spread=20 scale=80% src=https://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Dublin_-_Molly_Malone.jpg/270px-Dublin_-_Molly_Malone.jpg}",
      ],
        },
    ],
}

FYI, you cannot add {+pdf.columnspace:60}. This can not be set from the song (it's too late).

[gwyndaf]

Wow, that's fantastic, thanks Johan!

I'd expected that pdf.columnspace might not be changeable, but that's fine. Lines don't wrap, so it only affects where each column starts on the page, and still allows long entries to run across the column gap.

For similar reasons, I'm personally happy without a vertical divider line. Each (left-aligned) column is clearly defined by its left edge, and there's no problem with long ToC entries running across the divider line, which could look messier.

If long titles run into the next column, I think I can probably just add custom {toc_title} metadata to the relevant songs, and use that for shorter alternative ToC line entries (like earlier suggestion for extending new_song functionality).

Page numbers and leading also aren't an issue for me, as this is largely for use on devices, so hyperlinking is the important feature.

Observations {title} in preamble seems to duplicate (and take precedence over) label, but that seems like a logical application of the general principle that preamble could include many song-related directives.

That led me to wonder if I could set a {subtitle: } value in preamble, which would then be output on the subtitle line below the Contents title.

Seems that isn't possible, though. A fixed string in pdf.formats.first.subtitle outputs successfully on the subtitle line, and so does %{title}, but other metadata variables don't seem to. Perhaps that's because PDF layout is done later, so maybe it's a built-in limitation.

Not a big issue, though. That's simply exploring how far I can push the feature's potential! Simply getting all songs on a single Contents page is already very useful. :)

[sciurius]

6.042_010 has subtitle and meta. However, there is no context (no song) for metadata. Title and subtitle use the data from the first song in the list.

and there's no problem with long ToC entries running across the divider line,

Long toc entries will wrap, so no problem here.

[gwyndaf]

Nice. Subtitle now appears as I'd hoped it might :)

Long ToC entries (i.e. longer than column width) don't appear to wrap with multiple columns, but that's what I expected because I don't get line-wrapping on song pages with multiple columns.

In fact, I find that if I force a really long title, that entry doesn't wrap even in a single-column ToC. That's not a problem for my usage, though. I've got markup in my ToC entries, and know that gets problematic with line wrapping.

[sciurius]

Hmm. One extra incentive to rewrite that piece of code.

6.042_011 truncates the line if too long. Lousy, but better than nothing.

[gwyndaf]

That looks better on the page, and the ellipsis is a nice touch.

It throws up some error messages where the truncation splits a line in the middle of markup, but the PDF output looks as expected. That's a useful nudge to add some alternate short ToC titles to the relevant songs.

However, I also get another error, but no obvious effect on PDF output:

Use of uninitialized value in join or string at /usr/local/share/perl/5.36.0/String/Interpolate/Named.pm line 347.

My only observation right now is that it occurs with all instrument editions of my songbook, and the only case where it doesn't arise is the 'lyrics only' edition, which differs mainly in not having chords or annotations, but also some different preprocessor stuff.

I'll have to come back to this later in the week, when I've time to narrow down what might be causing it.

[sciurius]

It throws up some error messages where the truncation splits a line in the middle of markup,

Yes, it's a quick hack :). Do we want markup in toc lines? I think I might as well just strip the markup.

Use of uninitialized value in join or string at

Is this a 6.042_+ issue?

[gwyndaf]

Personally, I'd rather live with the markup errors than have markup stripped out of all ToC entries. I can probably avoid the errors by using shorter alternative titles, but markup's very useful for making long entries clearer: I appreciate that it's a quick (and very useful) hack, so wouldn't want it to use your time too much. A compromise might be to strip markup only from lines that are getting truncated, but perhaps that's more difficult because line length isn't known until markup is processed. With similar markup errors elsewhere, I've simply treated them as a reminder for me to shorten the lines!

Yes, I'm pretty sure the uninitialized value errors are new with 6.042_011. I'm busy today, but will try to narrow down where they occur in the next day or two.

[sciurius]

6.042_012 will truncate if necessary, and not remove markup.

[sciurius]

6.042_014 should do a reasonable job wrapping toc lines.

[gwyndaf]

Oh wow (6.042_014): you've wrapped lines with markup! If I recall similar cases previously, that's a pretty major challenge.

It looks like an extra line break might have crept in after each entry:

I don't know if that's by accident or by design, but makes it harder to control the layout, such as compact (and therefore fewer) Contents pages for larger songbooks. Although the space after helps to separate entries if wrapping happens, that already seems clear enough because the wrapped portion is indented.

I narrowed down the _011 uninitialized value errors to probably the Contents page, at which point it seemed sensible to pull _014, and they're now gone.

[sciurius]

I did some more experimenting and wonder what would be the best and intuitive approach.

First, the 'template' approach by providing a mini-song that basically gets prepended before the ToC. In the mini-song you can set title, subtitle, columns and columnspace (using {+pdf.columnspace 60} (this now works)). You can also do fun things like (spread) images, even a song on the (first) ToC page.

Second is to add some settings to the "contents", like title (instead of label), subtitle, columns and columnspace.

The 'contents' approach is more intuitive but limited. The 'template' alternative is more powerful but easier to screw up. What do you think?

[sciurius]

It looks like an extra line break might have crept in after each entry:

Oops. Fixed in 6.042_015.

[gwyndaf]

Space between entries now gone :)

Visually, at least. I think that extra line after might still be in the hyperlink hotspots. With mouse pointer over '9 to 5' (using muPDF reader): It looks like each title's hotspot also includes a line after, so the darker highlighting shows the overlap of the '9 to 5' hotspot and the '500 Miles' hotspot (i.e. extra line after).

Maybe not a major issue, but perhaps overlapping hotspots might cause unexpected effects with some PDF readers.

[gwyndaf]

The 'contents' approach is more intuitive but limited. The 'template' alternative is more powerful but easier to screw up. What do you think?

OK, I'm personally biased toward more control at the risk of screwing up. But stepping back a bit...

I don't know the broad spread of ChordPro usage, but would guess that some users simply use the built-in configs, and limit themselves to controlling song layout with directives in song files, so I think that directives used in songs should be (remain) simple and intuitive.

But, maybe it's reasonable to assume that anyone editing or creating their own config files accepts that they take on greater power, but also more scope to screw things up?

Since the "contents" settings are in the config file, perhaps that suggests that anyone editing them accepts the risks, so the more powerful 'template' approach would be more appropriate.

The risks could be mitigated by adding a new (disabled) section in "contents", which illustrates the 'template' approach with settings for title, subtitle, columns, columnspace (and maybe some fun stuff, commented out). That way, it's easy to change omit from true to false and get a 2-column ToC without much effort or risk, and maybe gives an easy first step to further tweaks.

I think your term 'template' is intuitively clearer, as it implies something song-like and familiar. My use of 'preamble' could echo ABC/Lilypond delegate settings, which are maybe more unfamiliar and scary.

[sciurius]

Visually, at least. I think that extra line after might still be in the hyperlink hotspots.

Fixed in _016. Nice feature of mupdf to show the link area. I should really start using it.

[gwyndaf]

Great. Hotspots now more sensible with _016.

mudpf's very fast to start up, so it's my default PDF viewer (on desktop and tablet), but quite minimal so I still need others for some features. I've compiled it myself for Linux, because the Debian packaged version lacks Unicode support, and was giving me some issues with the OpenGL version.

[gwyndaf]

Another interesting issue. I have a landscape/on-screen volume, using settings.columns: 2 in the config, so all songs flow over two columns. Previously, that's automatically put the standard ToC into two columns as well, but now it seems to be one-column: The above uses an existing, unchanged "contents" section, with no new features to set columns.

For comparison, a 3-column ToC with the same config, but using new column features, outputs as expected.

It looks like "contents" might now be assuming columns=1 by default, rather than inheriting the prevailing settings.columns value, which I think would be more intuitive (and greater continuity).

[sciurius]

I think your term 'template' is intuitively clearer, as it implies something song-like and familiar

Whatever alternative, it always involve a tailored config so the user should have some basic familiarity with creating/editing config files.

I think this is the 'best of both worlds': In the config, at the "contents", specify a template:

    "contents" : [
        { "fields"   : [ "songindex" ],
          ...
          "template" : "toc.cho",
        },
    ],

This template is processed as an ordinary song, and will have the actual table of contents appended.

Since it is an ordinary song, it will pick up a song-specific config toc.json if this exists. So in the end you have the simplicity of song directives and the powers of config.

The template file will be looked up next to the first song of the list of songs, and in the template directories in the config path.

A complicating factor is that you need to have a 'global' config to define the template file name for the ToC, and a song-specific config to configure the ToC.

Does that sound like a decent solution? You can try it in _017.

(It also addresses https://github.com/ChordPro/chordpro/issues/21#issuecomment-341926008.)

[sciurius]

It looks like "contents" might now be assuming columns=1 by default, rather than inheriting the prevailing settings.columns value

I think _017 addresses this as well. Note that preamble is gone now.

[gwyndaf]

That seems a great approach. I was coming to think that many potential ToC features come down to 'behaving like a regular song page', so having a `.cho' file for ToC layout seems a great approach.

I'll play with _017 and report back...

[gwyndaf]

Default ToC columns now seem to reflect the global config setting (unless over-ridden by local setting).

One observation so far: automatic column-breaking for multi-column ToC seems to ignore wrapped portions of entries at the bottom of the page, so they overflow into the footer area. Regular, single-line entries look ok and break to a new column as expected.

[gwyndaf]

New functionality is looking good, and makes it simple to include, for instance, a conditional 4-column layout for landscape/screen use:

{title: Contents at a Glance}
{subtitle: (full contents on next pages)}
{columns: 3}
{tocsize: 77%}
#{textsize: 77%}
{+pdf.spacing.toc 1.33}
{start_of_settings-screen}
  {columns: 4}
  {tocsize: 83%}
  {+pdf.spacing.toc 1.23}
{end_of_settings}

One interesting observation: I was getting a blank song page (after Contents sections), which I realised corresponded to the .cho ToC template, because my production script finds all the .cho files in a folder and uses them for --filelist. This is a situation of my own making, and I can easily work around it by using .cht as the template extension. I don't know if there's merit having ChordPro ignore any 'song' files that have already been specified as ToC templates, or just leave that in users' hands.

---

I'm wondering if it's necessary to add {tocsize} as a new directive, given the template approach. On the basis that ToC entries could be seen as 'song' content, would it be simpler and clearer (from user perspective) to use the existing {textsize} directive? That would (re-)use an existing familiar directive, even if it effectively sets the value of pdf.fonts.toc (which experienced config tweakers could set directly)?

I could see that having separate {tocsize} allows existing/default {textsize} to be used for preliminary text before the ToC entries, but that could also be covered by having successive {textsize} directives, with the last value determining ToC text. On the other hand, adding a new directive might increase (perceived) complexity for less experienced users.

---

If the template approach works out ok, a highly intuitive way of using it might that the default "contents" config entry could include a template value with a predefined .cho (or .cht) filename: if the file exists, it would be used as a template, and otherwise ignored (I guess that principle could apply to any template value). That could offer easy ToC customisation with song-like directives for basic users, while still allowing flexibility for users who are happy editing configs.

I'd suggest maybe a filename something like 0001_toc would be useful, so it appears (probably) first in directory listings, to reflect the order of pages in a book, with ToC coming before songs.

[sciurius]

One observation so far: automatic column-breaking for multi-column ToC seems to ignore wrapped portions of entries at the bottom of the page,

This should occur only when the very last line of the column needs wrapping. Page (column) space is checked before processing the toc item, assuming it will require only a single line. I need to restructure a few things before I can fix this.

[sciurius]

One interesting observation: I was getting a blank song page [...] don't know if there's merit having ChordPro ignore any 'song' files that have already been specified as ToC templates, or just leave that in users' hands.

I think this will happen once and then you get the workflow right. For example, like you did, by chosing a different extension or location for the templates.

I'm wondering if it's necessary to add {tocsize} as a new directive, given the template approach.

tocsize and friends have been around since 5.0.

On the basis that ToC entries could be seen as 'song' content,

Yes, but that doesn't imply that the ToC lines are text (song) lines. I don't think users will have problems dealing with song texts and Toc lines as separate things. Besides:

That would (re-)use an existing familiar directive, even if it effectively sets the value of pdf.fonts.toc

That would require some trickery I'm not looking forward to.

If the template approach works out ok, a highly intuitive way of using it might that the default "contents" config entry could include a template value with a predefined .cho (or .cht) filename: if the file exists, it would be used as a template, and otherwise ignored

Yes, I was thinking in the same direction. I'm not sure how to name this default template. Maybe the default name should only be looked up in the templates directories (that, I assume, currently noone has).

I think the approach is generally ok, just some fine tuning.

[gwyndaf]

Sounds good.

It's probably a rare case that the last line in a column is also wrapped, and the consequences are minor. Personally, I'm now adding a {title_short} to songs with long titles, and using it as first choice in my 'compact' ToC, so I'm unlikely to encounter line-wrapping with multiple columns.

Sorry, I thought I'd checked whether {tocsize} already exists in documentation, but seem to have missed it, so that idea's a bit redundant!

---

Having a possible default ToC template in the templates folder seems a good idea (better than amongst song files), so the same settings can work with different collections of songs. In that case, its name probably matters less, as long as it's meaningful. I suppose contents.cho or toc.cho might be the most obvious names for the default file, unless there's a good reason against those.

There still seems merit in looking for custom templates alongside the first song (as well as in templates), which could provide an easy way of setting ToC layouts for specific collections of songs.

I can see that looking for the default ToC template only in templates would avoid a clash with a song file of the same name (unlikely), but how might that work? If the default config includes a "contents" line of "template" : "contents.cho", but that value gets user-edited to "mycustomtoc.cho", how would ChordPro know to look for one (default) filename only in templates and the other in both templates and song location, when all that's changed is a config value?

[sciurius]

For example, toc and templates:toc will search for toc.cht in the templates directories only. toc.cht will search next to the 1st song and in the templates directories.

This is similar to --config foo (foo.json in the config directries) and --config foo.json (foo.json here or in the config directories).

[sciurius]

I have finalized the toc templace handling. I settled for a .cho extension since it is a song anyway.

Please try 6.042_021.

https://chordpro.org/beta/chordpro-configuration-generic/#customizing-the-table-of-contents

[gwyndaf]

Excellent. Production songbook and test samples still working nicely with no new issues arising, but I'll play about a bit further and let you know if I break anything.