jtex: Allow for nested files structure

[kai-tub]

Context

Hey, I would like to extract the individual issues discussed in https://github.com/kai-tub/latex-beamer-pure-minimalistic/discussions/75 for clarity and easier communication.

As discussed in the thread, I am more than happy to try to give my template first-class support for mystjs/jtex. As part of the first trial, I would strongly prefer not to be forced to have a flat file hierarchy when creating the template. My workflow would be as follows:

• Create the template.tex and template.yml files at the root of my template repository
• Create a git submodule of my actual beamter repository, as this guarantees to have the source fixed and keep the myst-interface separate from the beamer template itself

My reasoning is as follows:

• My CI/CD pipeline for the beamer presentation would have to be updated to exclude changes to the template.tex to trigger updates/releases/tests and I don't think that this is helpful because the repository contains the theme and I think a template for something other users aren't interested in using, just adds noise and additional complexity that isn't necessary
• I am the maintainer of the beamer presentation theme and would also be the maintainer of the template, but I don't think that it is reasonable to assume that this will be the case for all other nice-looking beamer themes.
  • Of course, one can always fork the original theme and inject the template code and maintain that separately but it would duplicate code that could easily become out-of-sync and produce confusing results, as using the theme "directly" might produce different outputs than the template itself if the sources are different
  • This would also happen with my suggestion, but then it would be clear that the template just needs to point to the newest commit of the theme
• It makes it easier to keep separate changelogs/releases for both components
  • If I update the theme interface to support more options, I would need an additional way of communicating these changes only to the template users
  • Last practical reason: In my theme, I 'utilize' folders to keep logos in a separate directory. So I would already need support for folders to correctly render my template. This would also be a change I wouldn't be happy to do, because it adds noise to the folder structure and might be confusing for future template creators

Proposal

Allow the template.tex and template.yml files to be in a separate directory from the other theme components. Allow files to be in separate directories to not restrict the 'expected' structure of themes.

Please let me know if I should further clarify my points :)

Tasks and updates

No response

[welcome[bot]]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:

[fwkoch]

Hey @kai-tub - thanks for breaking this out into a bite-sized issue!

I agree that the same-directory requirement is arbitrary and unnecessarily restrictive. (Going through all your specific use-cases is great!) For now, I think this specifically applies only to the template.yml#files field? For example template.yml#thumbnail has no same-directory restriction.

A few minor points to work through:

• How do we handle file paths in a os-agnostic way?
  • Probably just use / separator in template.yml which jtex resolves into system path.
• Do we allow for including entire folders? or even wildcard patters?
  • Seems easiest to just require files to be files (not folders, not wildcards)? We can always expand support if this becomes a burden.
• Do we need to allow template.tex in any folder?
  • Sounds like this is not required. For now, continue to require template.yml and template.tex at the template repository root.

Also worth noting we don't even respect these files when we copy template files: https://github.com/executablebooks/mystjs/blob/main/packages/jtex/src/jtex.ts#L244 - that just looks in the single template directory. As a part of this work, we need to change that to use files.

[rowanc1]

Thanks @kai-tub for splitting this out of the discussion, very helpful! :)

Re @fwkoch's comments:

Probably just use / separator in template.yml which jtex resolves into system path.

I think we should probably even validate on unix-style path separators in the template.yml and then remap these on any client.

Do we allow for including entire folders? [glob patterns]

Agree that we can probably kick this down the road, my guess is that we could accelerate this with a jtex check command or something to explicitly add and check that the files exist.

[kai-tub]

Thanks for the quick responses! I will just throw in a couple of comments from my side.

For now, I think this specifically applies only to the template.yml#files field?

Yes.

I think we should probably even validate on unix-style path separators in the template.yml and then remap these on any client.

Yeah, I would 'expect' the UNIX-style path separator to be the most appropriate. Also, because this is the way Python (pathlib) and LaTeX expect 'path-strings' to be formatted.

Do we allow for including entire folders? [glob patterns]

I agree that this isn't as important right now. Themes and other LaTeX files usually do not have that many files flying around. But if there were glob pattern support, I would also need a way to exclude specific files that only exist for "documentation" purposes. And by that time, I might've just as well listed all files explicitly. So my vote is for: I personally do not expect glob patterns to exist and would put it at the bottom of the priority list.

Agree that we can probably kick this down the road, my guess is that we could accelerate this with a jtex check command or something to explicitly add and check that the files exist.

Sounds reasonable to me. Maybe there is a way to nicely present all files that jtex would include in a tree structure? Something like jtex check --files/tree or similar?

Do we need to allow template.tex in any folder?

Also agree that root directory for template.tex/yml makes sense :)

But just for clarification, I would still be able to select a file as:

- logos/logo.png

and then expect the file logo.png to be inside of a logos directory, correct?

[fwkoch]

@kai-tub - I took a pass at this with #89 - it's now deployed to NPM. Keen for you to give it a try and see if it helps out (or if you run into any problems)!

Closing the issue for now, but feel free to re-open if you need!

[kai-tub]

Yes @fwkoch, thanks! It works! :+1:

I've added everything to my template:

• https://github.com/kai-tub/latex_beamer_pure_minimalistic_mystjs

But something that I forgot was that Beamer usually 'expects' local themes to live on the same "hierarchy" level as the sty files.

• https://tex.stackexchange.com/questions/123700/how-to-use-local-beamer-theme-in-project-source-folder

So when I configure my template to use a subdirectory/submodule, then I need to be aware of this Beamer limitation and change my template.tex file to not use the \usetheme command but the \usepackage command with the correctly resolved name.

IMHO I still very much prefer to have the 'nested-file' option and manage it as a submodule. Even if this means that I need to deviate from the classic code.

Another side-effect of having the files at different hierarchy levels is that the default pictures/logos are relative to the rendered .tex file (which I also forgot, sorry :disappointed: ! ). So I had to manually duplicate the logo files to my template root. Which I do not have anything against! I have two reasons:

1. I don't really mind having to duplicate this type of "dead" data, as I don't care what the logo looks like, and if it becomes "out-of-sync" with my "upstream" theme repository (I would not feel the same for the sty/tex files)
2. I expect the end-user to use a custom logo either way, so they would have to manage this either way

Both of these things maybe do open up the discussion about the implementation, though:

• Maybe it would be simpler to just allow to copy the contents of the directory to the root directory of the rendered output?
  • So if I select the directory external_lib/ from within the files section, its contents external_lib/theme.sty would be injected to the work-dir as theme.sty (instead of /external_lib/theme.sty?)
  • While keeping the nested directories the same?

Though I have a very specific workflow in mind (submodule for true source, adding template.yml/tex in other repo and calling it a day), maybe I am a bit biased towards the 'injection' solution? As I wrote above, there are solutions to work around the different hierarchy levels, so it is not a blocker, in my opinion.

I just wanted to give you my feedback :)

Sorry again for not seeing these issues/limitations before you implemented the feature :/

[kai-tub]

Yeah... So I looked a bit further into the issues I had when trying to get my docker environment (described in #97) running with a minimal environment, and I got bitten by the fact that an old version of my package exists on CTAN and that it was hiding an error by using files from installed packages... The issue is that when customizing a theme, one usually also customizes the font/color/inner/outer parts of the theme. And those are also resolved relative to the main tex file (template.tex) and not relative to the root sty file that describes the theme (when doing a local look-up).

So in my case, I would have to apply the following patch to the source of my beamer theme sty file if I want to keep the theme files inside a directory called latex-beamer-pure-minimalistic:

diff --git a/beamerthemepureminimalistic.sty b/beamerthemepureminimalistic.sty
index 87183b8..ffbe454 100644
--- a/beamerthemepureminimalistic.sty
+++ b/beamerthemepureminimalistic.sty
@@ -19,7 +19,7 @@
 % If problems/bugs are found or enhancements are desired, please contact
 % me over: https://github.com/kai-tub/latex-beamer-pure-minimalistic

-\ProvidesPackage{beamerthemepureminimalistic}[v2.0.3]
+\ProvidesPackage{latex-beamer-pure-minimalistic/beamerthemepureminimalistic}[v2.0.3]

 \mode<presentation>

@@ -33,10 +33,10 @@
 \ProcessOptionsBeamer

 % Settings
-\usefonttheme{pureminimalistic}
-\usecolortheme{pureminimalistic}
-\useoutertheme{pureminimalistic}
-\useinnertheme{pureminimalistic}
+\usepackage{latex-beamer-pure-minimalistic/beamerfontthemepureminimalistic}
+\usepackage{latex-beamer-pure-minimalistic/beamercolorthemepureminimalistic}
+\usepackage{latex-beamer-pure-minimalistic/beamerouterthemepureminimalistic}
+\usepackage{latex-beamer-pure-minimalistic/beamerinnerthemepureminimalistic}

 \setbeamertemplate{navigation symbols}{}%
 \beamerdefaultoverlayspecification{}

I wasn't aware of this issue because, in my quick tests, LaTeX simply looked up the theme files from my version from CTAN and used those instead of the files that I had locally...

So when I create a separate folder, I would still be required to touch the original theme repository, which I wanted to avoid by requesting this feature... I am really sorry that I wasn't aware of this Beamer limitation and that it is such a convoluted issue. :/

For further reference, one can also look into patching beamer code as described here:

• https://tex.stackexchange.com/questions/275600/beamer-themes-on-custom-folder

but that also seems like a hack rather than a solution.

The nested file structure feature is still useful for template authors (I think) when they want to include local files like figures or similar relative to the template.tex file. But it seems to have no benefit for my suggested workflow with a git submodule.

So I guess the correct request for my submodule workflow would be to have an option that allows me to:

• define a base directory to which the template.tex/yml and the other files from files directory are copied to or
• allow to include the entire contents of a directory as-is but not copying the directory itself while keeping the remaining folder structure as is
  • question: what happens with file collisions?

If I should clarify anything, please let me know! And sorry again for potentially leading you down the wrong path.