search

latex3 / l3build

A testing and building system for LaTeX
LaTeX Project Public License v1.3c
89 stars 17 forks source link

Basic YAML configuration for basic uses #280

Closed projetmbc closed 10 months ago

projetmbc commented 1 year ago

It might be practical to allow the use of a build.yaml file for situations where no Lua code needs to be implemented to test a project. This would require the use of a parseyaml function, for example.

The idea is not to make a full YAML parser, but just one that supports the use cases proposed by l3build.

On the usage side, if a build.lua file is present, it is launched, and it is the coder's responsibility to parse a possible build.yaml via the parseyaml function which would take the path of the YAML file as argument.

If you can provide me with concrete examples of basic configurations, I'm willing to get my hands dirty. That said, I'm pretty busy until June.

josephwright commented 1 year ago

Could you outline the advantage of YAML here over a simple Lua file? At the basic level, Lua for config. vars has the advantage that it reads very clearly.

projetmbc commented 1 year ago

The advantage of the YAML syntax is that you have fewer characters to type, and I find it very readable. If I take the example at the beginning of this file, this is what you would type for the configuration part.

module: l3kernel
bundle:

# Location of main directory: use Unix-style path separators
maindir: ..
docfiledir: ./doc

# Non-standard settings
checkconfigs:
  - build
  - config-backend
  - config-l3doc
  - config-plain

checkfiles:
  - l3names.def

# If we don't use strictly the YAML grammar, we can avoid 
# the use of the double quotes here.
cleanfiles:
  - "*.fmt"
  - "*.log"
  - "*.pdf"
  - "*.zip"

docfiles:
  - source3body.tex
  - l3prefixes.csv

installfiles:
  - l3dvipdfmx.def
  - l3dvips.def
  - l3dvisvgm.def
  - l3pdfmode.def
  - l3xdvipdfmx.def
  - l3str-enc-*.def
  - l3debug.def
  - l3sys.def
  - expl3.lua","expl3.ltx
  - "*.cls"
  - "*.sty"
  - "*.tex"

sourcefiles:
  - "*.dtx"
  - "*.ins"

tagfiles:
  - "*.dtx"
  - CHANGELOG.md
  - README.md
  - interface3.tex
  - l3styleguide.tex
  - l3syntax-changes.tex
  - l3term-glossary.tex
  - source3.tex
  - "*.ins"

typesetfiles:
  - expl3.dtx
  - l3docstrip.dtx
  - l3doc.dtx
  - interface3.tex
  - l3syntax-changes.tex
  - l3styleguide.tex
  - l3term-glossary.tex
  - source3.tex
  - l3prefixes.tex
  - l3news*.tex

typesetskipfiles:
  - source3-body.tex

typesetruns: 3
unpackfiles:
  - l3.ins

checkdeps:
  - maindir .. "/l3backend"

One can even imagine proposing a more organized structure like the following.

module: l3kernel
bundle:

# Location of main directory: use Unix-style path separators
dir:
  main: ..
  docfile: ./doc

# Non-standard settings
checkconfigs:
  - build
  - config-backend
  - config-l3doc
  - config-plain

files:
  check:
    - l3names.def

  clean:
    - "*.fmt"
    - "*.log"
    - "*.pdf"
    - "*.zip"

  doc:
    - source3body.tex
    - l3prefixes.csv

  install:
    - l3dvipdfmx.def
    - l3dvips.def
    - l3dvisvgm.def
    - l3pdfmode.def
    - l3xdvipdfmx.def
    - l3str-enc-*.def
    - l3debug.def
    - l3sys.def
    - expl3.lua","expl3.ltx
    - "*.cls"
    - "*.sty"
    - "*.tex"

  source:
    - "*.dtx"
    - "*.ins"

  tag:
    - "*.dtx"
    - CHANGELOG.md
    - README.md
    - interface3.tex
    - l3styleguide.tex
    - l3syntax-changes.tex
    - l3term-glossary.tex
    - source3.tex
    - "*.ins"

  unpack:
    - l3.ins

typeset:
  files:
    - expl3.dtx
    - l3docstrip.dtx
    - l3doc.dtx
    - interface3.tex
    - l3syntax-changes.tex
    - l3styleguide.tex
    - l3term-glossary.tex
    - source3.tex
    - l3prefixes.tex
    - l3news*.tex

  skip:
    - source3-body.tex

  runs: 3

checkdeps:
  - maindir .. "/l3backend"
davidcarlisle commented 1 year ago

If I take the example at the beginning of this file...

I think that demonstrates why I'm slightly opposed to the suggestion.

The simple parts are not much simpler than just using Lua and for anything except the most trivial parts you need Lua anyway, so in that file function update_tag(file,content,tagname,tagdate)....

It would complicate the documentation having to document yaml and lua config formats.

projetmbc commented 1 year ago

Can you give me several examples online, more or less sophisticated? I like to factorize APIs and codes...

When I propose a more organized structure, if it is not used in the Lua code, I will not accept to use it only in the YAML file.

My idea is just to imitate the Lua table of configurations. Finally, the documentation wouldn't have much to say about the YAML format. Today, YAML is a well known dialect: it is used with Docker, or for GitHub actions for example.

projetmbc commented 1 year ago

Another point.

I find it easier to maintain a code where the logic is in one place, like in the function update_tag(file,content,tagname,tagdate) function, and the configuration in another, like a YAML external file.

davidcarlisle commented 1 year ago

Can you give me several examples online, more or less sophisticated?

https://github.com/search?p=3&q=filename%3Abuild.lua+org%3Alatex3&type=Code

has several examples

u-fischer commented 1 year ago

The advantage of the YAML syntax is that you have fewer characters to type, and I find it very readable.

YAML may be readable. But it is nevertheless a second syntax. Nearly all build.lua I use have a lua function to handle the tagging. Some have additional lua code to change the test setup depending on the texlive version, or to load additional data for the ctan upload. I would therefore find it confusing to have to maintain a mix of lua and yaml syntax.

josephwright commented 1 year ago

I thought the same as @davidcarlisle but I can see some people might want to use a 'lighter' setup. So if you want to make a PR that works, I'll look at it - but I don't really want to work on this myself.

projetmbc commented 1 year ago

That's what I wanted to do. See in my initial comment : "If you can provide me with concrete examples of basic configurations, I'm willing to get my hands dirty. That said, I'm pretty busy until June.".

projetmbc commented 1 year ago

By testing the link proposed by davidcarlisle via a small Python script, unless I am mistaken, I obtained the results given below which show a majority use limited to configuration. The use of dofile has been ignored, as this indicates a desire to code some logic outside the build.lua file.

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3packages/xfp/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3experimental/xcoffins/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3ldb/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3htoks/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3str-format-new/build.lua

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Bundle-Tree/Module-Two/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3packages/l3doc/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3bigint/build.lua

- https://raw.githubusercontent.com/latex3/iftex/c3d739f74d059a312e0f3b5856754147e09740ed/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3experimental/l3str/build.lua

- https://raw.githubusercontent.com/latex3/mathtools/c81bd201ec20088174c65bec58a5ca563fc0862f/build.lua

+ https://raw.githubusercontent.com/latex3/latex3/25cce2f4e6aaca5e1956cde6f48a6d225106986f/l3kernel/build.lua
    > (function)  update_tag(file,content,tagname,tagdate)
    > (function)  cmdcheck()

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/xbox/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3packages/build.lua

- https://raw.githubusercontent.com/latex3/latex2e/b5ed870b95fbb08206397e2ec3bef47f60c698b0/required/graphics/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3packages/l3doc/examples/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3experimental/l3sys-shell/build.lua

- https://raw.githubusercontent.com/latex3/graphics-cfg/19d1238af17df376cd46333b229579b0f7f3a41f/build.lua

- https://raw.githubusercontent.com/latex3/latex2e/21e6961fb7b2b77a9e2e24c23c05c8eb1c322e0c/required/latex-lab/build.lua

+ https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3backend/build.lua
    > (function)  update_tag(file,content,tagname,tagdate)

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3packages/xparse/build.lua

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Bundle-Flat/Module-Two/build.lua

+ https://raw.githubusercontent.com/latex3/pdfresources/97105e2b5c3e0b88d8b57c7a5149fe5440efc8e0/build.lua
    > (function)  update_tag (file,content,tagname,tagdate)

- https://raw.githubusercontent.com/latex3/xxetex/3c56f3dfdb88e60fcd11c222ad2453cdbb713a28/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3trace/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3tree/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3experimental/l3benchmark/build.lua

- https://raw.githubusercontent.com/latex3/xpdftex/d3e59599ed080f21395bbd4869760d78e24cb34a/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3kernel-extras/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3experimental/l3bitset/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3fp-extras/build.lua

- https://raw.githubusercontent.com/latex3/latex2e/ac5dd70853491e32f004a484f9cd53405fd2db76/required/firstaid/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3check/build.lua

+ https://raw.githubusercontent.com/latex3/unicode-data/8a9133b46fc19de5cc890120bc8db6681743ee83/build.lua
    > (function)  update_tag(file,content,tagname,tagdate)
    > (function)  tag_hook(tagname)

+ https://raw.githubusercontent.com/latex3/latex2e/b5ed870b95fbb08206397e2ec3bef47f60c698b0/base/build.lua
    > (function)  bundleunpack ()

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3experimental/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/l3auxdata/build.lua

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Bundle-Flat/Module-One/build.lua

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Simple-Tree/build.lua

+ https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/build.lua
    > (function)  main(target)

+ https://raw.githubusercontent.com/latex3/latex2e/b5ed870b95fbb08206397e2ec3bef47f60c698b0/build-config.lua
    > (function)  tex(file,dir,mode)
    > (do) 
    > (function)  update_tag(file,content,tagname,tagdate)
    > (function)  update_tag_ltx(file,content,tagname,tagdate)
    > (local function)  fmt(engines,dest)
    > (local function)  mkfmt(engine)
    > (function)  checkinit_hook()
    > (function)  docinit_hook() return fmt({"pdftex"},typesetdir) end
    > (function)  typeset(file,dir)

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Simple-Flat/build.lua

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Bundle-Tree/Module-One/build.lua

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Bundle-Flat/build.lua

+ https://raw.githubusercontent.com/latex3/tex-ini-files/33e7db7adcc93935166edf6ec56783fe9305407d/build.lua
    > (function)  setversion_update_line (line, date, version)

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3trial/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3packages/xtemplate/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3experimental/xgalley/build.lua

- https://raw.githubusercontent.com/latex3/latex3/a03c65135041915cd1e03630e8b905d7660b558e/l3packages/l3keys2e/build.lua

- https://raw.githubusercontent.com/latex3/l3build/fbd4b80a7e3681d41ff1dabd6b9282fea04b6094/examples/Bundle-Tree/build.lua

+ 42 links over 50 tested without any code.
FrankMittelbach commented 1 year ago

In all honesty I'm not really in favor of competing syntax possibilities. Even if it might offer a "lighter" syntax in some cases it complicates matters for maintenances and it requires people to learn more to understand what they see in front of them. And the moment it goes mixed and matched it becomes more complicated.

projetmbc commented 1 year ago

I understand this point of view.

I will code this feature for myself, as I am a YAML addicted coder. :-) I will offer you my Lua patch, and you can punish me if you want. :-)

muzimuzhi commented 1 year ago

Personally I'd prefer TOML (spec v1.0.0) than YAML (spec v1.2.2), since a fully-featured YAML is much harder to parse, see the YAML Test Matrix.

projetmbc commented 1 year ago

TOML would not bring anything interesting compared to pure Lua as shown by the TOML conversion below of the example at the beginning of this discussion. Also, the idea is certainly not to parse full YAML, but just to limit to what is useful, i.e. YAML blocks, YAML text-or-integer variables and YAML lists.

module = "l3kernel"
bundle = ""
maindir = ".."
docfiledir = "./doc"
checkconfigs = [ "build", "config-backend", "config-l3doc", "config-plain" ]
checkfiles = [ "l3names.def" ]
cleanfiles = [ "*.fmt", "*.log", "*.pdf", "*.zip" ]
docfiles = [ "source3body.tex", "l3prefixes.csv" ]
installfiles = [
  "l3dvipdfmx.def",
  "l3dvips.def",
  "l3dvisvgm.def",
  "l3pdfmode.def",
  "l3xdvipdfmx.def",
  "l3str-enc-*.def",
  "l3debug.def",
  "l3sys.def",
  "expl3.lua\",\"expl3.ltx",
  "*.cls",
  "*.sty",
  "*.tex"
]
# ...
muzimuzhi commented 1 year ago

There is a package https://github.com/api7/lua-tinyyaml that could be used to implement the YAML file option. Is it a good idea?

Originally posted by @projetmbc in https://github.com/latex3/l3build/discussions/289#discussioncomment-6173538

BTW lua-tinyyaml is already uploaded to CTAN (see https://ctan.org/pkg/lua-tinyyaml), but currently only available in TeX Live distribution.

josephwright commented 10 months ago

I've given this a bit more thought and I'm going to close - we really don't want a proliferation of approaches.

projetmbc commented 10 months ago

I understand.