l3build manifest to be simplified

[wspr]

The l3build manifest output goes above and beyond what is actually needed as a distributed manifest. For example, it is unlikely that a long list of test files is needed, especially since they are not included in a CTAN package release.

I just ran l3build manifest on a simple package (breqn) and it generated the following: MANIFEST.txt

BY DEFAULT, the following sections are included:

## Repository manifest

## TDS manifest

## CTAN manifest

What do we want to see by default here? Should the Repository manifest be removed entirely? And only include ONE of the TDS and CTAN manifests? (TDS preferentially I assume.)

What about the "section descriptions" that currently look like:

### Source files
These are source files for a number of purposes, including the `unpack` process which
generates the installation files of the package. Additional files included here will also
be installed for processing such as testing.

My current thinking is they should stripped out entirely. (No harm in keeping the facility but insert no text by default.)

Any further thoughts?

[wspr]

For what it's worth, I still very much like the "extended" abilities of l3build manifest that generates a table of files and their descriptions:

https://github.com/wspr/unicode-math/blob/master/MANIFEST.md

This requires descriptions are embedded within the file in some way (either a comment in the header or extract the text from the first \section in the file, say). To me that makes the manifest a far more useful part of the repository documentation as a whole. (And possibly another reason that I included the repository files in the manifest in the first place.)

[FrankMittelbach]

shows that you have not enough test files :-). More seriously, yes I like that format too, maybe we can just keep it but drop the test files (or maybe just list them incompletely, eg 5 or 10 (or configurable?) per test directory with [...] if there are more and with a mentioning that only a sample of the tests is listed (in case the code finds more). Just a thought, but with the core LaTeX that part really blows it in my opinion

[FrankMittelbach]

Ok I now made some experiments and think I found a couple of issues:

first test (latex2e/base)

• ran l3build ctan (takes ages :-) )
• ran l3build manifest that generated a MANIFEST.md alright and it contained the repository files and the TDS files but the CTAN section remained empty!

second test (widows-and-orphans one of my smaller packages)

• ran ctan and manifest as before

Here everything came out right in MANIFEST.md. However, it clealry shows a flaw in the workflow.

Having done this the newly created MANIFEST.md is not part of the zip file (and it should be because the point is to distribute it to CTAN after all). This means I would need to regenerate

• rerun l3build ctan (to get the manifest included in the zip)
• rerun l3build manifest (to get the manifest listed in itself)
• rerun l3build ctan (to get the updated manifest into the zip)

given that l3build ctan for latex2e nearly runs 30 minutes on my machine that is 1.5 hours in total :-(

bottom line

I think it is ok to require ctan manifest to have the zip files around (but I think it should not silently do something if they aren't there.

But in addition I think it should a) include itself in its listing and b) should automatically insert or update the zip to include the newly generated manifest

finally, maybe I did something wrong but I think that the build.lua in latex2e/base is somewhat differently and perhaps the reason that the manifest file there is just incomplete. I guess some investigation is needed there.

[FrankMittelbach]

Followup: in windows and orphans the situation is reversed (I overlooked that): here the TDS claims to contain only the .dtx and ins file but there is no listing of the generated .sty file.

So maybe the logic to figure them out still needs some working on

[FrankMittelbach]

So maybe the logic to figure them out still needs some working on

ah, found that, one needs to first run l3build unpack for this

So maybe manifest target needs to enforce

• l3build clean
• l3build unpack
• l3build ctan
• and then fix the zip as suggested earlier

[davidcarlisle]

Frank wrote

So maybe manifest target needs to enforce

• l3build ctan
• and then fix the zip as suggested earlier

I think we shouldn't have a manifest target. The manifest is only useful for ctan zip files so if we are going to have it itin a particular package it shouldn't be possible to build ctan without the right manifest.

I would get rid of the separate target and just make it a boolean variable that you can set in a package build.lua which if true includes a generated manifest in the zip,

David

[FrankMittelbach]

Am 25.04.20 um 13:15 schrieb David Carlisle:

I think we shouldn't have a manifest target. The manifest is only useful for ctan zip files so if we are going to have it itin a particular package it shouldn't be possible to build ctan without the right manifest.

I would get rid of the separate target and just make it a boolean variable that you can set in a package build.lua which if true includes a generated manifest in the zip,

maybe, but I think having the manifest as part of the repository is also useful. For starters that means you see a summary of what is distributed and where and you get diffs if things are changing (or not changing when they should)

but I agree running a "manifest" target as part of "ctan" target to ensure that it is up-to-date is a good thing, but I think this doesn't rule out that it exists separately as a target as well.

anyway, current situation doesn't seem good so any alteration is most likely an improvement of the workflow

[josephwright]

Looking back here, it seems that what is wanted is something like a packmanifest boolean that is used only by the CTAN target, and that lists exactly what goes into the CTAN upload. That would be a significant change from what we have now, but we can do that inl3build (as in the end only the .tlg files have any real need for stability).

Assuming that is what is wanted, it presumably runs just before zipping, taking the files from the distribution directory, including itself. Does it also want to copy the manifest back to the 'source' directory for those people who check it in? Or perhaps that's not a great approach, as the point of an auto-manifest would be it's .. auto-generated.

Does that make sense?

[wspr]

Thinking about manifests for the first time in a while and with some different perspectives on automatic build processes, I see two (separate) possibilities for the point of a manifest:

1. Documentation.

Having a MANIFEST.md file in the docs/ part of a repo which can be typeset for viewing would be a “nice addition” to a standard GitHub pages kind of setup (e.g., I’ve been experimenting a little with some online fontspec documentation pages here: http://latex3.github.io/fontspec/ ).

This approach probably only provides much utility with at least some of the extra annotation features I built into l3build manifest that surely only fontspec uses.

This approach would not need (or want, I think) to document the files that make up the .tds directory for CTAN.

2. Build validation.

With a checked-in MANIFEST file, a post-build check could ensure that any changes to the .tds structure are actually intentional. E.G., a diff could be printed with an “are you sure this change was intentional?” prompt.

This approach would only really be helpful for the .tds directory, I don’t think you would need (or want) to be manually approving changes in the main repo directory.

…

I don’t really see much utility in including the manifest file in the CTAN package, although one of my original motivations in writing this code was to avoid having to least each and every source code in the readme when saying “this package is made up of the following source files and is released under LPPL”. It’s not really clear to me if that level of fussiness is really necessary.

Not sure if this helps your thought processes, Joseph. It seems to me like we should consider splitting up the feature into two in line with the two different use cases above.

I haven’t ended up using this feature as much as I expected, but now that we have GitHub actions running l3build I feel like we should investigate some more “modern” use cases before stripping out the functionality.