Indents not being applied in noweave output

RobertBaruch commented 2 years ago

Test file t.nw (same as examples/test.nw):

% Copyright 1991 by Norman Ramsey.  All rights reserved.
% See file COPYRIGHT for more information.
<<*>>=
one <<two>> <<three>>   # uses two and three
@ %def one
<<two>>=
first of two
second of two
third of two
@ %def fish fowl duck
@ %def two
<<three>>=
first of three
 second of three
  third of three
@ %def three

$ noweave t.nw > t.tex
$ pdflatex t.tex
This is pdfTeX, Version 3.141592653-2.6-1.40.24 (TeX Live 2022) (preloaded format=pdflatex)
 restricted \write18 enabled.
entering extended mode
(./t.tex
LaTeX2e <2022-06-01> patch level 2
L3 programming layer <2022-06-16>
(/usr/local/texlive/2022/texmf-dist/tex/latex/base/article.cls
Document Class: article 2021/10/04 v1.4n Standard LaTeX document class
(/usr/local/texlive/2022/texmf-dist/tex/latex/base/size10.clo))
(/usr/local/tex/inputs/noweb.sty)
(/usr/local/texlive/2022/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def)
(./t.aux) [1{/usr/local/texlive/2022/texmf-var/fonts/map/pdftex/updmap/pdftex.m
ap}] (./t.aux) )</usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts
/cm/cmr10.pfb></usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts/c
m/cmsy10.pfb></usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts/cm
/cmti10.pfb></usr/local/texlive/2022/texmf-dist/fonts/type1/public/amsfonts/cm/
cmtt10.pfb>
Output written on t.pdf (1 page, 42739 bytes).
Transcript written on t.log.

Result is in t.pdf, which shows none of the expected indenting in codeblock <<three>>.

The problem doesn't seem to be with pdflatex since a \begin{verbatim} block retains indents.

nrnrnr commented 2 years ago

I can't duplicate the problem. (The indentation is fine for me.)

What's the checksum on your t.tex? I get

$ md5sum t.tex
51eda6cbefcb75ab1767aa5743850a11  t.tex

If yours is different, let's see it.

Details about your system, please?

I've got

This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=pdflatex)

RobertBaruch commented 2 years ago

I did get a different checksum. Here's t.tex:

\documentclass{article}\usepackage{noweb}\pagestyle{noweb}\noweboptions{}\begin{document}\nwfilename{t.nw}\nwbegindocs{0}\nwdocspar
% Copyright 1991 by Norman Ramsey.  All rights reserved.
% See file COPYRIGHT for more information.
\nwenddocs{}\nwbegincode{1}\moddef{*}\endmoddef
one \LA{}two\RA{} \LA{}three\RA{}   # uses two and three
\eatline
\nwendcode{}\nwbegincode{2}\moddef{two}\endmoddef
first of two
second of two
third of two
\eatline
\eatline
\nwendcode{}\nwbegincode{3}\moddef{three}\endmoddef
first of three
 second of three
  third of three
\eatline
\nwendcode{}\end{document}

And here's my tex version:

$ pdflatex --version
pdfTeX 3.141592653-2.6-1.40.24 (TeX Live 2022)
kpathsea version 6.3.4
Copyright 2022 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.37; using libpng 1.6.37
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.03

nrnrnr commented 2 years ago

Your .tex file works correctly for me in pdflatex.

I'm not sure where your Noweb installation comes from, so I'm not sure what else we can do here.

RobertBaruch commented 2 years ago

Hmm, I cloned noweb from git. I can delete it all and try again...

RobertBaruch commented 2 years ago

I tried it in two different ways:

apt install noweb
git clone https://github.com/nrnrnr/noweb.git + compile & install

In both cases, indents were not applied.

This was all using TeX Live 2022.

I installed TeX Live 2019, and got the indents. So there is something that happened in TeX Live 2020, 2021, or 2022 that is causing whatever noweb uses to indent to not work anymore.

I don't really want to use 2019 because usually the latest version of TeX Live is recommended, and tlmgr to install packages tries to install the latest, and complains about 2019:

$ tlmgr repository list
(running on Debian, switching to user mode!)
List of repositories (with tags if set):
    http://mirror.ctan.org/systems/texlive/tlnet (main)

$ tlmgr --verify-repo=none install tikz
tlmgr: Local TeX Live (2019) is older than remote repository (2022). 
Cross release updates are only supported with
  update-tlmgr-latest(.sh/.exe) --update
See https://tug.org/texlive/upgrade.html for details.

And then that page has a big notice on it about how upgrading is dangerous and you should just install the latest version.

nrnrnr commented 2 years ago

OK, I can well believe it's a TeXlive issue. I've been bitten before.

I'm running Debian stable. That's TeXlive 2020. It looks like TeXlive 2022 is available on Debian unstable. But I can't anticipate setting up an unstable system any time soon---my plate is full right now, and it could be as long as a year.

Maybe there's an easier way of setting up a test environment? Or maybe you can provide me with access to a system on which to test? In that case I can likely track this issue down sometime between now and mid-August. (You could also try filing a bug report against Debian unstable, but given that it's manifestly an upstream issue, they might not be interested.)

RobertBaruch commented 2 years ago

I'm ok with having to use a stable version -- I can just apt install texlive-full which seems to load nearly all the packages in CTAN, versioned according to 2019 (or I guess 2020). I'm on Ubuntu 20.04.3 LTS, which pins 2019.

Probably best to leave this bug open and revisit when the major stable releases pin TeX Live to 2021 or 2022.

As for a test environment, you should be able to install 2022 in parallel with 2020, but you'd probably have to change some env variables like TEXINPUTS, PATH, and point the tlmgr repo to the right one. At least, that's all I had to do.

Then again, it's hard to know whether the installation is hermetic. Docker is probably the best option here.

dbosk commented 2 years ago

Works correctly on TeXLive 2021 (Ubuntu 22.04 LTS).

I can reproduce that it works incorrectly on TeXLive 2022:

$ docker run -it -v /tmp:/outside texlive/texlive:latest
root@e0fff1176e4e:/# apt update
[...]
root@e0fff1176e4e:/# apt install noweb
[...]
root@e0fff1176e4e:/# cp /outside/test.tex .
root@e0fff1176e4e:/# TEXMFHOME=/usr/share/texmf pdflatex test.tex
[...]

Yields:

arcfide commented 2 years ago

I've been bitten by this on MikTeX, too. I'm not sure what's going on. The verbatim environment works but not the noweb indentation, with or without tabs.

arcfide commented 2 years ago

Not just on Tex Live, but MikTeX also has this issue with its latest updates.

arcfide commented 2 years ago

I've made some progress on this. In support.nw, chunk 8b, the definition of \@begincode includes the following lines:

<<zap ligatures, fix spaces 10c>>
\nowebsize \setupcode

This appears to be where the problem occurs. It looks like there might be some clobbering of catcodes or the like. Chunk 10c includes the code that we want, namely, \@vobeyspaces. However, this doesn't seem to be in force by the time the actual code is rendered. I can add a line like \let =\ to the beginning of a code chunk and things work after that.

Strangely, I've found that if you swap the above two lines, then things work, that is, if you fix spaces after \nowebsize \setupcode are called. I also am not sure if we really need to be calling \setupcode two times, once in chunk 8b, and a second time in chunk 10c. At any rate, I have a minor workaround for the moment, but I'm not sure what the real fix should be.

nrnrnr commented 2 years ago

It's great to hear of some progress here, but I'm about to go on vacation for three weeks, then business travel. I'll try to remember to get back to this around the beginning of October.

Mechtilde commented 1 year ago

Hello i also stumpled about this problem with the Update to TexLive 2022 in Debian.

If some provide me a patched file I will be able to test it .

meta-ed commented 1 year ago

This problem, or one indistinguishable from the description above, is now seen with Noweb 2.12 on the Slackware 15.0 platform. There was a recent security patch to TeX Live by the Slackware 15.0 maintainers. The patch brought TeX Live forward from texlive-2021.210418 to texlive-2023.230322. With texlive-2023.230322, indentation is broken. Reverting to texlive-2021.210418 restores the correct indentation behavior.

Is the request still open for access to a system on which to test?

Mechtilde commented 1 year ago

Hello,

you can use the patch also to path your version of noweb under slackware as I did it for Debian

It will be released as stable next weekend.

I already use it regularly.

Kind regards

Am 06.06.23 um 14:43 schrieb meta-ed:

This problem, or one indistinguishable from the description above, is now seen with Noweb 2.12 on the Slackware 15.0 platform. There was a recent security patch to TeX Live by the Slackware 15.0 maintainers. The patch brought TeX Live forward from texlive-2021.210418 to texlive-2023.230322. With texlive-2023.230322, indentation is broken. Reverting to texlive-2021.210418 restores the correct indentation behavior.

Is the request still open for access to a system on which to test?

-- Mechtilde Stehmann

Debian Developer

PGP encryption welcome

F0E3 7F3D C87A 4998 2899 39E7 F287 7BBA 141A AD7F

nrnrnr commented 1 year ago

Fantastic. I will put it on my list for after the latest stable release this weekend.

nrnrnr commented 1 year ago

Fixed. If anybody wants to try it out, I'll take the "RC" off the version tag.

meta-ed commented 1 year ago

There is an unexpected dependency on the .git directory. When building from the tarball, probably there shouldn't be such a dependency.

make[1]: *** No rule to make target '../../.git/refs/heads/master', needed by 'gitversion.c'. Stop.

I do not find gitversion.c in 2.12, only 2.13rc, so I assume that's a dependency that crept in between the two point releases.

nrnrnr commented 1 year ago

Thanks for catching this bug. I've pushed v2_13rc3, which seems to resolve it.

meta-ed commented 1 year ago

I see rc2 but not rc3. I'll look again tonight. EDIT: I should add in case that's just a typo and it's rc2, I tried building rc2 and ran into the same issue.

nrnrnr commented 1 year ago

My bad. I pushed the commit but not the tag.

meta-ed commented 1 year ago

Making progress! The build now fails at: find: ‘/tmp/SBo/package-noweb-2.13rc3/usr/man’: No such file or directory Investigating, it appears the build writes the manpages to /usr/man instead of $DESTDIR/usr/man.

EDIT: I'm using the Slackware 15.0 build script that was developed for 2.12 to produce the binary package for Slackware 15. As I look deeper into it, apparently $DESTDIR is patched in by the build script maintainer exactly for this reason. So I need to go back and update that patch to work with 2.13.

nrnrnr commented 1 year ago

I accepted a PR that made several changes to the Makefiles, but I don't remember this quite being an issue. Can you please let me know which file it tried to deposit in the bad place?

meta-ed commented 1 year ago

A script by Kyle Guinn builds 2.12 for Slackware 15.0. That script applies a patch to 2.12 makefiles before running make install. The patch lets installed files be dropped in a temporary tree. The tree is then tarballed and becomes the binary distribution.

Going by Kyle's patch, I patched 2.13rc3, and successfully created a binary distribution of 2.13rc3.

Then I tested these four cases:

Texlive 2021, Noweb 2.12 -- works Texlive 2023, Noweb 2.12 -- indents don't work Texlive 2021, Noweb 2.13rc3 -- works Texlive 2023, Noweb 2.13rc3 -- works

Example of what the patch does:

-       mkdir -p $(BIN) $(LIB)
-       sed "s@|LIBDIR|@$(LIBNAME)@;s@|GITVERSION|@$(GITVERSION)@" shell/noweb > $(BIN)/noweb
-       chmod +x $(BIN)/noweb

becomes

+       mkdir -p $(DESTDIR)$(BIN) $(DESTDIR)$(LIB)
+       sed "s@|LIBDIR|@$(LIB)@;s@|GITVERSION|@$(GITVERSION)@" shell/noweb > $(DESTDIR)$(BIN)/noweb
+       chmod +x $(DESTDIR)$(BIN)/noweb

nrnrnr commented 1 year ago

Frankly, if I were writing such a patch, I'd be patching lines 22 to 28 and leaving the rest alone. That's the procedure suggested in step 1 of the INSTALL file.

An even better plan would be to create a shell script that invokes Make while setting those variables on the command line.

Obviously, Slackware can do as it likes.

meta-ed commented 1 year ago

I doubt I'll explain this well, but I'll try. And maybe you understand this already, and I'm not understanding the Noweb makefiles. If so I apologize for wasting your time.

The problem is not specific to Slackware. Any time an application is to be packaged as a binary tarball, the build procedure can't install the pieces to the running system library. The pieces need to go into an empty tree that can be tarballed and later extracted to target systems.

That's easily done by editing lines 22 to 28. I can set BIN=/tmp/build/usr/bin, LIB=/tmp/build/usr/libexec/noweb, MAN=/tmp/build/usr/man, etc. and that creates the empty tree perfectly.

Now the pathnames in the resulting tarball will not include the /tmp/build prefix. It will contain usr/bin/noweb usr/libexec/noweb/markup usr/man/noweb.1.gz etc. This lets the tarball extract into the target system's root directory structure.

But the change in pathname breaks Noweb at run-time. For example, /usr/libexec/noweb/markup exists, but Noweb scripts are still looking for it in /tmp/build/usr/libexec/noweb/markup.

nrnrnr commented 1 year ago

Ah, I think I see the problem. The expectation is as follows:

Noweb is installed into a temporary directory.
The directory is used to build a tarball containing relative pathnames.
The tarball is later to be unpacked into a final destination directory.
The files installed in the temporary directory should contain pathnames relative to the final destination directory.

Such functionality could be provided by altering src/Makefile so that the install targets place files into the temporary directory. The Debian build scripts have a patch like this as well. I'd consider a PR that would make this sort of change simpler, but I think I don't want to do it myself. The documentation could be gnarly, and it would be of interest only to package maintainers. Who, by and large, already know what they are doing.

@Mechtilde take note.

meta-ed commented 1 year ago

@Mechtilde can you, Kyle Guinn, and I maybe collaborate on a single pull request that would support package creation?

turboencabulator commented 1 year ago

A script by Kyle Guinn builds 2.12 for Slackware 15.0. That script applies a patch to 2.12 makefiles before running make install. The patch lets installed files be dropped in a temporary tree. The tree is then tarballed and becomes the binary distribution.

FYI the makefile patch mentioned here is largely PR #10, plus or minus some things, but the basic purpose was to make it easier to create a distro package. I think we deemed it too extensive of a change, so I've been trying to get it pulled in in smaller pieces, which has kind of stalled lately. I can restart that effort.

Mechtilde commented 1 year ago

Hello,

yes I'm willing to help as far as I can.

I opened also a bugreport https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040171 and asking the former maintainer for help.

I had problems to adapt the Makefiles for Debian packages

Regards

Am 30.06.23 um 22:12 schrieb meta-ed:

@Mechtilde can you, Kyle Guinn, and I maybe collaborate on a single pull request that would support package creation?

-- Mechtilde Stehmann

Debian Developer

PGP encryption welcome

F0E3 7F3D C87A 4998 2899 39E7 F287 7BBA 141A AD7F

turboencabulator commented 1 year ago

I'm not too familiar with Debian's packaging procedures, but I'm browsing https://sources.debian.org/src/noweb/2.12-1/debian/patches/ and can comment on what's been done since that release. I think all three of these patches are no longer needed:

buildflags.diff: Adding $(CFLAGS) and $(LDFLAGS) has already been done. This patch also adds $(CPPFLAGS), I don't know if that variable is needed elsewhere. If it is needed, I'm wondering if we need to manually pass it to a sub-make (see the next bullet). Same applies to $(LDFLAGS), I see we're passing $(CC) and $(CFLAGS) but not $(LDFLAGS), and I imagine we need to do it for all or none.
recursive-make-special-variable.diff: Replacing make with $(MAKE) should already be done, which is probably the most important part of this patch. There is an idiom of (cd dir; make target) which is now (cd dir && $(MAKE) target) and this patch uses $(MAKE) -C dir target instead. I personally prefer the -C syntax here, but I'm not sure how portable it is for whichever flavors of make that we're targeting, so I left it out.
toascii.diff: This has already been done.

meta-ed commented 1 year ago

@turboencabulator To your point about portability: The -C DIR (--directory=DIR) option is not supported by standard Unix make. It complicates porting outside the Gnu ecosystem.

ko56 commented 1 year ago

Hi, I still have the indentation problem with noweb 2.13, built from git on FreeBSD 13, with TeXLive 2023. (But, as others already mentioned, I don't have the problem with TeXLive 2021.) Here is my examples/test.tex file:

\documentclass{article}\usepackage{noweb}\pagestyle{noweb}\noweboptions{}\begin{document}\nwfilename{/usr/home/ko/build/noweb/examples/test.nw}\nwbegindocs{0}% Copyright 1991 by Norman Ramsey.  All rights reserved.% ===> this file was generated automatically by noweave --- better not edit it
% See file COPYRIGHT for more information.
\nwenddocs{}\nwbegincode{1}\moddef{*}\endmoddef\nwstartdeflinemarkup\nwenddeflinemarkup
one \LA{}two\RA{} \LA{}three\RA{}   # uses two and three
\eatline
\nwendcode{}\nwbegincode{2}\moddef{two}\endmoddef\nwstartdeflinemarkup\nwenddeflinemarkup
first of two
second of two
third of two
\eatline
\eatline
\nwendcode{}\nwbegincode{3}\moddef{three}\endmoddef\nwstartdeflinemarkup\nwenddeflinemarkup
first of three
 second of three
  third of three
\eatline
\nwendcode{}\nwbegindocs{4}\nwdocspar
\nwenddocs{}\end{document}

And here is some information on TeXLive:

[ko@wiley ~]$ pkg info texlive-texmf
texlive-texmf-20230313
Name           : texlive-texmf
Version        : 20230313
Installed on   : Wed Aug  9 12:41:12 2023 EDT
Origin         : print/texlive-texmf
Architecture   : FreeBSD:13:*
Prefix         : /usr/local
Categories     : print
Licenses       : 
Maintainer     : tex@FreeBSD.org
WWW            : https://tug.org/texlive/
Comment        : TeX Live Typesetting System, texmf Tree
Annotations    :
    cpe            : cpe:2.3:a:tug:texlive:20230313:::::freebsd13:x64
    repo_type      : binary
    repository     : FreeBSD
Flat size      : 2.83GiB
Description    :
TeX Live macro packages and fonts.

WWW: https://tug.org/texlive/
[ko@wiley ~]$

I assume nobody else has a problem with TeXLive 2023?

arcfide commented 1 year ago

I'm not familiar with FreeBSD's git packaging; can you confirm that this includes the commit that "fixed" this issue?

ko56 commented 1 year ago

The FreeBSD package I'm using only contains texlive. Noweb 2.13 I've built myself from the git source, so I assume it contains the fix.

nrnrnr commented 1 year ago

The relevant commits are https://github.com/nrnrnr/noweb/commit/a6a9cb2167977146ca3f979267099a263b102ec3 and https://github.com/nrnrnr/noweb/commit/657a70b8d4649e9152266e77fc447098ecca3140.

ko56 commented 1 year ago

The relevant commits are a6a9cb2 and 657a70b.

I built noweb 2.13 from the git source on Aug. 3 2023, so I assume these fixes are there, no?

ko56 commented 1 year ago

Problem solved, sorry to have taken everyone's time. texlive was finding the 2.12 version of noweb.sty, not the 2.13 version.

yurrriq commented 1 year ago

nrnrnr / noweb