interaction between theorem environments and \hangindent

[blumens]

Brief outline of the bug

When putting a theorem in a paragraph that uses \hangindent, the indent is also applied to the part of the paragraph after the theorem. This happens both with plain latex and with the amsthm package.

I'm not sure this is intended behaviour: either the text after the theorem is considered to be part of the same paragraph, in which case it should be indented in accordance to its line number in the whole paragraph, or it is considered to be a new paragraph, in which case \hangindent and \hangafter should be reset.

Of course backwards compatibility might make it impossible to change the current behaviour.

Minimal example showing the bug

\RequirePackage{latexbug}
\documentclass{article}
%\usepackage{amsmath}
%\usepackage{amsthm}
%\usepackage{thmtools}

\newtheorem{Def}{Definition}

\begin{document}
The Theorem of B\"uchi can be extended to the monadic second-order theory of
the infinite complete binary tree.
As there currently does not exist a purely model-theoretic proof of this fact,
we present the standard automata-theoretic version here.
Let us start by fixing our notation regarding trees.

\hangindent=2em\hangafter=-2%
The Theorem of B\"uchi can be extended to the monadic second-order theory of
the infinite complete binary tree.
As there currently does not exist a purely model-theoretic proof of this fact,
we present the standard automata-theoretic version here.
Let us start by fixing our notation regarding trees.
\begin{Def}
foo
\end{Def}
The Theorem of B\"uchi can be extended to the monadic second-order theory of
the infinite complete binary tree.
As there currently does not exist a purely model-theoretic proof of this fact,
we present the standard automata-theoretic version here.
Let us start by fixing our notation regarding trees.

\end{document}

Log file (required) and possibly PDF file

<!--- Use drag-and-drop  (remember: .log file is required!) -->
This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023/Arch Linux) (preloaded format=pdflatex 2023.7.22)  29 JUL 2023 15:12
entering extended mode
 restricted \write18 enabled.
 %&-line parsing enabled.
**test.tex
(./test.tex
LaTeX2e <2022-11-01> patch level 1
L3 programming layer <2023-02-22>
(/usr/share/texmf-dist/tex/latex/latexbug/latexbug.sty
Package: latexbug 2022/06/14 v1.0n Bug-classification
)
(/usr/share/texmf-dist/tex/latex/base/article.cls
Document Class: article 2022/07/02 v1.4n Standard LaTeX document class
(/usr/share/texmf-dist/tex/latex/base/size10.clo
File: size10.clo 2022/07/02 v1.4n Standard LaTeX file (size option)
)
\c@part=\count185
\c@section=\count186
\c@subsection=\count187
\c@subsubsection=\count188
\c@paragraph=\count189
\c@subparagraph=\count190
\c@figure=\count191
\c@table=\count192
\abovecaptionskip=\skip48
\belowcaptionskip=\skip49
\bibindent=\dimen140
)
\c@Def=\count193

(/usr/share/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def
File: l3backend-pdftex.def 2023-01-16 L3 backend support: PDF output (pdfTeX)
\l__color_backend_stack_int=\count194
\l__pdf_internal_box=\box51
) (./test.aux)
\openout1 = `test.aux'.

LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 9.
LaTeX Font Info:    ... okay on input line 9.
LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 9.
LaTeX Font Info:    ... okay on input line 9.
LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 9.
LaTeX Font Info:    ... okay on input line 9.
LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 9.
LaTeX Font Info:    ... okay on input line 9.
LaTeX Font Info:    Checking defaults for TS1/cmr/m/n on input line 9.
LaTeX Font Info:    ... okay on input line 9.
LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 9.
LaTeX Font Info:    ... okay on input line 9.
LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 9.
LaTeX Font Info:    ... okay on input line 9.

[1

{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] (./test.aux) ) 
Here is how much of TeX's memory you used:
 1773 strings out of 476025
 42020 string characters out of 5796532
 1851388 words of memory out of 5000000
 22301 multiletter control sequences out of 15000+600000
 512941 words of font info for 34 fonts, out of 8000000 for 9000
 1141 hyphenation exceptions out of 8191
 34i,5n,38p,161b,1309s stack positions out of 10000i,1000n,20000p,200000b,200000s
</usr/share/
texmf-dist/fonts/type1/public/amsfonts/cm/cmbx10.pfb></usr/share/texmf-dist/fon
ts/type1/public/amsfonts/cm/cmr10.pfb></usr/share/texmf-dist/fonts/type1/public
/amsfonts/cm/cmti10.pfb>
Output written on test.pdf (1 page, 37390 bytes).
PDF statistics:
 23 PDF objects out of 1000 (max. 8388607)
 13 compressed objects within 1 object stream
 0 named destinations out of 1000 (max. 500000)
 1 words of extra memory for PDF output out of 10000 (max. 10000000)

[FrankMittelbach]

Basically \hangindent is a low-level primitive which is a simplified version of another low-level primitive \parshape, ie only one of them can be active at a given time. Neither of them is supported LaTeX syntax at the document level, that is they should not be used by users.

LaTeX uses \parshape together with a redefinition of \par to build all of its display environments and ther left and right margin indentation, ie all lists, theorems, center quote, etc. If you set up \hangindent or \parshape on the document level the way you did and this is directly followed by any display environment (without a \par before that display) then LaTeX interprets the special shape as something that was set up on the outside and should be restored after the display --- this is how nested lists or quote set up nested indentations.

This is why you see a repeat of your setting after the display. (inside the display you don't see it because there LaTeX's \parshape setting overwrites it.)

Resolution:

• don't use it in the first place, it will fail in LaTeX in many places and it isn't supported syntax
• if you do nevertheless, make sure your paragraph explicitly ends before a display starts; in this example it is enough

[blumens]

Does this mean there is no supported way to change the parshape at all? Which means that all packages like lettrine or wrapfig are a priory buggy?

[FrankMittelbach]

no it means that they know what they are doing and account for LaTeX's handling of parshape (or if they don't know then yes they end up being buggy ... but not those two)

[FrankMittelbach]

correction ... lettrine has the same issue it seems, perhaps we have to take a closer look at this after all

[blumens]

Your answer suggests that there is a supported way to change the parshape. I suppose the only documentation is in source2e?

[u-fischer]

lettrine explicitly mentions the issue in its documentation

If a list has to be included in a paragraph starting with \lettrine, it is necessary to add the command \parshape=0 just after the end of the list (starting a new paragraph just before or just after the list works too). Remember that ‘quote’, ‘quotation’, ‘abstract’ environments are implemented as lists in LaTeX.

[blumens]

I took a quick look at source2e and \parshape is mentioned only three times. I could not find any explanation of its use in list environments (only the raw tex code). So if you decide that the behaviour is as intended, could you please add some explanation to the documentation?

[FrankMittelbach]

page 712 has pseudo code for lists (in which parshape is used). This usage dates back to LaTeX 2.09 ie before 1990.

The problem is basically this: if you use your own parshape or hangindent and your current paragraph ends inside a group (which is the case if you go \begin{theorem} or \begin{itemize} because \begin starts a group and only then the environment issues a \par in its code) then the parshape gets reset at this \par but given that happens in a group it is restored after the group, which explains why you see your hangindent after the theorem ended again.

[FrankMittelbach]

Your answer suggests that there is a supported way to change the parshape. I suppose the only documentation is in source2e?

not for authors on document level or at least nothing other than using the generic list environment, but not with respect to something like hangindent. lettrine for example builds a special parshape taking into account it is is already inside some display like quote or itemize. But the basic problem outlined above remains (which is why lettrine documents it). This is as I said very old code and I must confess I had forgotten about this low-level TeX dependency, ie the way it restores these parshapes.

[FrankMittelbach]

Done a bit more analysis: wrapfig handles the situation gracefully, by canceling the cutout even if the next \par happens in a group, but Donald's approach is rather complicated. lettrine uses the simpler solution of documenting it as a limitation as Ulrike mentioned.

Providing a general interface for this from the LaTeX kernel side, is not impossible but rather heavy on resources and it wouldn't resolve the general issue that something like \hangindent doesn't work in other situations either, e.g., try your setting inside of a list environment or inside quote. So my guess is that we aren't going to spend time on this, at least not anytime soon. But I'll keep it open because it is a somewhat interesting challenge (not to directly support the low-level \hangindent but perhaps to provide a more LaTeX like interface one day).

However, for now use the advice given in lettrine if you want to cut holes the way you did and add \parshape=0 after a following list to prevent the cutout from reappearing there.