[FR] `refsection` support for end notes

I've released this week an end notes package called postnotes (https://ctan.org/pkg/postnotes) and have made an effort for it to work well with biblatex. And, though I could get some things working, I did hit a wall with refsections, thus this request.

The particular problem end notes (in general) pose for biblatex's citations is that the note is called in a certain place in the document where a given context/environment is in place, but the content of the note is printed (and expanded) later on in the document in a different context.

The approach I've taken in postnotes is to store some relevant variables at the point the note is called, as metadata to the note, and then restore them at the moment the note gets printed in the end. Besides general settings, like setting blx@footnote to true, I thought refsections, refsegments, and refcontexts were relevant biblatex entities which should be handled by such store/restore approach (I might have missed things in this list though). And I could take this approach in what I found to be reasonable ways for refsegments and refcontexts which, as far as I grasped, are handled on the LaTeX side of things, and I could find variables which I could store. Well, at least as far as my understanding of the code and testing went. Even if I'm acquainted with these features as an user, this was really my first serious contact with their implementation. So, I'm not sure I've done things in the best possible/appropriate way (comments and suggestions are much welcome, btw).

But refsections are more complex in this regard. For refsections the citation data must go to the .bcf file, and the way \refsection is defined to do this complicates things for the task at hand. For starters, the fact that the citation data must be written to the .bcf file means that we cannot just store/restore some variables in this case. Furthermore, \refsection pretty much presumes a given refsection will only be started/opened once: it unconditionally sets a label for the section which, of course, can only be done once and, most importantly, the optional argument of the environment (which receives the resources) is used to set a local assignment to \blx@bibfiles, based on which the relevant information is written to the .bcf file, and when the group closes the information is gone.

So the request I'm making is either to be able to "reopen" a previously closed refsection or the ability to "append/add" new citations such a refsection. In practice, I'd like to be able to store the counter \c@refsection at the point the note is called (and perhaps some more info), and be able to still make citations belonging to that refsection at the point the note is printed. Which currently, as far as I can tell, is not possible (well, I could make it work without the optional argument, but not in a "reasonable way" and not with the argument).

This sounds like you want the \assignrefcontext* macros which are part of the user-facing API (p. 107 of the current manual). These allow you to assign citations to refcontexts which are you aren't currently "in".

Hi @plk , thanks for looking into this!

However, unless I'm missing something you are seeing, I don't see how the \assignrefcontext* macros could solve the issue, or at least not in the way I was hoping for when I asked. Perhaps there is a way for an user to carefully build a document around them so that they get things working, but I don't see how to reach a general solution "from the package's side", which is what I'd like to. And as mentioned initially, even if I'm not sure if in the best ways, I could get refcontexts and refsegments working (as far as I can see), what I could not do is a similar job for refsections. How could \assignrefcontext* restore refsection's information?

Perhaps this is my fault in keeping things abstract and presuming you'd have in mind the general setting of an end notes package. Hopefully a couple of MWEs can help here, at least to illustrate. You'll recognize the examples, they are just adjusted to use postnotes.

First some standard use of refsection combined with end notes (example 10).

\documentclass{book}

\usepackage[american]{babel}
\usepackage{csquotes}
\usepackage[style=numeric,backend=biber]{biblatex}
\addbibresource{biblatex-examples.bib}

\usepackage{postnotes}
\AddToHook{cmd/chapter/before}{%
  \postnotesection{\section*{Notes to chapter \pnthechapternextnote}}}
\counterwithin*{postnote}{chapter}

% This disables the package's current support for biblatex features.  In the
% current state of things, enabling them for this document results in all
% citations being undefined.
\RemoveFromHook{postnotes/print/begin}[postnotes]
\RemoveFromHook{postnotes/print/eachnote}[postnotes]

\usepackage{hyperref}

\begin{document}

\chapter{Title of first chapter}
\begin{refsection}
This is just filler text \postnote{\parencite{massa}.}
This is just filler text \postnote{\parencite{augustine}.}
This is just filler text \postnote{\parencite{cotton}.}
This is just filler text \postnote{\parencite{hammond}.}
\printbibliography[heading=subbibliography]
\end{refsection}

\chapter{Title of second chapter}

\begin{refsection}
This is just filler text \postnote{\parencite{hammond}.}
This is just filler text \postnote{\parencite{massa}.}
This is just filler text \postnote{\parencite{cotton}.}
This is just filler text \postnote{\parencite{murray}.}
\printbibliography[heading=subbibliography]
\end{refsection}

\chapter{Title of third chapter}
\begin{refsection}
This is just filler text \postnote{\parencite{murray}.}
This is just filler text \postnote{\parencite{augustine}.}
This is just filler text \postnote{\parencite{cotton}.}
This is just filler text \postnote{\parencite{bertram}.}
\printbibliography[heading=subbibliography]
\end{refsection}

\printpostnotes

\end{document}

This document behaves as the situation would make us expect, which is not to say how we'd want. Since citations are only expanded at \printpostnotes all citations are considered to be outside the chapter refsections (refsection 0?) and thus belong to a single refsection, being numbered from 1 to 6. The \printbibliography calls in each refsection correspondingly result empty.

A labelprefix example (example 94):

\documentclass{article}

\usepackage[american]{babel}
\usepackage{csquotes}
\usepackage[style=numeric,sorting=none,defernumbers=true,backend=biber,notetype=endonly]{biblatex}
\addbibresource{biblatex-examples.bib}

\DeclareBibliographyCategory{cat1}
\addtocategory{cat1}{yoon}
\DeclareBibliographyCategory{cat2}
\addtocategory{cat2}{piccato}
\assignrefcontextcats[sorting=none,labelprefix=Q]{cat2}
\assignrefcontextentries[labelprefix=Q]{yoon}
\assignrefcontextkeyws*[sorting=none,labelprefix=S]{secondary}
\defbibfilter{cats}{category=cat1 or category=cat2}

\DeclareRefcontext{rcone}{labelprefix=R}
\DeclareRefcontext{rctwo}{labelprefix=T}
\DeclareRefcontext{rcthree}{sorting=nty}

\usepackage{postnotes}

% This disables the package's current support for biblatex features.  In the
% current state of things, enabling them for this document results in 'worman'
% being undefined and the labels of the rest of the document being wrong.
\RemoveFromHook{postnotes/print/begin}[postnotes]
\RemoveFromHook{postnotes/print/eachnote}[postnotes]

\usepackage{hyperref}

\begin{document}

This is a publication by Aristotle:
\footcite{aristotle:anima}

These are not publications by Aristotle:
\footcite{yoon,worman,piccato,nussbaum}

This is another publication by Aristotle:
\footcite{aristotle:physics}

\begin{refcontext}{rcone}
\printbibliography[keyword=primary, title={Aristotle Publications}]
\end{refcontext}

\newrefcontext[labelprefix=S]{rctwo}
\printbibliography[notkeyword=primary, title={Other publications}]
\endrefcontext

\begin{refcontext}[labelprefix=Q]
\printbibliography[filter=cats, title={Other publications}]
\end{refcontext}

\begin{refcontext}[sorting=nty]
\printbibliography[resetnumbers, notkeyword=primary, title={More Other publications}]
\footcite{nussbaum}
\end{refcontext}

\section*{Refsection}
\begin{refsection}
\footcite{worman}
\begin{refcontext}[labelprefix=T]{rcone}
  \printbibliography[resetnumbers, notkeyword=primary, title={More Other
      publications}]
\end{refcontext}
\end{refsection}

\printpostnotes

\end{document}

In this document, the worman citation inside the refsection comes with its global label ("3"), instead of the expected "T1" and the \printbibliography inside the refsection is empty, when it shouldn't.

I'd like these kinds of documents to be viable, but currently they are far from it. The general problem is the one mentioned from the start: the notes' contents, the citations included, get expanded at \printpostnotes, in a different context from the one which "they belong to" (where the mark is) thus tricking biblatex. The current support of postnotes for biblatex's features makes things worse for these documents (you can try it, commenting the \RemoveFromHooks out). Alas, they are known to be incomplete and unable to handle refsections. So I definitely have work to do on my side, but the thing is I don't stand a chance of trying to fix it, for the reasons exposed initially: the tools in biblatex to handle "a local refsection which behaves like the one I have stored at specific point in the document", as far as I can see, are not available. Thus the request.

The problem is the same for other end notes packages (I've tested the two documents above with endnotes with equivalent results, I see no reason why enotez or pagenotes/memoir would behave any different). But postnotes does not send the notes' contents to a file, rather stores them in a token list variable which is later used directly for typesetting, but in a (grouped) context which makes somewhat easier (than for packages which rely on an external file) to control the environment where this stored variable gets finally expanded. And that's the added flexibility I'm trying to exploit to improve support for biblatex, but I've hit a wall with refsections.

Roping in @moewew in case he has any comments here.

@plk and @moewew I can provide more information on how this "store/restore" process is being handled at postonotes' side, and what I attempted for biblatex so far in particular, if you think it is useful. I kept the request lean not to pour you guys with unsolicited stuff but, of course, I can expand on what you think may be relevant/useful.

And a further comment. I said I could get refsegment and refcontext working, that's true for the relevant (biblatex) example documents, which I tested. But I'm sufficiently ignorant of the inner working of this machinery to be somewhat uncertain about it the way I handled things. That's to say I'd also very much welcome some more "institutionalized" means of "saving" and "locally restoring" not only refsections, but also refsegements and refcontexts.

I think it would be interesting if this feature set of biblatex could be made to work with end notes functionality. And, as far as I can tell, these things have been incompatible for a long time. That's not to say the way I'm conceiving the solution is "best", or even "correct". I know it may be tricky, and perhaps not even viable, but you two know better than me, so I had to ask.

(And, @moewew , before you mumble at me, recall it was you who put me on the track: https://tex.stackexchange.com/questions/597359/reset-chapter-endnotes#comment1594585_597389 ;-)

I had seen this issue, but it looked like a bigger project that I unfortunately don't have the time for right now.

Over the last years, new features have meant that biblatex has amassed quite a lot of "context" for each citation and tracking and restoring that correctly (especially if the citation command itself is not executed at the point it appears in the code) might be quite tricky. (Plus in your case, we'd have to think about which bits of the document should decide the context: The place where the footnote appears in the text or the footnote when it is actually printed at the end. Of course this matters less if you track in-text and footnotes separately, but still...)

For now, let me link to https://github.com/plk/biblatex/issues/1116 for how we currently "restore" refsections in .lof files and friends and to https://github.com/plk/biblatex/issues/307 and https://github.com/plk/biblatex/issues/344 for related questions on reopnening refsections/segments.

@moewew I know this is a tricky request, bound to be quite some work, so I understand perfectly well. And I appreciate you both even considering this. Thank you.

(Plus in your case, we'd have to think about which bits of the document should decide the context: The place where the footnote appears in the text or the footnote when it is actually printed at the end. Of course this matters less if you track in-text and footnotes separately, but still...)

That is very true as well, and a delicate issue. So far I had thought about it in the following lines: things related to the page tracker (ibid., op. cit., etc.) should belong to the "printing"; things that relate to document subdivisions, like refsection, refsegment, and refcontext, should belong to the "mark". But I bet there is more "context" which I didn't think of (not even with your hint). And also these things I did think of are clearly debatable.

This is really difficult - the refsection number is part of almost every internal variable - just search for c@refsection in biblatex.sty and to be safe, every variable containing this would need to be saved/restored. Now, I suppose we could do that - have a master list of all variable that contain the refsection but I'm not entirely sure even that would address this as refsections really are a fundamental structure that it was was never planned to switch.

@plk I understand. But I had to ask, the use case is concrete, and affects any end notes package for the same reasons. Thank you (both) very much for considering it though.

However, I do wonder if you might try saving the refsection number of the postnote definition with the postnote and then doing:

  \c@refsection#1\relax

Before anything else, where the postnote is printed. #1 is the refsection where the postnote was defined. So many things are conditionalised by this number and it may be all you need. Just be sure to restore the current value when the postnote output is done.

I had tried to restore the counter itself, if that's what you have in mind (this is currently disabled from the package):

https://github.com/gusbrs/postnotes/blob/c3ea82517528d03f606f654101c98d792e5ca981/postnotes.dtx#L2068-L2111

(The postnotes/note/store hook is called where the postnote is set and the postnotes/print/note/begin one where the note is printed).

But I had at least one other big problem in that the citations were written to the .bcf file, at which point the last currently open refsection was active. So I did try "reopening" the original one with (also currently disabled):

https://github.com/gusbrs/postnotes/blob/c3ea82517528d03f606f654101c98d792e5ca981/postnotes.dtx#L2192-L2246

I could get most of the relevant biblatex example files to work with that structure (I think there was one exception, I can dig it if you wish). I also could not handle the optional argument, as initially reported. But, most importantly, I could also defeat and confuse biblatex, and it felt wrong to trample onto biblatex's internals in such way.

plk / biblatex

[FR] `refsection` support for end notes #1226