newrefcontext breaks numeric styles

[moewew]

I wanted to play around with labelprefix in the newest dev version of 3.4 and found that adding \newrefcontext[labelprefix=A] resets all numbers in citations to zero before \printbibliography.

See

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[style=numeric,backend=biber,defernumbers=true]{biblatex}

\addbibresource{biblatex-examples.bib}

\begin{document}
\newrefcontext[labelprefix=A]
\cite{sigfridsson,worman}
\printbibliography
\cite{sigfridsson,worman}
\end{document}

examples 16, 17, and 19 in the docs didn't work for me either. The prefix is always dropped and with numeric citations I get only "0".

---

BTW: I find it quite confusing the \newrefcontext seems to take only an optional argument ([...]), but when I omit it, biblatex complains; is it really a mandatory argument?

[plk]

This works fine for me with latest 3.4 and biber 2.5, all example docs are fine too - are you sure you have the latest biber?

The argument to \newrefcontext is a bit strange, I admit. It used to be optional but now isn't. Hmm.

[moewew]

Interesting, ... I just pulled Biber and biblatex from sourceforge and can still reproduce the problem in my MWE as well as the test docs.

The (I guess) relevant part of the .aux file is

\abx@aux@sortscheme{nty}
\abx@aux@refcontext{nty/global/}
\abx@aux@refcontext{nty/global/A}
\abx@aux@cite{sigfridsson}
\abx@aux@cite{worman}
\abx@aux@number{2}{sigfridsson}{0}{nty/global/A}{1}
\abx@aux@number{3}{worman}{0}{nty/global/A}{2}
\abx@aux@refcontextdefaultsdone

The .bbl contains both entries twice, once in \sortlist[entry]{nty/global/A} and once in \sortlist[entry]{nty/global/}.

The .bcf has

  <bcf:bibdata section="0">
    <bcf:datasource type="file" datatype="bibtex">biblatex-examples.bib</bcf:datasource>
  </bcf:bibdata>
  <bcf:section number="0">
    <bcf:citekey order="1">sigfridsson</bcf:citekey>
    <bcf:citekey order="1">worman</bcf:citekey>
    <bcf:citekey order="2">sigfridsson</bcf:citekey>
    <bcf:citekey order="2">worman</bcf:citekey>
  </bcf:section>

---

I notice that the problem occurs as soon as I have defernumbers set to true as well as a non-empty argument to \newrefcontext that differs from the default settings.

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[style=numeric,backend=biber,defernumbers=true]{biblatex}

\addbibresource{biblatex-examples.bib}

\begin{document}
\newrefcontext[sorting=nyt]
\cite{sigfridsson,worman}
\printbibliography
\cite{sigfridsson,worman}
\end{document}

for example, is also problematic.

---

Re the "optional" argument: biblatex is fine with using an empty optional argument [], but I'm not sure if that makes sense.

[plk]

Very odd, this is what I get for your second test case:

[moewew]

Very odd indeed, I guess we'll have to assume that something is wrong at my end until someone comes along and can reproduce the issue ...

I assume the auxiliary files I described above look right?

[plk]

Those files look ok. Every citation will always appear in all sortlists in the .bbl. This is by design so that biber does all the heavy work and biblatex can just do a lookup into the right context when it needs to. It looks like you don't have the latest versions somehow - no biber warnings about mismatched .bcf versions? Are you able to pull/build from git dev - that's always the most up-to-date.

[moewew]

I have yet to manage to build Biber from source, but I have pulled all biblatex files from dev now and still have the problem.

The first \cite gives me "[0,0]" the one after \printbibliography gives [1,2] as expected.

[plk]

You don't need to build biber if you have perl 5.18 or later installed - you can just use it as is by cloning the repo, running "perl Build.PL" and then "./Build installdeps" and then "./Build install". As long as the resulting "biber" perl program is first in your $PATH, you'll pick up the "raw perl " version and always be up to date. I never use the "built" PAR::Packer version, only this way.

[plk]

We can compare checksums - my biblatex_.sty gives:

MD5 (biblatex_.sty) = 1adcefb197105c0529722567b447b7df

[moewew]

Looks good.

1a dc ef b1 97 10 5c 05 29 72 25 67 b4 47 b7 df

[plk]

Can you find the bbl.pm you are using in biber? Mine is:

MD5 (lib/Biber/Output/bbl.pm) = 99ac4de3c5502253cd2185a8194e96cd

[moewew]

That one is different, I still have the one from sourceforge. Maybe we should compare the .bbl files for my initial example

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[style=numeric,backend=biber,defernumbers=true]{biblatex}

\addbibresource{biblatex-examples.bib}

\begin{document}
\newrefcontext[labelprefix=A]
\cite{sigfridsson,worman}
\printbibliography
\cite{sigfridsson,worman}
\end{document}

I get

MD5 hash of ./initialexample.bbl:
ae 8d 21 46 b9 77 b3 bb fe e7 e3 82 9b de 41 fe 28 4a fd e2

[plk]

Looks like it's your biber version:

MD5 (../btest/test1.bbl) = b8cb9deeb2b31504b61c5fc382b70999

That's the difficulty of keeping up with dev changes that require biblatex/biber to be in sync.

[moewew]

Aha! Since I still can't get my head around Perl and building I will wait for a new version of the binaries on sourceforge and then report back.

[plk]

It's a bit strange as I updated the binaries yesterday - perhaps your SF mirror isn't up to date? It should say how old they are on SF.

[moewew]

Mhh, I could imagine that happening once, but since we started this discussion I have downloaded Biber three times from sourceforge all with the same result, the zip contains a file last modified yesterday around five pm.

Just for completeness, here is the full .bbl

% $ biblatex auxiliary file $
% $ biblatex bbl format version 2.7 $
% Do not modify the above lines!
%
% This is an auxiliary file used by the 'biblatex' package.
% This file may safely be deleted. It will be recreated by
% biber as required.
%
\begingroup
\makeatletter
\@ifundefined{ver@biblatex.sty}
  {\@latex@error
     {Missing 'biblatex' package}
     {The bibliography requires the 'biblatex' package.}
      \aftergroup\endinput}
  {}
\endgroup

\refsection{0}
  \sortlist[entry]{nty/global/A}
    \entry{sigfridsson}{article}{}
      \name{author}{2}{}{%
        {{hash=484061f383fb3b729627e12ab42c1963}{%
           family={Sigfridsson},
           family_i={S\bibinitperiod},
           given={Emma},
           given_i={E\bibinitperiod}}}%
        {{hash=b7e299b632e5db12681c2decc8ce023f}{%
           family={Ryde},
           family_i={R\bibinitperiod},
           given={Ulf},
           given_i={U\bibinitperiod}}}%
      }
      \strng{namehash}{d9015d9f08448ab0cb194bd964d7b44b}
      \strng{fullhash}{d9015d9f08448ab0cb194bd964d7b44b}
      \field{sortinit}{S}
      \field{sortinithash}{fd1e7c5ab79596b13dbbb67f8d70fb5a}
      \field{labelprefix}{A}
      \field{labelnamesource}{author}
      \field{labeltitlesource}{title}
      \field{abstract}{Four methods for deriving partial atomic charges from the quantum chemical electrostatic potential (CHELP, CHELPG, Merz-Kollman, and RESP) have been compared and critically evaluated. It is shown that charges strongly depend on how and where the potential points are selected. Two alternative methods are suggested to avoid the arbitrariness in the point-selection schemes and van der Waals exclusion radii: CHELP-BOW, which also estimates the charges from the electrostatic potential, but with potential points that are Boltzmann-weighted after their occurrence in actual simulations using the energy function of the program in which the charges will be used, and CHELMO, which estimates the charges directly from the electrostatic multipole moments. Different criteria for the quality of the charges are discussed.}
      \field{annotation}{An \texttt{article} entry with \texttt{volume}, \texttt{number}, and \texttt{doi} fields. Note that the \textsc{doi} is transformed into a clickable link if \texttt{hyperref} support has been enabled}
      \field{indextitle}{Methods for deriving atomic charges}
      \field{journaltitle}{Journal of Computational Chemistry}
      \field{langid}{english}
      \field{langidopts}{variant=american}
      \field{number}{4}
      \field{title}{Comparison of methods for deriving atomic charges from the electrostatic potential and moments}
      \field{volume}{19}
      \field{year}{1998}
      \field{pages}{377\bibrangedash 395}
      \range{pages}{19}
      \verb{doi}
      \verb 10.1002/(SICI)1096-987X(199803)19:4<377::AID-JCC1>3.0.CO;2-P
      \endverb
    \endentry
    \entry{worman}{book}{}
      \name{author}{1}{}{%
        {{hash=045c9cb614b0ed4bfdd1e37d400e2a7b}{%
           family={Worman},
           family_i={W\bibinitperiod},
           given={Nancy},
           given_i={N\bibinitperiod}}}%
      }
      \list{location}{1}{%
        {Austin}%
      }
      \list{publisher}{1}{%
        {University of Texas Press}%
      }
      \strng{namehash}{045c9cb614b0ed4bfdd1e37d400e2a7b}
      \strng{fullhash}{045c9cb614b0ed4bfdd1e37d400e2a7b}
      \field{sortinit}{W}
      \field{sortinithash}{99e3ba1b3f78bb6f073e7fa7ac11636b}
      \field{labelprefix}{A}
      \field{labelnamesource}{author}
      \field{labeltitlesource}{shorttitle}
      \field{annotation}{A \texttt{book} entry. Note the \texttt{sorttitle} and \texttt{indextitle} fields}
      \field{indextitle}{Cast of Character, The}
      \field{langid}{english}
      \field{langidopts}{variant=american}
      \field{shorttitle}{Cast of Character}
      \field{subtitle}{Style in {Greek} Literature}
      \field{title}{The Cast of Character}
      \field{year}{2002}
    \endentry
  \endsortlist
  \sortlist[entry]{nty/global/}
    \entry{sigfridsson}{article}{}
      \name{author}{2}{}{%
        {{hash=484061f383fb3b729627e12ab42c1963}{%
           family={Sigfridsson},
           family_i={S\bibinitperiod},
           given={Emma},
           given_i={E\bibinitperiod}}}%
        {{hash=b7e299b632e5db12681c2decc8ce023f}{%
           family={Ryde},
           family_i={R\bibinitperiod},
           given={Ulf},
           given_i={U\bibinitperiod}}}%
      }
      \strng{namehash}{d9015d9f08448ab0cb194bd964d7b44b}
      \strng{fullhash}{d9015d9f08448ab0cb194bd964d7b44b}
      \field{sortinit}{S}
      \field{sortinithash}{fd1e7c5ab79596b13dbbb67f8d70fb5a}
      \field{labelnamesource}{author}
      \field{labeltitlesource}{title}
      \field{abstract}{Four methods for deriving partial atomic charges from the quantum chemical electrostatic potential (CHELP, CHELPG, Merz-Kollman, and RESP) have been compared and critically evaluated. It is shown that charges strongly depend on how and where the potential points are selected. Two alternative methods are suggested to avoid the arbitrariness in the point-selection schemes and van der Waals exclusion radii: CHELP-BOW, which also estimates the charges from the electrostatic potential, but with potential points that are Boltzmann-weighted after their occurrence in actual simulations using the energy function of the program in which the charges will be used, and CHELMO, which estimates the charges directly from the electrostatic multipole moments. Different criteria for the quality of the charges are discussed.}
      \field{annotation}{An \texttt{article} entry with \texttt{volume}, \texttt{number}, and \texttt{doi} fields. Note that the \textsc{doi} is transformed into a clickable link if \texttt{hyperref} support has been enabled}
      \field{indextitle}{Methods for deriving atomic charges}
      \field{journaltitle}{Journal of Computational Chemistry}
      \field{langid}{english}
      \field{langidopts}{variant=american}
      \field{number}{4}
      \field{title}{Comparison of methods for deriving atomic charges from the electrostatic potential and moments}
      \field{volume}{19}
      \field{year}{1998}
      \field{pages}{377\bibrangedash 395}
      \range{pages}{19}
      \verb{doi}
      \verb 10.1002/(SICI)1096-987X(199803)19:4<377::AID-JCC1>3.0.CO;2-P
      \endverb
    \endentry
    \entry{worman}{book}{}
      \name{author}{1}{}{%
        {{hash=045c9cb614b0ed4bfdd1e37d400e2a7b}{%
           family={Worman},
           family_i={W\bibinitperiod},
           given={Nancy},
           given_i={N\bibinitperiod}}}%
      }
      \list{location}{1}{%
        {Austin}%
      }
      \list{publisher}{1}{%
        {University of Texas Press}%
      }
      \strng{namehash}{045c9cb614b0ed4bfdd1e37d400e2a7b}
      \strng{fullhash}{045c9cb614b0ed4bfdd1e37d400e2a7b}
      \field{sortinit}{W}
      \field{sortinithash}{99e3ba1b3f78bb6f073e7fa7ac11636b}
      \field{labelnamesource}{author}
      \field{labeltitlesource}{shorttitle}
      \field{annotation}{A \texttt{book} entry. Note the \texttt{sorttitle} and \texttt{indextitle} fields}
      \field{indextitle}{Cast of Character, The}
      \field{langid}{english}
      \field{langidopts}{variant=american}
      \field{shorttitle}{Cast of Character}
      \field{subtitle}{Style in {Greek} Literature}
      \field{title}{The Cast of Character}
      \field{year}{2002}
    \endentry
  \endsortlist
\endrefsection
\endinput

[plk]

That's the correct timestamp. Ok, I'm going to add the git commit ID as a debug version stamp to help this. Give me fifteen mins.

[dcpurton]

@moewew I've mastered a good system for building and testing both biblatex and biber that you might be interested in: It doesn't involve me installing anything to either my system or personal texmf tree and I can quickly and easily move back and forward from version to version:

(Below assumes you're running bash as a shell - sorry - no good for Windows)

Here's what I do:

For biblatex (e.g. if I want to test against 3.3 and the current dev version)

git checkout 3.3
obuild/build.sh build 3.3
mkdir texmf-3.3
obuild/build.sh install 3.3 texmf-3.3
git checkout dev
obuild/build.sh build 3.4
mkdir texmf-dev
obuild/build.sh install 3.4 texmf-dev

I now have both 3.3 and 3.4dev build and installed in subdirectories of the git tree

I can then quickly update the dev build by just running

git pull
obuild\build.sh build 3.4
obuild\build.sh install 3.4 texmf-dev

For biber (e.g. if I want to test against 2.4 and the current dev version)

(make sure you have appropriate prerequisite perl packages installed first)

git checkout v2.4
perl Build.PL
./Build build
mkdir blib-2.4
mv blib blib-2.4/
git checkout dev
perl Build.PL
./Build build
mkdir blib-dev
ln -s blib blib-dev/blib

I now have both 2.4 and 2.5dev installed in my biber git tree.

Once again, I can update the dev build with a simple

git pull
./Build distclean
perl Build.PL
./Build build

The magic shell script that sets the build environment up:

This is in my path (called setenv_biblatex_dev.sh).

#!/bin/bash

BIBLATEXPATH="/home/dcpurton/development/biblatex/texmf-$1/tex/latex/biblatex"
BIBERPATH="/home/dcpurton/development/biber/blib-$2"

TEXINPUTS="$BIBLATEXPATH//:"

export TEXINPUTS=$TEXINPUTS
export BIBERPATH

biber() {
    perl -Mblib=$BIBERPATH $BIBERPATH/blib/script/biber "$@"
}

export -f biber

bash --rcfile <(echo "PS1='\[\033[01;33m\]biblatex-$1::biber-$2\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '") -i

Usage

setenv_biblatex_dev.sh biblatexversion biberversion

e.g., to test the dev version, I run: setenv_biblatex_dev.sh dev dev

(Exit the subshell to return to default settings in your normal texmf tree)

to test biblatex 3.3, I run: setenv_biblatex_dev.sh 3.3 2.4

(again exit the subshell to return to normal state).

The prompt is changed so I now what version I'm running.

Warnings

This is a complete hack and the shell script is pretty unforgiving... Make sure you always give it the right arguments as no error checking is done.

There may also be other problems I don't know about...

You'll need to change the values of BIBLATEXPATH and BIBERPATH in the script to point to where ever you git clone is for each one.

Anyway, posted here in case you find it useful.

[moewew]

@dcpurton Thank you very much. Unfortunately, I'm running Windows, so I will have to make do with my manual installation to a local folder.

But while you are here, may I ask what you get from the MWEs above with the most recent dev versions of biblatex and Biber?

[dcpurton]

Ah Windows.. could be a good time to move to Linux ;)

I think I see the same behaviour as @moewew in the first case, right?

Here's my output for your three MWE's above. I build them with pdflatex

---

---

---

---

[plk]

I'm getting confused here - which is the MWE which does not work for both of you?

[dcpurton]

My 3 images are in order of this thread. I assumed the problem is with the first one print [0, 0]. The others seem to look reasonable to me. But II confess I don't know the expected output.

[plk]

Hmm, I get the right results in all cases. I am going to update biblatex/biber and put idents into all biber files so we can see exactly which files are being used.

[dcpurton]

It could be some quirk of how I'm running biber or my local perl setup (up to date debian testing with three bonus perl modules that biber required [libencode-eucjpascii-perl_0.03; libmozilla-ca-perl_20160104; libunicode-normalize-perl_1.25]. I built these bonus modules as debian packages with dh_make_perl. A potential problem is that libunicode-normalize-perl is actually already part of a base perl package, so the wrong things might have been overwritten).

Biber gets run (essentially) as: perl -Mblib=/home/dcpurton/development/biber /home/dcpurton/development/biber/blib/script/biber

It reports version 2.5 (beta)

[plk]

Very much doubt it's anything to do with that - the problem looks like refcontext issues which were fixed a while ago.

[plk]

perhaps @dcpurton, you can also post here the checksum of the bbl.pm you are using?

[dcpurton]

99ac4de3c5502253cd2185a8194e96cd

[plk]

Hmm, I've uploaded new binaries to SF for biber 2.5 - perhaps @moewew can try this? To be sure here, this is what give you both problems after pdflatex, biber, pdflatex, pdflatex?

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[style=numeric,backend=biber,defernumbers=true]{biblatex}

\addbibresource{biblatex-examples.bib}

\begin{document}
\newrefcontext[labelprefix=A]
\cite{sigfridsson,worman}
\printbibliography
\cite{sigfridsson,worman}
\end{document}

[moewew]

@dcpurton Thanks for getting involved here. What I find really interesting is that you get problematic output only once. What is even more puzzling is that the first and third MWE should be the same, yet the error occurs only with the first for you.

@plk Just downloaded the current dev build from sourceforge, unfortunately Biber aborts on me every time complaining it can't

mkdir C:\Users\<UserName>\AppData\Local\Temp\par-4d6f7269747a\cache-f7c28b31c34d92d30110e8b78dca242b7131aa1b\inc\lib\version\: File exists; 
at /loader/HASH(0x257daf4)/Archive/Zip/Archive.pm line 198.

[plk]

Hmm, I am trying a new packer version - which version did you download, win32 or win64?

[moewew]

I used the 32 bit versions as always, just trying with 64 bit.

[moewew]

Now that is a surprise, the 64 bit version runs. The output of my MWE is unchanged, though, I still get [0, 0] for the first \cite command. No changes on the second MWE either. MD5 of .bbl is unchanged.

[plk]

I realised that the MD5 thing is useless unless you are using the raw perl version because the packing routines change the source files by stripping comments etc. This is really mysterious. can you create checksums for the kpsewhich reported biblatex_.sty and biblatex_.def files?

[moewew]

Does your .bbl look as the one I posted above or is there a diff?

MD5s for the files (freshly pulled from git) are

./biblatex_.def:
f1 e3 a8 1e 83 10 84 38 99 92 ea 9d 4b 45 77 e0 95 22 6c ee
./biblatex_.sty:
18 ea a9 3d 6f be ae c9 8c 43 5f f9 4c fa a2 83 9e 29 e8 e5

[plk]

.bbl is identical so it must be biblatex. your hashes are different. Can you email me your biblatex_.sty?

[moewew]

Sorry, I'm an idiot. I hashed using SHA1, MD5 is as above

1a dc ef b1 97 10 5c 05 29 72 25 67 b4 47 b7 df

[plk]

Ah, that's correct ... Anyway, where are you getting the dev zip? which page? This can be a little bit out of date depending. I am mostly updating the development folder TDS and flat dists on Sourceforge - can you try one of those?

[moewew]

github offers to download the branch as .zip, that's what I used here. (Normally I rely on the sourceforge snapshots, but if git has some newer files I go with that.) The result is the same with sourceforge files.

[plk]

Email your _.sty and _.def to me at the address at the top of the biber .pdf doc and I'll diff them with mine.

[moewew]

I now know why you can't reproduce the problem and I should have thought about that earlier.

The behaviour is similar to https://github.com/plk/biblatex/issues/386

1. pdflatex: undefined refs
2. Biber
3. pdflatex: [0, 0]
4. pdflatex: [A2, A2]
5. pdflatex: [0, 0]
6. pdflatex: [0, 0]

etc. pp.

You probably stop at step 4, while I used latexmk and went one step further to 5 and every subsequent compile run didn't help us recover.

[plk]

Are you prompted for any more runs after step 4?

[moewew]

No, the fourth step ends without warnings. I wonder why latexmk even does another run because neither the .aux, .bbl nor .bcf files change from after run 3 to after run 4.

Interestingly running Biber with old temporary files present does not restart the above sequence at 2., rather we are immediately landed at 5. and can't recover. I only managed to get proper output after trashing the auxiliaries.

[plk]

The .aux file situation with refcontext and defernumbers is quite complex. It's supposed to be ok when there are no more re-run messages - do you have the latest latexmk?

[moewew]

latexmk version is 4.44.

[plk]

The question is whether this is bad behaviour of biblatex. I was under the impression that when the document reaches a state where there are no re-run messages, it's finished. Further runs may well do odd things when there are .aux dependencies. What do you think?

[moewew]

I think the current situation is undesirable. I could only get biblatex to work properly again after trashing all auxiliary files. Since people often write a bit then compile and write another bit this would make life really hard for such a workflow.

[plk]

Could you get the win32 biber from SF again and try it? There was a bug in the new release of a module I use to build it. Hopefully this is fixed now.

[moewew]

Biber unpacks properly again, in both 32 and 64 bit.

[dcpurton]

My results are the same as @moewew (I must have accidentally run pdflatex an extra time when I first tested the MWE). Good pick up. But the latest biber dev does not fix things for me. An extra run of pdflatex still produces [0, 0]. Just to make sure I checked with biber from sourceforge as well as from git. I agree with @moewew that this is undesirable. I wouldn't expect different output on subsequent runs of *latex.

[plk]

I'm looking at this - the biber update wasn't intended to fix this issue, that was something else entirely and to do with an upstream module I depend on.

[plk]

Should be fixed now in 3.4 dev.