Support for biber

[shiblon]

Originally reported on Google Code with ID 113

Biber is a UTF-8 able replacement for Bibtex for use with package biblatex.  The current
system for handling bibliographies doesn't work with the combination of these two,
since if biblatex is supplied option backend=biber, it stops writing bibliography information
to .aux and instead writes it to a .bcf file.

http://biblatex-biber.sourceforge.net/
http://www.ctan.org/tex-archive/help/Catalogue/entries/biblatex.html

It will need more thinking than I've done to solve the issue (ie. let bibtex and biber
coexist), but I provide my dirty hack as a starting point.  First of all

-----
BIBTEX := biber
-----

and then in the rule for creating .d, after the get-bibs line:

-----
if [ -n $*.bcf ]; then \
    grep datasource $*.bcf | sed -e 's/^[^>]*>//g' \
        -e 's/<.*$$//g' -e '/\.bib$$/!s/$$/.bib/' \
        -e 's!^!$(addprefix $*.,bbl aux aux.make): !' >> $*.d; \
fi; \
-----

This solves the simple case and allows me to return to writing my thesis ;)
I can provide more info / files if needed.

Reported by jaripekka.juhala on 2011-02-10 19:49:57

[shiblon]

Interesting.  So, essentially, you just changed BIBTEX to biber, and changed $*.aux
to $*.bcf?  Does that sum it up correctly?

Reported by shiblon on 2011-03-02 20:30:46

[shiblon]

I'd love for a bit more detail here, not sure how to fit your snippet into the makefile
and would like to use Biber.

Reported by jebarker on 2011-03-20 07:20:54

[shiblon]

Me, too :)

Reported by shiblon on 2011-03-21 14:21:39

[shiblon]

Thanks, this works. Just do exactly what he described. Put his snippet behind

----
    $(call get-bibs,$*.aux.make,$(addprefix $*.,bbl aux aux.make)) >> $*.d; \
----
this line.

Reported by tobias.neumann on 2011-08-01 10:39:14

[shiblon]

OK - before I jump in and make this change, I want to be absolutely sure that I understand
what is going on (fiddling with bibtex is fraught with peril in the makefile, and it
is useful to chart a course carefully first).

Let's make sure that I know the facts.

- the bibtex binary is still used, but with a special option backend=biber?  Or is
there a biber binary?
- Is biber "latex source compatible" with bibtex?  In other words, does it still use
commands like \bibliography and \bibliographystyle?
- Biber writes bib information into .bcf instead of .aux, but it is exactly the same
information?

So far so good?

Reported by shiblon on 2011-08-03 19:01:39

[shiblon]

- the bibtex binary is still used, but with a special option backend=biber?  Or is >
there a biber binary?
Both. There is a biber binary, but one can also use bibtex. In order to use biber one
has to load the package biblatex with the option backend=biber. Only that produces
a .bcf file. Without this option, no .bcf file is produced.

- Is biber "latex source compatible" with bibtex?  In other words, does it still use
commands like \bibliography and \bibliographystyle?
I think some effort has been made to make it backwards compatible, but the default
is now to use different commands to produce bibliographies (e.g., \printbibliograph).

- Biber writes bib information into .bcf instead of .aux, but it is exactly the same
information?
It does write to .bcf; I doubt that it is the same information, but haven't checked.

Reported by philipp.woelfel on 2011-09-19 05:59:13

[shiblon]

Wow, it's been a while since I looked at this.

I think that the patch is wrong, too.  The test [ -n $*.bcf ] will *always* be true,
so the code will *always* execute.  That can't be right.  I think it's possible that
[ -s '$*.bcf' ] was intended instead, but can't be sure until I've had a better explanation
of what this code snippet is supposed to do.

Thoughts and explanations welcome.

Reported by shiblon on 2011-11-11 19:18:22

[shiblon]

I'll look into this.

Reported by Holger.Dell on 2012-05-16 18:57:13

• Labels added: Type-Enhancement
• Labels removed: Type-Defect

[shiblon]

Somehow the biber hack doesn't work for me. It doesn't matter whether I insert it as
posted originally or the proposed fix by shiblon, I still get the "Please (re)run Biber
on the file" message without another run being triggered :/

The *.bib files are all listed in the .d file but there doesn't seem to be a recognition
of undefined references.

Reported by pinkbunny43 on 2012-09-11 20:12:06

[shiblon]

What is strange though is the fact that not even touching one of the bib files triggers
biber. I still get these, too:
I found no \citation commands---while reading file main.aux
I found no \bibdata command---while reading file main.aux
I found no \bibstyle command---while reading file main.aux

Could this be a problem of 2.2.1-alpha9 or am I just being stupid?

Reported by pinkbunny43 on 2012-09-11 21:31:34

[shiblon]

... [ -s $*.bcf ] ... works fine for me. Since -n .. doesn't make sense, -s seems most
suitable: the command should not be executed if there is no bcf file or an empty file.

As jaripekk..., I'm using the biber command instead of bibtex.

Reported by jfgrote on 2013-01-08 15:47:15

[shiblon]

Reported by Holger.Dell on 2013-05-21 16:52:02

[llbit]

I am also using biber, with the Makefile additions suggested above. This seems to work fine for my thesis.

Biber version 2.7 (beta).

I use XeLaTeX:

XeTeX 3.14159265-2.6-0.99998 (TeX Live 2017/Debian)

Makefile version: 2.2.1-alpha10

Here is the diff:

diff --git a/Makefile b/Makefile
index ea4dc52..ad0edc9 100644
--- a/Makefile
+++ b/Makefile
@@ -3334,6 +3334,11 @@ endif
        $(call get-graphics,$*) >> $*.d; \
        $(call get-log-index,$*,$(addprefix $*.,aux aux.make)) >> $*.d; \
        $(call get-bibs,$*.aux.make,$(addprefix $*.,bbl aux aux.make)) >> $*.d; \
+if [ -s $*.bcf ]; then \
+    grep datasource $*.bcf | sed -e 's/^[^>]*>//g' \
+        -e 's/<.*$$//g' -e '/\.bib$$/!s/$$/.bib/' \
+        -e 's!^!$(addprefix $*.,bbl aux aux.make): !' >> $*.d; \
+fi; \
        $(EGREP) -q "# MISSING stem" $*.d && $(SLEEP) 1 && $(RM) $*.pdf; \
        $(EGREP) -q "# MISSING format" $*.d && $(RM) $*.pdf; \
        $(call move-if-exists,$*.$(build_target_extension),$*.$(build_target_extension).1st.make); \

And Makefile.ini:

BUILD_STRATEGY := xelatex
BIBTEX := biber

[shiblon]

Hey, @llbit , thanks for the update! I altered the sed a bit (removing the need for grep), moved it into its own file, and used the sed inclusion mechanism to pull it into the makefile under get-bcf-bibs. Please take a look at commit ef16cd4cd4c902fd35d8d0096b1285c33d2046ee, run the ./build script, and se if the resulting Makefile does what it is supposed to do.

[llbit]

@shiblon I tried your changes but unfortunately the bibliographies disappear when I rebuild with the Makefile from commit ef16cd4cd4c902fd35d8d0096b1285c33d2046ee.

I have all my TeX code in this repository if you want to try reproducing the issue: https://github.com/llbit/joqvist-thesis

I build my thesis with this command: make clean; make thesis. When I build it with my edited Makefile I get 205 pages, when I build with your updated version I get 193 pages because the bibliographies are missing.

[shiblon]

That seems pretty odd. Is it just not running latex enough times, perhaps? I'll take a look and see if I can make it happen over here.

I wonder what the difference is between the changes I made and the changes you suggested. As I recall, they were pretty similar...

On Sat, Feb 2, 2019 at 10:39 AM Jesper Öqvist notifications@github.com wrote:

@shiblon https://github.com/shiblon I tried your changes but unfortunately the bibliographies disappear when I rebuild with the Makefile from commit ef16cd4 https://github.com/shiblon/latex-makefile/commit/ef16cd4cd4c902fd35d8d0096b1285c33d2046ee .

I have all my TeX code in this repository if you want to try reproducing the issue: https://github.com/llbit/joqvist-thesis

I build my thesis with this command: make clean; make thesis. When I build it with my edited Makefile I get 205 pages, when I build with your updated version I get 193 pages because the bibliographies are missing.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/shiblon/latex-makefile/issues/126#issuecomment-459974667, or mute the thread https://github.com/notifications/unsubscribe-auth/ACPH8EF8DaUlWzaGcMv8-4_Bcht76DfHks5vJbFNgaJpZM4Yw3tY .

[k4rtik]

@llbit your patch shown in https://github.com/shiblon/latex-makefile/issues/126#issuecomment-441275351 works for me with the default build strategy. Thanks for sharing.

[shiblon]

OK, it looks like what happened is the Makefile wasn't regenerated in commit ef16cd4 (in fact, the generated Makefile isn't in the source repo).

Did you try regenerating on commit ef16cd4? That should have done the trick. Running the ./build script from the repo root should get you there.

It's probably a good idea for me to put the generated Makefile into the repo, though, so I'm committing that change now. Try the Makefile from commit d4770f94a4bf5cee1323094b8a6d92bcbe3aa5d6.

[k4rtik]

I am sure I had to build the Makefile on first time use. I tried the Makefile you committed and it gives me the following output with a suspicious sed error:

$ make
NOTE: You may ignore warnings about the following files:

     main.d

Makefile:2894: main.d: No such file or directory
= main.tex --> main.d main.pdf.1st.make (0-1) =
sed: can't read main: No such file or directory
= main.tex --> main.pdf (1-2) =
= main.tex --> main.pdf (1-3) =
= main.tex --> main.pdf (1-4) =
= main.tex --> main.pdf (1-5) =
LaTeX Warning: Citation 'sup19' on page 2 undefined on input line 193.
[...]
LaTeX Warning: Citation 'qhtt20' on page 26 undefined on input line 706.

LaTeX Warning: Empty bibliography on input line 711.

Underfull \vbox (badness 5832) has occurred while \output is active []

LaTeX Warning: There were undefined references.

Package biblatex Warning: Please (re)run Biber on the file:
(biblatex)                main
(biblatex)                and rerun LaTeX afterwards.

Success!  Wrote 30 pages, 288102 bytes to main.pdf

I am on macOS if it helps. The only difference in the two makefiles is the following:

-if [ -s $*.bcf ]; then \
-    grep datasource $*.bcf | sed -e 's/^[^>]*>//g' \
-        -e 's/<.*$$//g' -e '/\.bib$$/!s/$$/.bib/' \
-        -e 's!^!$(addprefix $*.,bbl aux aux.make): !' >> $*.d; \
-fi; \
+       $(call get-bcf-bibs,$*,$(addprefix $*.,bbl aux aux.make)) >> $*.d; \

I get a clean build after reverting your change to the previous version:

$ make
NOTE: You may ignore warnings about the following files:

     main.d

Makefile:2894: main.d: No such file or directory
= main.tex --> main.d main.pdf.1st.make (0-1) =
= references.bib main.aux --> main.bbl =
= main.tex --> main.pdf (1-2) =
= main.tex --> main.pdf (1-3) =
Underfull \vbox (badness 10000) has occurred while \output is active []
[...]
Underfull \vbox (badness 10000) has occurred while \output is active []

Success!  Wrote 33 pages, 320484 bytes to main.pdf

[shiblon]

Question, when you say "previous version", does that mean there is a version that works, but that I broke it recently?

Can you help me know which version that would be?

[shiblon]

@k4rtik , I have committed dcf476152a5e46b5a14f59c37b448af17be29e39 which might help - can you give it a try and see if it fixes anything? I don't foresee it making a difference, but it might. If you also happen to have that "previous version" that works, let me know and we'll figure this out.

[k4rtik]

Sorry, I realize how calling the version I had "previous" could be confusing. I was referring to the stable Makefile generated from your repo + @llbit's patch applied as I mentioned in https://github.com/shiblon/latex-makefile/issues/126#issuecomment-733803367.

I tried your new commit on the bcf branch but it does not change the error. sed still complains that it can't read main and no .bbl file is generated.

[shiblon]

Can you post a full error please?

[k4rtik]

Here:

$ make clean
$ make
NOTE: You may ignore warnings about the following files:

     main.d

Makefile:2896: main.d: No such file or directory
= main.tex --> main.d main.pdf.1st.make (0-1) =
sed: can't read main: No such file or directory
= main.tex --> main.pdf (1-2) =
= main.tex --> main.pdf (1-3) =
= main.tex --> main.pdf (1-4) =
= main.tex --> main.pdf (1-5) =
LaTeX Warning: Citation 'sup19' on page 2 undefined on input line 193.

[...]

LaTeX Warning: Citation 'wpe2016' on page 3 undefined on input line 205.

Underfull \vbox (badness 10000) has occurred while \output is active []

LaTeX Warning: Citation 'nan2008' on page 4 undefined on input line 207.

[...]

LaTeX Warning: Citation 'gri1981' on page 4 undefined on input line 211.

Overfull \hbox (14.73712pt too wide) in paragraph at lines 211--212
\T1/ptm/m/n/10.95 grams such as [] [[]] and [] [[]] in the spirit of [],
 []

LaTeX Warning: Citation 'hoa1969' on page 4 undefined on input line 221.

LaTeX Warning: Citation 'flo67' on page 4 undefined on input line 221.

Underfull \vbox (badness 10000) has occurred while \output is active []

LaTeX Warning: Citation 'nan2008' on page 6 undefined on input line 285.

[...]

LaTeX Warning: Citation 'un2019' on page 14 undefined on input line 521.

Underfull \vbox (badness 1642) has occurred while \output is active []

Underfull \vbox (badness 10000) has occurred while \output is active []

LaTeX Warning: Citation 'dun2019' on page 23 undefined on inp
ut line 668.

LaTeX Warning: Citation 'nan2008' on page 23 undefined on input line 668.

LaTeX Warning: Citation 'abs07' on page 23 undefined on input line 668.

Overfull \hbox (89.53943pt too wide) in paragraph at lines 668--669
[]\T1/ptm/m/n/10.95 We would like to men-tion that the type sys-tem pre-sented
in this doc-u-ment is not bidi-rec-tional [[]]
 []

LaTeX Warning: Citation 'un2019' on page 25 undefined on input line 689.

[...]

LaTeX Warning: Citation 'qhtt20' on page 26 undefined on input line 706.

LaTeX Warning: Empty bibliography on input line 711.

Underfull \vbox (badness 5832) has occurred while \output is active []

LaTeX Warning: There were undefined references.

Package biblatex Warning: Please (re)run Biber on the file:
(biblatex)                main
(biblatex)                and rerun LaTeX afterwards.

Success!  Wrote 30 pages, 288102 bytes to main.pdf

EDIT: edited out unnecessary detail/output.

[shiblon]

Thank you! Found it. Let's try commit bc1162568c9b6843eabfe6043bb48f042401325e

Thanks for taking the time to try these out. Very helpful!

[shiblon]

If you would like to post a bcf file (sanitized, if you desire) for me to use as a test, I can write one up that will ensure this continues working :)

[k4rtik]

With the commit https://github.com/shiblon/latex-makefile/commit/bc1162568c9b6843eabfe6043bb48f042401325e the sed error goes away, but I still get the warning about undefined references and an empty (0-byte) main.bbl.

I am not sure what you mean by sanitized, but I posted one at https://gist.github.com/k4rtik/8af925f9ea25df7166351b217b8932d1

[shiblon]

Thanks @k4rtik that's quite useful. I just put a test in there to ensure that parsing is happening properly. It looks like it is.

Now, as to the empty main.bbl, that's an interesting problem, and I think it's actually unrelated to this bcf change. It is possible that you need to delete main.bbl and try again - if it was already there from previous failed runs, that might trigger that particular error.

Sorry to go back and forth so much on this, but if you can try a clean make with the commit I posted, that might help. Barring that, try just building main.d and see what it contains. If it has the right "references.bib" line in it, then the bcf parser is working.

make main.d

would do that for you, without then deleting it afterward (as sometimes happens with make).

[k4rtik]

Each time I am sharing any log, I make sure to make clean first and I just double-checked with the latest commit that it's not the old main.bbl lying around. It is still generated empty.

I do see the following at the very bottom of main.d:

.SECONDEXPANSION:
main.bbl main.aux main.aux.make: references.bib

[k4rtik]

Oh, you may be right. I see that if I go back to my stable Makefile, even that is generating an empty .bbl file.

[k4rtik]

I confirm that the changes in the bcf branch work fine on my linux machine. I do recall seeing this empty .bbl issue on macOS a week ago, so I concur that it is unrelated and hence, this issue can be considered resolved.

[k4rtik]

I suspect my issue on macOS is some sort of file handle being left behind. I rebooted my machine and tested again and now it works fine.

Thanks so much for looking into and making these changes @shiblon. :)

[shiblon]

My pleasure, and sorry it was so long between fixes. Back in the game now 👍 Meanwhile, if you find something that can be better handled related to the bbl file issue, let's start a new bug (can reference this one).

Tagged with v2.2.1.