Makefile always rebuilds everything when package "biblatex" is present

[shiblon]

Originally reported on Google Code with ID 159

Observed in latex-makefile versions: 2.2.0 and 2.2.1-alpha8

Using the file test.tex:

\documentclass{article}
\usepackage{biblatex}
\begin{document}
Aa
\end{document}

Expected behavior: The first 'make' builds test.pdf, and subsequent calls of 'make'
will not try to rebuild anything.

Observed behavior: 'make' never reaches a fixed point and always rebuilds test.pdf.

Test case: When I remove "\usepackage{biblatex}", then the first and second invocation
of 'make' rebuild test.pdf, and starting with the third invocation 'make' reports that
nothing needs to be done. (This still deviates from the expected behavior, but at least
it does reach a fixed point)

Reported by yllohy on 2012-05-07 20:03:29

---

• Attachment: test.tex

[shiblon]

This actually happens in a lot of cases, and is unfortunately unavoidable in most of
them because there is no way to sanely generate an accurate dependency graph with LaTeX
(multiple runs required, sometimes with interstitial tools and a final latex run, etc.).

Sorry the news isn't better.

Reported by shiblon on 2012-05-09 11:16:45

• Status changed: WontFix

[shiblon]

Attached is an update that fixes this and two other bugs, and adds some feature. The
diff it to 2.2.1-alpha8. Let me know if you want separate patches or more information.

Explanation:
1) *.1st.make wasn't handled correctly in the main run-latex rule:
  - "$*.1st.*.make" is a typo that caused make to rebuild when it wasn't necessary.
  - the MV command should be CP as otherwise the 1st.make file would have to be rebuilt
in the next invocation
  - the 1st.make should be generated if it doesn't exist yet (I don't really know how
this could happen at all. Maybe because the main run-latex rule fires before the 1st.make
rule fires)
2) The two test-run-again commands found the line "Package: rerunfilecheck 2011/04/15
v1.7 Rerun checks for auxiliary files (HO)" in my log-file, which caused make to think
it had to rebuild, ad infinitum
3) To address the bug related to biblatex above, I added a check to the rule "%.bbl:
%.auxbbl.make" that tests whether the dependencies have been changed (rather than only
their timestamp). If they haven't, it suffices to touch the output file, and we don't
need to rebuild it.
4) I made a similar change to the rule that generates the 1st.make. This was necessary
because even on trivial files it took two invocations of make to reach the end.
5) I added some switchable echo commands before latex-run and bibtex-run happen, so
that we can figure out _why_ the Makefile thought it needed to rebuild something.

Reported by yllohy on 2012-05-09 22:39:18

---

• Attachment: Makefile
• Attachment: diff

[shiblon]

I'll go ahead and comment on the diff, reproduced here (surrounded by vertical whitespace):

diff --git a/Makefile b/Makefile
index 29cd3f2..9c03281 100644
--- a/Makefile
+++ b/Makefile
@@ -907,6 +907,13 @@ GRAY   ?= $(call get-default,$(GREY),)
 # Utility Functions and Definitions
 #

+# This is a DEBUG function that prints a message to justify why
+# something had to be rebuilt. Set ECHO_JUSTIFY to something to
+# activate
+ECHO_JUSTIFY ?=
+# $(call echo-justify,message)
+echo-justify                 = $(if $(ECHO_JUSTIFY),$(ECHO) $1,echo 1)

This looks interesting - seems like a useful addition.

+
 # Don't call this directly - it is here to avoid calling wildcard more than
 # once in remove-files.
 remove-files-helper    = $(if $1,$(RM) $1,$(sh_true))
@@ -979,6 +986,19 @@ test-different     = ! $(DIFF) -q '$1' '$2' >/dev/null 2>&1
 test-exists-and-different  = \
    $(call test-exists,$2) && $(call test-different,$1,$2)

+# This checks whether <file> has changed since the last time it was 'stored'
+# $(call has-changed,file)
+has-changed = ! $(call test-exists-and-different,$1,$1.previous.make)
+# This 'stores' <file> if it exists
+# $(call has-changed,file)
+store = $(call copy-if-exists,$1,$1.previous.make)
+# This checks whether any file in the list has changed
+# $(call has-changed,list of files)
+has-any-changed = $(sh_true) $(foreach file,$1, || $(call has-changed,$(file)))
+# This 'stores' all the files in the list
+# $(call store-all,list of files)
+store-all = $(sh_true) $(foreach file,$1, ; $(call store,$(file)))

As you have already discovered, the above won't work properly.  There are various reasons
for this, one of which is filesystem metadata caching that causes file changes to not
show up in a timely manner.  Another is execution order: many times you want to test
the freshness of a file, but by the time you do it's already too late because this
would have been executed while building the dependency graph.

Anyway, it's a thorny problem.

+
 # Return value 1, or value 2 if value 1 is empty
 # $(call get-default,<possibly empty arg>,<default value if empty>)
 get-default    = $(if $1,$1,$2)
@@ -2310,7 +2330,7 @@ $(SED) \
 enlarge_beamer = $(PSNUP) -l -1 -W128mm -H96mm -pletter

 # $(call test-run-again,<source stem>)
-test-run-again = $(EGREP) -q '^(.*Rerun .*|No file $1\.[^.]+\.)$$' $1.log
+test-run-again = $(EGREP) '^(.*Rerun .*|No file $1\.[^.]+\.)$$' $1.log | $(EGREP)
-q -v '^(Package: rerunfilecheck.*Rerun checks for auxiliary files.*)$$'

That's an interesting one - I hadn't seen that Package:rerunfilecheck thing before.
 How standard is that?

 # This tests whether the build target commands should be run at all, from
 # viewing the log file.
@@ -2319,7 +2339,8 @@ define test-log-for-need-to-run
 $(SED) \
 -e '/^No file $(call escape-fname-regex,$1)\.aux\./d' \
 $1.log \
-| $(EGREP) -q '^(.*Rerun .*|No file $1\.[^.]+\.|No file .+\.tex\.|LaTeX Warning: File.*)$$'
+| $(EGREP) '^(.*Rerun .*|No file $1\.[^.]+\.|No file .+\.tex\.|LaTeX Warning: File.*)$$'
\
+| $(EGREP) -q -v '^(Package: rerunfilecheck.*Rerun checks for auxiliary files.*)$$'
 endef

 # LaTeX invocations
@@ -2856,24 +2877,29 @@ endif
    run=0; \
    for i in 1; do \
        if $(call test-exists,$*.bbl.cookie); then \
+           $(call echo-justify,Running latex because $*.bbl.cookie is present); \
            run=1; \
            break; \
        fi; \
        if $(call test-exists,$*.run.cookie); then \
+           $(call echo-justify,Running latex because $*.run.cookie is present); \
            run=1; \
                break; \
        fi; \
        if $(call \
        test-exists-and-different,$*.auxtarget.cookie,$*.auxtarget.make);\
        then \
+           $(call echo-justify,Running latex because $*.auxtarget.cookie differs from $*.auxtarget.make);
\
            run=1; \
            break; \
        fi; \
        if $(call test-log-for-need-to-run,$*); then \
+           $(call echo-justify,Running latex because $*.log indicated that this is necessary);
\
            run=1; \
            break; \
        fi; \
-       if [ ! -e $*.1st.*.make ]; then \
+       if [ ! -e $@.1st.make ]; then \

I'm not sure why you think this was a typo.  It wasn't, so far as I remember: we definitely
want $*.1st.*.make (in particular, your change will break latex sub-includes, and I'm
not sure how $@ works at all, frankly).  Why is this change here?

+           $(call echo-justify,Running latex because $@.1st.make does not exist); \
            run=1; \
            break; \
        fi; \
@@ -2889,10 +2915,14 @@ endif
            ); \
            $(call run-latex,$*); \
            $(CP) '$*.log' '$*.'$(RESTARTS)-$$i'.log'; \
-           $(call test-run-again,$*) || break; \
+           if $(call test-run-again,$*); then \
+                   $(call echo-justify,latex rerun requested by $*.log); \
+           else break; \
+           fi; \

If you're going to do this, it's best to just call test-run-again once and use proper
shell conditions to do the echo justify.

        done; \
+       $(CP) '$@' '$@.1st.make'; \
    else \
-       $(MV) '$@.1st.make' '$@'; \
+       $(CP) '$@.1st.make' '$@'; \

This is simply wrong.  CP is incorrect here, because you want to remove the .1st file
in some cases (particularly if you have graphics dependencies, etc., that will trigger
a rerun of make with new .d files, you really don't want that .1st.make file hanging
around messing with the new dependency calculation).

I don't doubt that it helps to solve your particular problem, but in this case the
cure is worse than the disease for lots of other users.

    fi; \
    $(call copy-with-logging,$@,$(BINARY_TARGET_DIR)); \
    $(call latex-color-log,$*)
@@ -2911,8 +2941,14 @@ endif
 # don't need to be changed, thus possibly avoiding a rebuild trigger.
 %.bbl: %.auxbbl.make
    $(QUIET)\
+   if $(call has-any-changed,$^); then \
+       $(call store-all,$^); \
+   else \
+       $(TOUCH) $@; exit 0; \
+   fi; \
    $(if $(filter %.bib,$^),\
        $(call echo-build,$(filter %.bib,$?) $*.aux,$@); \
+       $(call echo-justify,Running bibtex because dependencies of $@ changed); \
        $(call run-bibtex,$*); \
        $(TOUCH) $@.cookie; \
    ) \
@@ -3163,8 +3199,14 @@ endif
 #  not creating a .1st.make file..
 #
 %.$(build_target_extension).1st.make %.d %.aux %.aux.make %.fls: %.tex
-   $(QUIET)$(call echo-build,$<,$*.d $*.$(build_target_extension).1st.make,$(RESTARTS)-1)
    $(QUIET)\
+   if $(call has-any-changed,$^); then \
+       $(call store-all,$^); \
+   else \
+       $(TOUCH) $@; exit 0; \
+   fi; \
+   $(call echo-build,$<,$*.d $*.$(build_target_extension).1st.make,$(RESTARTS)-1); \
+   $(call echo-justify,Running latex for the \"first\" time to build .d and .$(build_target_extension).1st.make);
\
    $(call run-latex,$*,-recorder) || $(sh_true); \
    $(CP) '$*.log' '$*.$(RESTARTS)-1.log'; \
    $(call die-on-import-sty,$*.log); \

As you have noticed, there are monsters, here.  Unless you have a very deep understanding
of how make is invoked, reinvoked, and the interaction between those invocations and
the loopy latex runs, you're going to run into trouble trying to calculate dependencies
this way.

This is one of the problems with an open source project like this: the level of understanding
required to even make small changes is enormous, so I don't get much in the way of
people able to contribute or take it over.  I've been doing this for a long time, and
I've been ready to let it go for a while, now, but nobody can take it.

Sigh.

Anyway, I'm afraid that having make do extra work is part of the game, here.  There's
no fixing it for all cases because we quickly run into the Halting Problem.  We can't,
for example, really tell what latex's includes will do until we have included and parsed
them, and then we're starting to ask questions like "will this program include this
file?"  You just can't answer that *in general*, because it's impossible.  The workarounds
for the common cases work well for the common cases, but increasingly the bug reports
I'm receiving are pushing the boundaries of the hack.

Given that it *is* a huge hack, I have to draw the line for increasingly complex fixes
somewhere.  This one, unless you've found a real bug, is firmly over that line.  :-(

Reported by shiblon on 2012-05-10 07:36:15