search

yochju / latex-makefile

Automatically exported from code.google.com/p/latex-makefile
Other
0 stars 0 forks source link

Makefile always rebuilds everything when package "biblatex" is present #159

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Observed in latex-makefile versions: 2.2.0 and 2.2.1-alpha8

Using the file test.tex:

\documentclass{article}
\usepackage{biblatex}
\begin{document}
Aa
\end{document}

Expected behavior: The first 'make' builds test.pdf, and subsequent calls of 
'make' will not try to rebuild anything.

Observed behavior: 'make' never reaches a fixed point and always rebuilds 
test.pdf.

Test case: When I remove "\usepackage{biblatex}", then the first and second 
invocation of 'make' rebuild test.pdf, and starting with the third invocation 
'make' reports that nothing needs to be done. (This still deviates from the 
expected behavior, but at least it does reach a fixed point)

Original issue reported on code.google.com by yll...@googlemail.com on 7 May 2012 at 8:03

Attachments:

GoogleCodeExporter commented 9 years ago 
This actually happens in a lot of cases, and is unfortunately unavoidable in 
most of them because there is no way to sanely generate an accurate dependency 
graph with LaTeX (multiple runs required, sometimes with interstitial tools and 
a final latex run, etc.).

Sorry the news isn't better.

Original comment by shiblon on 9 May 2012 at 11:16

GoogleCodeExporter commented 9 years ago 
Attached is an update that fixes this and two other bugs, and adds some 
feature. The diff it to 2.2.1-alpha8. Let me know if you want separate patches 
or more information.

Explanation:
1) *.1st.make wasn't handled correctly in the main run-latex rule:
  - "$*.1st.*.make" is a typo that caused make to rebuild when it wasn't necessary.
  - the MV command should be CP as otherwise the 1st.make file would have to be rebuilt in the next invocation
  - the 1st.make should be generated if it doesn't exist yet (I don't really know how this could happen at all. Maybe because the main run-latex rule fires before the 1st.make rule fires)
2) The two test-run-again commands found the line "Package: rerunfilecheck 
2011/04/15 v1.7 Rerun checks for auxiliary files (HO)" in my log-file, which 
caused make to think it had to rebuild, ad infinitum
3) To address the bug related to biblatex above, I added a check to the rule 
"%.bbl: %.auxbbl.make" that tests whether the dependencies have been changed 
(rather than only their timestamp). If they haven't, it suffices to touch the 
output file, and we don't need to rebuild it.
4) I made a similar change to the rule that generates the 1st.make. This was 
necessary because even on trivial files it took two invocations of make to 
reach the end.
5) I added some switchable echo commands before latex-run and bibtex-run 
happen, so that we can figure out _why_ the Makefile thought it needed to 
rebuild something.

Original comment by yll...@googlemail.com on 9 May 2012 at 10:39

Attachments:

GoogleCodeExporter commented 9 years ago 
I'll go ahead and comment on the diff, reproduced here (surrounded by vertical 
whitespace):

diff --git a/Makefile b/Makefile
index 29cd3f2..9c03281 100644
--- a/Makefile
+++ b/Makefile
@@ -907,6 +907,13 @@ GRAY   ?= $(call get-default,$(GREY),)
 # Utility Functions and Definitions
 #

+# This is a DEBUG function that prints a message to justify why
+# something had to be rebuilt. Set ECHO_JUSTIFY to something to
+# activate
+ECHO_JUSTIFY ?=
+# $(call echo-justify,message)
+echo-justify                 = $(if $(ECHO_JUSTIFY),$(ECHO) $1,echo 1)

This looks interesting - seems like a useful addition.

+
 # Don't call this directly - it is here to avoid calling wildcard more than
 # once in remove-files.
 remove-files-helper    = $(if $1,$(RM) $1,$(sh_true))
@@ -979,6 +986,19 @@ test-different     = ! $(DIFF) -q '$1' '$2' >/dev/null 2>&1
 test-exists-and-different  = \
    $(call test-exists,$2) && $(call test-different,$1,$2)

+# This checks whether <file> has changed since the last time it was 'stored'
+# $(call has-changed,file)
+has-changed = ! $(call test-exists-and-different,$1,$1.previous.make)
+# This 'stores' <file> if it exists
+# $(call has-changed,file)
+store = $(call copy-if-exists,$1,$1.previous.make)
+# This checks whether any file in the list has changed
+# $(call has-changed,list of files)
+has-any-changed = $(sh_true) $(foreach file,$1, || $(call has-changed,$(file)))
+# This 'stores' all the files in the list
+# $(call store-all,list of files)
+store-all = $(sh_true) $(foreach file,$1, ; $(call store,$(file)))

As you have already discovered, the above won't work properly.  There are 
various reasons for this, one of which is filesystem metadata caching that 
causes file changes to not show up in a timely manner.  Another is execution 
order: many times you want to test the freshness of a file, but by the time you 
do it's already too late because this would have been executed while building 
the dependency graph.

Anyway, it's a thorny problem.

+
 # Return value 1, or value 2 if value 1 is empty
 # $(call get-default,<possibly empty arg>,<default value if empty>)
 get-default    = $(if $1,$1,$2)
@@ -2310,7 +2330,7 @@ $(SED) \
 enlarge_beamer = $(PSNUP) -l -1 -W128mm -H96mm -pletter

 # $(call test-run-again,<source stem>)
-test-run-again = $(EGREP) -q '^(.*Rerun .*|No file $1\.[^.]+\.)$$' $1.log
+test-run-again = $(EGREP) '^(.*Rerun .*|No file $1\.[^.]+\.)$$' $1.log | 
$(EGREP) -q -v '^(Package: rerunfilecheck.*Rerun checks for auxiliary 
files.*)$$'

That's an interesting one - I hadn't seen that Package:rerunfilecheck thing 
before.  How standard is that?

 # This tests whether the build target commands should be run at all, from
 # viewing the log file.
@@ -2319,7 +2339,8 @@ define test-log-for-need-to-run
 $(SED) \
 -e '/^No file $(call escape-fname-regex,$1)\.aux\./d' \
 $1.log \
-| $(EGREP) -q '^(.*Rerun .*|No file $1\.[^.]+\.|No file .+\.tex\.|LaTeX 
Warning: File.*)$$'
+| $(EGREP) '^(.*Rerun .*|No file $1\.[^.]+\.|No file .+\.tex\.|LaTeX Warning: 
File.*)$$' \
+| $(EGREP) -q -v '^(Package: rerunfilecheck.*Rerun checks for auxiliary 
files.*)$$'
 endef

 # LaTeX invocations
@@ -2856,24 +2877,29 @@ endif
    run=0; \
    for i in 1; do \
        if $(call test-exists,$*.bbl.cookie); then \
+           $(call echo-justify,Running latex because $*.bbl.cookie is present); \
            run=1; \
            break; \
        fi; \
        if $(call test-exists,$*.run.cookie); then \
+           $(call echo-justify,Running latex because $*.run.cookie is present); \
            run=1; \
                break; \
        fi; \
        if $(call \
        test-exists-and-different,$*.auxtarget.cookie,$*.auxtarget.make);\
        then \
+           $(call echo-justify,Running latex because $*.auxtarget.cookie differs from 
$*.auxtarget.make); \
            run=1; \
            break; \
        fi; \
        if $(call test-log-for-need-to-run,$*); then \
+           $(call echo-justify,Running latex because $*.log indicated that this is 
necessary); \
            run=1; \
            break; \
        fi; \
-       if [ ! -e $*.1st.*.make ]; then \
+       if [ ! -e $@.1st.make ]; then \

I'm not sure why you think this was a typo.  It wasn't, so far as I remember: 
we definitely want $*.1st.*.make (in particular, your change will break latex 
sub-includes, and I'm not sure how $@ works at all, frankly).  Why is this 
change here?

+           $(call echo-justify,Running latex because $@.1st.make does not exist); \
            run=1; \
            break; \
        fi; \
@@ -2889,10 +2915,14 @@ endif
            ); \
            $(call run-latex,$*); \
            $(CP) '$*.log' '$*.'$(RESTARTS)-$$i'.log'; \
-           $(call test-run-again,$*) || break; \
+           if $(call test-run-again,$*); then \
+                   $(call echo-justify,latex rerun requested by $*.log); \
+           else break; \
+           fi; \

If you're going to do this, it's best to just call test-run-again once and use 
proper shell conditions to do the echo justify.

        done; \
+       $(CP) '$@' '$@.1st.make'; \
    else \
-       $(MV) '$@.1st.make' '$@'; \
+       $(CP) '$@.1st.make' '$@'; \

This is simply wrong.  CP is incorrect here, because you want to remove the 
.1st file in some cases (particularly if you have graphics dependencies, etc., 
that will trigger a rerun of make with new .d files, you really don't want that 
.1st.make file hanging around messing with the new dependency calculation).

I don't doubt that it helps to solve your particular problem, but in this case 
the cure is worse than the disease for lots of other users.

    fi; \
    $(call copy-with-logging,$@,$(BINARY_TARGET_DIR)); \
    $(call latex-color-log,$*)
@@ -2911,8 +2941,14 @@ endif
 # don't need to be changed, thus possibly avoiding a rebuild trigger.
 %.bbl: %.auxbbl.make
    $(QUIET)\
+   if $(call has-any-changed,$^); then \
+       $(call store-all,$^); \
+   else \
+       $(TOUCH) $@; exit 0; \
+   fi; \
    $(if $(filter %.bib,$^),\
        $(call echo-build,$(filter %.bib,$?) $*.aux,$@); \
+       $(call echo-justify,Running bibtex because dependencies of $@ changed); \
        $(call run-bibtex,$*); \
        $(TOUCH) $@.cookie; \
    ) \
@@ -3163,8 +3199,14 @@ endif
 #  not creating a .1st.make file..
 #
 %.$(build_target_extension).1st.make %.d %.aux %.aux.make %.fls: %.tex
-   $(QUIET)$(call echo-build,$<,$*.d 
$*.$(build_target_extension).1st.make,$(RESTARTS)-1)
    $(QUIET)\
+   if $(call has-any-changed,$^); then \
+       $(call store-all,$^); \
+   else \
+       $(TOUCH) $@; exit 0; \
+   fi; \
+   $(call echo-build,$<,$*.d 
$*.$(build_target_extension).1st.make,$(RESTARTS)-1); \
+   $(call echo-justify,Running latex for the \"first\" time to build .d and 
.$(build_target_extension).1st.make); \
    $(call run-latex,$*,-recorder) || $(sh_true); \
    $(CP) '$*.log' '$*.$(RESTARTS)-1.log'; \
    $(call die-on-import-sty,$*.log); \

As you have noticed, there are monsters, here.  Unless you have a very deep 
understanding of how make is invoked, reinvoked, and the interaction between 
those invocations and the loopy latex runs, you're going to run into trouble 
trying to calculate dependencies this way.

This is one of the problems with an open source project like this: the level of 
understanding required to even make small changes is enormous, so I don't get 
much in the way of people able to contribute or take it over.  I've been doing 
this for a long time, and I've been ready to let it go for a while, now, but 
nobody can take it.

Sigh.

Anyway, I'm afraid that having make do extra work is part of the game, here.  
There's no fixing it for all cases because we quickly run into the Halting 
Problem.  We can't, for example, really tell what latex's includes will do 
until we have included and parsed them, and then we're starting to ask 
questions like "will this program include this file?"  You just can't answer 
that *in general*, because it's impossible.  The workarounds for the common 
cases work well for the common cases, but increasingly the bug reports I'm 
receiving are pushing the boundaries of the hack.

Given that it *is* a huge hack, I have to draw the line for increasingly 
complex fixes somewhere.  This one, unless you've found a real bug, is firmly 
over that line.  :-(

Original comment by shiblon on 10 May 2012 at 7:36