suspected potential race condition when rebuilding dvi files

[GoogleCodeExporter]

What steps will reproduce the problem?
1. untar testcase.tar.gz to a new directory
2. check out lastest Makefile in the trunk (r33)
3. run `make'

What is the expected output? What do you see instead?

Expected:

NOTE: You may ignore warnings about the following files:

     test.d

Makefile:1442: test.d: No such file or directory
= test.tex --> test.d test.dvi (1) =
mv: cannot stat `test.dvi': No such file or directory
sleep 1;
= pstex-test.fig --> pstex-test.pstex =
= pstex-test.fig pstex-test.pstex --> pstex-test.pstex_t =
= test.tex --> test.d test.dvi (1) =
Success!  Wrote 1 page, 516 bytes
= test.dvi --> test.ps =
= test.ps --> test.pdf =
rm test.paper.make test.embed.make

Reality:
NOTE: You may ignore warnings about the following files:

     test.d

Makefile:1442: test.d: No such file or directory
= test.tex --> test.d test.dvi (1) =
mv: cannot stat `test.dvi': No such file or directory
= pstex-test.fig --> pstex-test.pstex =
= pstex-test.fig pstex-test.pstex --> pstex-test.pstex_t =

! Emergency stop.
<read *> 

l.10 \input{pstex-test.pstex_t}
                               ^^M
ead from terminal in nonstop modes)

make: *** [test.dvi] Error 1

What version of the product are you using? On what operating system?

Makefile (r33) from the trunk
GNU Make 3.81
GNU sed 4.1.5
GNU bash 3.2.33(1)-release
TexLive where `latex --version' shows `3.141592-1.40.3-2.2 (Web2C 7.5.6)'
Linux distro: Ubuntu Hardy (8.04)

Please provide any additional information below.

I suspects that there's a race condition when Make finishes its first dvi
building. Sometimes after the file referred by `! LaTeX Error: File XXX not
found' is remade, Make chooses the wrong rule (%.dvi: %.bbl %.aux) and
outputted fatal messages. Trying to prove my concept, I hacked the Makefile
by adding `sleep 1' and then it never fails. (patch file attached) I
couldn't reproduce this bug with debug information by running `make -d'
since additional output seems to slow down the speed and thus avoid this
problem.

Original issue reported on code.google.com by favonia on 22 Mar 2008 at 3:02

Attachments:

• testcase.tar.gz
• Makefile.patch

[GoogleCodeExporter]

Accepting.  This may not be a race condition, but a stupid error on my part, 
certainly.

Just to be sure, can you try it with -j 1 to make sure that make is not using
multiple processes?

Original comment by shiblon on 26 Mar 2008 at 4:22

• Changed state: Accepted

[GoogleCodeExporter]

I have a fix coming up for this in r34.  The problem was not a race condition, 
but an
issue that came up with pstex handling.  The first run of LaTeX is supposed to
generate a log file that is parsed, and from that the dependencies are created. 
After that first run, it is run *again* (usually), provided that the first run
failed.  The detection for a first run requiring another run was broken, as was 
the
detection of errors when \input commands fail (like for pstex_t files).

I added special handling for failed \input commands when they are referring to a
pstex_t file, and added a condition to the command that checks when latex is
indicating a need to be run again.

This should fix all of the issues with pstex, without introducing the need to 
sleep.
 Please let me know if this works for you.

Original comment by shiblon on 28 Mar 2008 at 3:42

• Changed state: Fixed

[GoogleCodeExporter]

Sorry for late reply.

As I did a little more research on this problem, I found that the newest version
still have similar problems when these are two or more pstex files. 
Fortunately, I
could, finally, obtain debug messages from both successful & failing runs
(testresult). I think the core problem is more like this:

1. Make runs the first `latex'
2. Make generates %.d
3. Make generates xxx.pstex and xxx.pstex_t very fast
4. Depending on the timestamps, it may lead to two possible results:
4a.If xxx.pstex and %.d have the same timestamp, Make feels that %.d
   is up to date, and so skips regenerating %.d ==> doomed :(
4b.Otherwise, Make would regenerate %.d again ==> safe :D

To me it seems the behavior of Make depends on the timing, and
that's what I mean by `race condition'. :O

Original comment by favonia on 31 Mar 2008 at 6:12

Attachments:

• testcase.tar.gz
• testresults.tar.gz

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Therefore, I think this means that it's not a bug,
but an unavoidable situation following second-based
timestamps.

In addition, in my opinion the best solution is
to adopt nanosecond timestamps or whatever better
timestamps. Unfortunately ext3 does not support it.
(However ext4 will!)

For now I may introduce additional `touch' or `sleep'
to avoid hazards.

Original comment by favonia on 31 Mar 2008 at 7:20

[GoogleCodeExporter]

I will take a look at your test results soon and see if I can find any other
non-timestamp-related issues.  Thanks for being patient with me, and sorry this 
has
been such a problem for you :-(

Original comment by shiblon on 31 Mar 2008 at 1:13

• Changed state: Accepted

[GoogleCodeExporter]

The way that we discover pstex files, by parsing a failed log file, is what is
causing the trouble here.  The .dvi file is not created by the first run of 
LaTeX,
and it causes a lot of other stuff to bail (on my machine, anyway).  I'm 
looking into it.

Original comment by shiblon on 9 Apr 2008 at 9:32

• Changed state: Started

[GoogleCodeExporter]

Okay, I think I get this, now.  It's a mess, unfortunately.  What happens is 
this:
latex tries to build the file, notices that there is an error, and stops.  We 
look at
the log file, see that what's missing is a pstex_t file, and happily build it.  
Then
latex runs *again*, and notices that a *different* pstex_t file is missing, etc.

I'm not sure why timing has anything to do with this.  I would encourage you to 
try
-j1 as a make argument. I'm guessing that you'll notice that the nondeterminism 
goes
away in that case.

At any rate, since the pstex_t feature has only been requested by one person, 
I'm
guessing it isn't all that popular a feature.  Perhaps using a blunt approach 
would
work best: if there *any* pstex dependency is found, build *every* fig into a 
pstex
file and try again.  I'll see if that's feasible.

Original comment by shiblon on 9 Apr 2008 at 9:36

[GoogleCodeExporter]

I dug further into this, and found the problem that was causing the seeming
nondeterminism.  Blech.

The .d file had dependencies on .pstex files, *not* pstex_t files.  So, it was 
happy
to continue when the .pstex file was there, but what was really needed was the
.pstex_t file.  Fixing the dependency should fix your nondeterminism.

The issue of multiple LaTeX invocations being required to find them all remains,
however.  I have a hack in the Makefile that is currently turned off that 
builds all
fig files into .pstex_t files when it finds any of them missing.  To turn it 
on, set
the variable "PSTEX_BUILD_ALL_HACK" when you make, like this:

make PSTEX_BUILD_ALL_HACK=1 test

This is in r37 (version 2.1.18, no download available until I get the thumbs 
up).

Original comment by shiblon on 9 Apr 2008 at 10:06

[GoogleCodeExporter]

Sorry for late reply as I was coping with midterms. :Q

I've tried "-j1" without "PSTEX_BUILD_ALL_HACK=1" and it skips building either 
the
second or the third pstex_t file. I'm still trying to figure it out.

PSTEX_BUILD_ALL_HACK=1 would definitely work. Even better, it makes Make faster 
since
latex is invoked fewer times.

Thanks for the hack. :)

Original comment by favonia on 17 Apr 2008 at 2:52

[GoogleCodeExporter]

That's just plain weird.  It should work with or without PSTEX_BUILD_ALL_HACK 
set, it
should just be slower without it :-)  As it happens, it is not working 
correctly on
my laptop, but it is working correctly on my home machine.  I am going to try 
to get
to the bottom of it, but am going to change the hack to be on by default 
meanwhile. 
See r38 and let me know if it works for you.

Original comment by shiblon on 17 Apr 2008 at 6:18

[GoogleCodeExporter]

Marking this as fixed, since without the hack there does not appear to be a 
graceful
way of handling this.  The hack is now the default behavior, and r39 adds some
documentation about what it does, so I'm satisfied.

Original comment by shiblon on 17 Apr 2008 at 6:51

• Changed state: Fixed