Support for pre-processed, non-shell files

[shiblon]

Originally reported on Google Code with ID 133

My project requires that I run my TeX file through a pre-processor before creating a
PDF. Specifically:

1. Edit foo.tex
2. Run lhs2tex, producing foo.f.tex
3. Run pdflatex foo.f.tex.

"foo.f.tex" is the file that pdflatex sees, but its really an intermediate file. This
causes several problems:

1. Have to execute "make foo.f.pdf" rather thank "make foo".
2. Have to write Makefile rules in terms of "foo.f.tex" rather than 
"foo.tex"

One solution is to add ".f.tex.sh" files for each ".tex" file. This creates the right
dependencies, but executes the scripts every time. 

The best solution I have found so far is two-fold:

1. Redefine the .sh rule so it has no recipe:
  %.tex:: %.tex.sh

2. Define my own rule to run lhs2tex:
  %.f.tex: %.tex
    lhs2tex --poly -o $@ $<

This doesn't allow me to execute "make foo" (I still have to write "make foo.f.pdf")
but otherwise it works.

I've attached a tar file containing a sample directory with Makefile 2.2.0 and Targets.ini
set up so you can build foo.f.pdf. If you just build foo.pdf, I have LaTeX report an
error. 

Reported by jgbailey on 2011-06-06 16:18:37

---

• Attachment: foo.tar.gz

[shiblon]

Thanks for the specific report, from our discussion on the group.

I'm curious about why we can't name things the other way around, e.g.,

1) Edit foo.f.tex
2) Run lhs2tex, producing foo.tex
3) Win

How important is it that foo.tex -> foo.f.tex and not the other way around?

Also, by way of information, the rule for things like "make foo" is the %: rule, which
is somewhat complicated (but can probably be fixed appropriately anyway).  What is
not trivial is getting it to decide which of the two proposed code paths to take *at
the right time*.  The source file name is pretty fundamental to how a lot of things
work, and if the dependencies can't reasonable tell whether the foo.tex file should
first produce a foo.f.tex first (which they can't, not without clairvoyance), then
the makefile will always do the wrong thing for someone.

If, however, you specify that foo.f.tex is the deepest leaf in the dependency tree,
you can make this work painlessly for everyone.  Of course, .f.tex is just an example
- we could name it foo.lhstex or something if we wanted, and we can have that particular
bikeshedding discussion right long with the more fundamental one above, if you'd like.
 :-)

Reported by shiblon on 2011-06-13 17:55:09

[shiblon]

No reason the source can't be named foo.f.tex (or foo.lhs.tex, or
similar). .tex is nice to use because the files are much more tex than
they are Haskell, in this case.

How might I update the rules to test using foo.f.tex as the original
source, over foo.tex?

Reported by jgbailey on 2011-06-13 17:59:23

[shiblon]

Well, the main trick is getting the makefile to recognize that .lhs.tex or .f.tex is
a valid source file.  The hack you used to pretend that there is a script for those
works, but I can put this together properly with a little bit of effort.  What should
the filename pattern be (.lhs.tex, .f.tex, etc.?)

Reported by shiblon on 2011-06-13 18:17:01

[shiblon]

How about lhs.tex? Could that be problem if you added a rule in the
futeur that dealt with .lhs? For example, if someone had literate
files and they wanted to generate PDFs from them.

Reported by jgbailey on 2011-06-13 18:23:25

[shiblon]

Sure, if you think that .lhs.tex might be used for something else, that is important
to consider.  I don't have any intuition for how likely that is, though.

Reported by shiblon on 2011-06-13 18:25:22

[shiblon]

By the way, this will definitely work better if the extension is not .tex.  We might
want to make it .lhstex instead (without the extra dot)

Reported by shiblon on 2011-06-13 18:26:36

[shiblon]

I think lhs.tex is fine and not likely to collide. I meant, would that
collide with .lhs (alone)?

Reported by jgbailey on 2011-06-13 18:29:55

[shiblon]

.lhstex sounds reasonable to me.

Reported by jgbailey on 2011-06-13 18:32:56

[shiblon]

lhs.tex is not only likely to collide with .tex, it is guaranteed.  It looks like, after
reading the lhs2tex documentation, the standard suffix is simply .lhs, so that's what
the current patch uses.

The revision with the relevant changes is rad5564965992, and a full makefile with them
in is attached.  Let me know how it works for you!

Reported by shiblon on 2011-06-13 18:36:31

---

• Attachment: Makefile

[shiblon]

The patched version gives an error with "make foo", "make help" and assume all other
commands:

    Makefile:2899: *** multiple target patterns.  Stop.

I'm running GNU Make 3.81, from cygwin.

Reported by jgbailey on 2011-06-13 20:12:47

[shiblon]

Wow, it doesn't do that for me.  Is it possible that some of your targets in .ini files
are conflicting with my changes, now?

Reported by shiblon on 2011-06-14 15:51:03

• Status changed: Started

[shiblon]

Its because Make 3.81 has a stupid bug on Windows that misinterprets
the ":" character. The rule on line 2899 is:

%.gls:  %.glo %.tex $(shell $(KPSEWHICH) nomencl.ist || $(ECHO) nomencl.ist)
    $(QUIET)$(call echo-build,$<,$@)
    $(QUIET)$(call run-makeindex,$<,$@,$*.glg,nomencl.ist)

Which expands to (on my system)

%.gls:  %.glo %.tex
c:/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist
    $(QUIET)$(call echo-build,$<,$@)
    $(QUIET)$(call run-makeindex,$<,$@,$*.glg,nomencl.ist)

And that causes Make to file.

Escaping the colon on that line and the other that uses KPSEWHICH
(2904) fixes the problem.

Did you already solve that elsewhere?

Reported by jgbailey on 2011-06-14 17:23:15

[shiblon]

Hmm, is it really a colon?  I don't see an errant colon anywhere in there.

Try changing the $(shell ...) line to be wrapped in $(strip), like this:

$(strip $(shell $(KPSEWHICH) nomencl.ist || $(ECHO) nomencl.ist))

I think that will fix it.

Reported by shiblon on 2011-06-14 18:12:03

[shiblon]

Oh, never mind.  I get it.  It's from the C:/ thing.  I will send you a new fix shortly.

Reported by shiblon on 2011-06-14 18:12:37

[shiblon]

Incidentally, are you using cygwin?  If so, I can just do this:

$(call path-norm,$(shell $(KPSEWHICH) nomencl.ist || $(ECHO) nomencl.ist))

That might do it.  Would you try that for me?

Reported by shiblon on 2011-06-14 18:13:52

[shiblon]

Attached with path-norm fix.  I can't test it very well, but it's here in case you're
using cygwin.

Reported by shiblon on 2011-06-14 18:15:23

---

• Attachment: Makefile

[shiblon]

A couple of problems here:

  * Typo on line 2944, "%<" should be "$<".
  * Running "make test" (with test.lhs) always re-makes the PDF, even if test.lhs hasn't
changed.
  * lhs2tex's output is the original file with a bunch of TeX inserted. Any errors
reported will use line numbers based on the intermediate .tex file. However, that file
is deleted after make runs -- so you can't see where the error occurred! Unfortunately
the line numbers are too far off for guessing where they happen in the original source.
    * This is easiest to see with an underfull hbox warning. The attached file demonstrates
the problem. 
  * "make test" produces a strange "empty" line of output. See "=  -->  (0-0) =" below:

    >make test
    =  -->  (0-0) =
    = test.tex --> test.d test.pdf.1st.make (0-1) =
    sed: can't read test.fls: No such file or directory
    = test.tex --> test.pdf (1-2) =

I do get a PDF though, so its working somewhat!

Reported by jgbailey on 2011-06-15 21:29:53

---

• Attachment: test.lhs

[shiblon]

Oh, how embarrassing.  Thanks for being my debugger :-p

Two things fixed in this version:

- Typo fixed (which I believe addresses the re-running and empty output line problem)
- made .tex files based on .lhs and .rst files into .SECONDARY targets to avoid deletion

New version attached.  Thanks for your patience thus far.

Reported by shiblon@google.com on 2011-06-16 14:17:21

---

• Attachment: Makefile

[shiblon]

The .tex files hang around now, but it still remakes each time. I also still see the
odd output ("=  -->  (0-0) =").

Getting better though!

Reported by jgbailey on 2011-06-16 15:03:17

[shiblon]

OK - the files are getting rebuilt every time because of the following reasoning:

- Since we don't know much about dependencies of .lhs files (I assume that you can
import other haskell?), we can't reliably detect when those dependencies change.  So,
we rebuild every time.

I have, however, removed that constraint by differentiating between "scripts" and "stuff
that generates source files", and removed the latter from the .PHONY declaration. 
If that works, mission accomplished (but it does mean that automatic dependency generation
will be broken for those files, as mentioned above).

Odd output is a different and very interesting problem.  It turns out that this is
the first source generator I've supported that uses -o for the output file, and reverses
the order.  Anyway, I fixed it, I think, and would love you to have a look at it.

Reported by shiblon on 2011-06-16 20:49:27

---

• Attachment: Makefile

[shiblon]

What about recovering dependencies from the generated TeX file? That is, you can't know
what lhs2tex might include (and it can include arbitrary stuff), but you already have
machinery for detecting what the TeX file is looking for. Can that machinery be applied
to the TeX file in this case?

For example, it would be really nice to still have references and citations built automatically.

I will download and try the file now - just some preliminary comments.

Reported by jgbailey on 2011-06-16 21:22:49

[shiblon]

Working really well! I found I need to write rules based on the .tex file. So if A.tex
is produced from A.lhs and includes B.tex (which is in turn produced by B.lhs), I write:

   A.tex: B.tex

I also found that I have to run make twice before it stops producing output. The following
shows me buildling "test.pdf" from "test.lhs". Note that test.lhs has no other dependencies
(which doesn't seem to make a difference anyway):

But I am really happy with where the Makefile is at now!

Reported by jgbailey on 2011-06-16 22:02:32

[shiblon]

Regarding automatic dependency generation, I *believe* that works properly already (we
do extract dependencies from the .tex, even when it's generated).

What I meant to say, but did so poorly, was that if there are *Haskell* dependencies
that are handled by lhs2tex directly, *those* won't show up.

There's a similar problem with .rst files that use the special "include" comment, for
example.  Does that make sense?

Regarding writing rules based on the .tex file, can you help me understand what problem
that solves?  It really shouldn't be necessary to do

A.tex: B.tex

Make often keeps producing output after multiple runs, so unless you're finding that
your document is not built fully after the first time, that is just the way it goes
:-)

Anyway, let's get this right - let me know even the little things that are broken,
and we'll figure it out!

Reported by shiblon on 2011-06-16 23:03:50

[shiblon]

Regarding "A.tex: B.tex" -

If I have a file "A.lhs" and it \inputs "B.tex", A.pdf won't rebuild when B.tex changes
unless I have the rule.

It doesn't seem specific to .lhs files, either. The same occurs when I have C.tex,
which \inputs B.tex. Changing B.tex won't cause C.pdf to rebuild unless the rule is
present

Is it supposed to behave differently?

Reported by jgbailey on 2011-06-16 23:26:56

[shiblon]

Yes, it is supposed to behave differently.  Hmmm.  "input" is a somewhat tricky beast,
though.  If you can send me a minimal example of what makes this fail, that needs to
be fixed in general.  tex -> tex dependencies are definitely supposed to work (they
should end up in the .d file).

Reported by shiblon on 2011-06-17 12:07:09

[shiblon]

Attached file demonstrates the problem. a.tex \inputs b.tex. After making a.pdf, changing
b.tex will not cause a.pdf to re-make. Notice in the transcript below that I have to
make "a" twice before make stops building:

  >make a
  NOTE: You may ignore warnings about the following files:

       a.d

  Makefile:2703: a.d: No such file or directory
  = a.tex --> a.d a.pdf.1st.make (0-1) =
  sed: can't read a.fls: No such file or directory
  = a.tex --> a.pdf (1-2) =
  Success!  Wrote 1 page, 10836 bytes to a.pdf

  10:50:00.03 Mon 06/20/2011 ] C:\Users\baileyj\Source\foo+
  >make a
  = a.tex --> a.d a.pdf.1st.make (0-1) =
  sed: can't read a.fls: No such file or directory
  = a.tex --> a.pdf (0-2) =
  Success!  Wrote 1 page, 10836 bytes to a.pdf

  10:50:17.91 Mon 06/20/2011 ] C:\Users\baileyj\Source\foo+
  >make a
  make: Nothing to be done for `a'.

At this point I altered b.tex and ran make again. No change:

  10:50:25.48 Mon 06/20/2011 ] C:\Users\baileyj\Source\foo+
  >make a
  make: Nothing to be done for `a'.

Reported by jgbailey on 2011-06-20 17:54:49

---

• Attachment: foo.tar.gz

[shiblon]

I'm having trouble reproducing this issue.  To be extra clear, we are talking about
two unrelated things:

1) making a.pdf, then touching b.tex, causes a.pdf to not rebuild when typing "make
a.pdf"

2) make output does not stop after the first invocation of make.

Problem 1 is real.  Problem 2 is something that is just inherent in the way that make
is running, and is not fixable.  I figured out once upon a time why this happened,
and then I figured out that there was absolutely not way to express the dependencies
to make that would fix this without breaking something bigger.

So, Problem 1.  I just tried doing this with your files (thanks), and I note that "make
a" works the first time, and creates a.d that has "b.tex" listed as a dependency. 
That's correct and working.

Now, if I "touch b.tex" and run make again, it works.

So, I'm having a hard time reproducing this.  I wonder if something about the path-norm
function is not working properly under cygwin. If that's the case, then that would
explain what's going on.  Unfortunately, that's hard to debug.

One thing you can try is to change the line

USE_CYGPATH := <stuff>

so that it reads

USE_CYGPATH :=

Then if it works, that will really tell us something.  My best guess right now is that
it's something to do with our invocation of cygpath not working properly.

Another thing you can do is run make with the SHELL_DEBUG=1 environment variable, e.g.

make SHELL_DEBUG=1 a

If you see invocations of cygpath in there, that means we're at least trying to call
it.  If that's happening, then try running cygpath manually on b.tex and let me know
what the output looks like (including newlines).  I'm guessing we're parsing it wrong
or invoking it incorrectly, somehow.

Reported by shiblon on 2011-06-20 18:22:29

[shiblon]

Making USE_CYGPATH a no-op resulted in another "multiple target
patterns" error due to "C:\..." so I couldn't test w/o cygpath.

I ran "make SHELL_DEBUG=1 a" with a clean directory. I zipped up the
results (including all intermediate files) and attached them to this
comment.

The make transcript is in the file "make.a.transcript".

Let me know what else I can try.

Reported by jgbailey on 2011-06-20 18:50:47

[shiblon]

Oh, right.  Of course.

Can you try changing the get-cygpath function to something else?  Currently it reads
like this (and I am trying to remember why we call it twice):

define get-cygpath
$(shell $(CYGPATH) -u "$(shell $(CYGPATH) -s -w $1)")
endef

I'd like to see it read more like this for now:

define get-cygpath
$(shell $(CYGPATH) -u "$1")
endef

I'm going to look through the bugs and see if I can find out why we call it twice.

Reported by shiblon on 2011-06-20 19:41:18

[shiblon]

No effect - touching b doesn't cause a.pdf to rebuild.

Reported by jgbailey on 2011-06-20 19:59:51

[shiblon]

OK - I added some output to the path-norm function that might help us debug this.  You
should see CYGPATH_OUTPUT= in your build output.  I'd like to see what it says when
it does that.

Thanks for helping me debug.  I don't have cygwin and don't know anyone who does nearby
:)

Reported by shiblon on 2011-06-20 20:24:18

---

• Attachment: Makefile

[shiblon]

I've attached a transcript, but here are the relevant CYGPATH OUTPUT lines:

  + cygpath -u c:/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist
  Makefile:2912: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE
  ...
  + cygpath -u c:/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist
  Makefile:2917: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE
  ...
  + cygpath -u c:/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist
  Makefile:2912: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE
  ...
  + cygpath -u c:/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist
  Makefile:2917: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE

Reported by jgbailey on 2011-06-20 20:31:53

---

• Attachment: make.a.transcript

[shiblon]

OK - something else is amiss, then.  It looks like there is no mention of b.tex in there
at all, and there should be.

Can you look at a.d and see what it contains for me?

Reported by shiblon on 2011-06-20 20:33:53

[shiblon]

Here you go:

  >cat a.d
  # vim: ft=make
  .PHONY: a._graphics
  .SECONDEXPANSION:

Reported by jgbailey on 2011-06-20 20:42:31

[shiblon]

Now we're getting somewhere!  sed is not working the way we expect on your system.

So, I'd like to see your .log files, if you would.  I will see where we're barfing
and why.  Meanwhile, can you find out what version of sed is installed?

Reported by shiblon on 2011-06-20 20:45:56

[shiblon]

  GNU sed version 4.2.1

I've attached all the relevant log files. Glad to run some experiments
as needed.

Reported by jgbailey on 2011-06-20 20:52:02

[shiblon]

I think you might have forgotten the attachment :)

Reported by shiblon on 2011-06-20 20:54:29

[shiblon]

Here they are - they are in the email but didn't end up here ...

Reported by jgbailey on 2011-06-20 20:56:38

---

• Attachment: a.log
• Attachment: a.0-1.log
• Attachment: a.1-2.log

[shiblon]

OK - that's just bizarre. Your latex is not complaining about a missing b. How about
the .fls file?

Reported by shiblon on 2011-06-20 20:59:25

[shiblon]

Why would pdflatex complain that b is missing? b.tex exists in that
directory; a. tex inputs b, using \input{b}.

I've attached the .fls file.

Reported by jgbailey on 2011-06-20 21:02:55

[shiblon]

Good point - it wouldn't.  Where are you attaching these things, by the way?  I don't
see them in the tracker *or* in my emails...

Reported by shiblon on 2011-06-21 14:58:21

[shiblon]

Also, you can just inline the .fls file - it should be pretty small.

Reported by shiblon on 2011-06-21 17:42:50

[shiblon]

Sorry for the delay. Below is a transcript of the make command, followed by the contents
of "pdflatex4660.fls". 

The transcript shows sed complaining that it can't read "a.fls." Is it a problem that
the file is named "pdflatex4660.fls" and not "a.fls"? 

== transcript ==

>make a
NOTE: You may ignore warnings about the following files:

     a.d

Makefile:2699: a.d: No such file or directory
Makefile:2912: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE
Makefile:2917: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE
= a.tex --> a.d a.pdf.1st.make (0-1) =
sed: can't read a.fls: No such file or directory
Makefile:2912: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE
Makefile:2917: CYGPATH OUTPUT=/cygdrive/c/Users/baileyj/Programs/texlive/2010/texmf-dist/makeindex/nomencl/nomencl.ist=DONE
= a.tex --> a.pdf (1-2) =
Success!  Wrote 1 page, 10800 bytes to a.pdf

== pdflatex4660.fls ==

PWD C:/Users/baileyj/Source/foo
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf.cnf
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf/web2c/texmf.cnf
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf-var/web2c/pdftex/pdflatex.fmt
INPUT a.tex
OUTPUT a.log
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf-dist/tex/latex/base/article.cls
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf-dist/tex/latex/base/article.cls
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf-dist/tex/latex/base/size10.clo
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf-dist/tex/latex/base/size10.clo
OUTPUT a.aux
INPUT b.tex
INPUT b.tex
OUTPUT a.pdf
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf-var/fonts/map/pdftex/updmap/pdftex.map
INPUT a.aux
INPUT c:/Users/baileyj/Programs/texlive/2010/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb

Reported by jgbailey on 2011-06-22 19:02:57

[shiblon]

Yes!  That's the problem.  We cannot detect dependencies if %.fls is not there (or is
named something else).  So, why is it creating pdflatex4660.fls instead of a.fls? 
That's something I have never seen before.

Is 4660 a process ID?  Does it change if you keep running it?

Reported by shiblon@google.com on 2011-06-22 19:11:13

[shiblon]

Process Monitor shows that pdflatex creates the file, using the PID of
that instance of pdflatex.

Reported by jgbailey on 2011-06-22 19:19:54

[shiblon]

OK - good to know.  Are you using MikTex?  What's your distribution?

This is either a pdflatex bug or there is a new command line flag that dictates this
behavior.  We'll need to figure out which of those it is and then hopefully it's just
a new command line flag (otherwise you'll need to use a different pdflatex binary that
doesn't have this bug).

Reported by shiblon on 2011-06-22 19:28:06

[shiblon]

TeX Live 2010 on Windows 7, with cygwin. What flag tells pdflatex to
create the .fls file normally, or does it always do it?

pdfTeX 3.1415926-1.40.11-2.2 (Web2C 2010)
kpathsea version 6.0.0
Copyright 2010 Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX).
Compiled with libpng 1.4.3; using libpng 1.4.3
Compiled with zlib 1.2.5; using zlib 1.2.5
Compiled with xpdf version 3.02pl4

Reported by jgbailey on 2011-06-22 19:36:46

[shiblon]

OK - I found this: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=575731.

We can work around this.  I've attached a fix that hopefully works properly.  Let me
know how it goes for you.

Reported by shiblon on 2011-06-22 19:38:15

---

• Attachment: Makefile

[shiblon]

I get an error when building:

>make a
NOTE: You may ignore warnings about the following files:

     a.d

Makefile:2708: a.d: No such file or directory
= a.tex --> a.d a.pdf.1st.make (0-1) =
cp: cannot stat `a.log': No such file or directory
egrep: a.log: No such file or directory
sed: can't read a.log: No such file or directory
Error: failed to create a.aux
make: *** [a.d] Error 1

The directory contains a bunch of "a.tex.XXX" files, as well as a ".fls" file:

a.tex
a.tex.aux
a.tex.log
a.tex.pdf
b.tex
pdflatex6452.fls

Reported by jgbailey on 2011-06-22 19:42:46

[shiblon]

Grrr.  A one-character fix.  Sorry.

Reported by shiblon on 2011-06-22 19:46:35

---

• Attachment: Makefile