Feature request: Integration with SVN

[GoogleCodeExporter]

I keep my latex documents in an subversion repository. So that I can tell
printed drafts apart, I use a perl script makesvntex.pl to automatically
generate a file svn.nobuild.tex which contains commands for the revision
number etc. which I can then put in the footer.

It seems to me that regenerating svn.tex should really be triggered by a
subversion commit, but as far as I know this has to either be done on the
svn *server* side, or by writing a wrapper for the svn commit command,
which I'd rather avoid.

The most general solution (and perfectly satifactory for me) is for
svn.nobuild.tex to be rebuilt every time any of the source files are
changed (I'm sure this is quite easy to implement in Makefile.ini but I
haven't managed to get it right).

A more satisfactory solution would be for the makefile to check the
subversion version and rebuild svn.tex where necessary but I guess this
would require much more work and be specific to subversion

Original issue reported on code.google.com by mojo...@gmail.com on 1 Feb 2010 at 8:01

[GoogleCodeExporter]

What about creating a file called

svn.nobuild.tex.sh

and using that?  The sh script would call your pl script and then generation 
happens 
properly.

Original comment by shiblon on 1 Feb 2010 at 5:44

[GoogleCodeExporter]

I though of that and assumed that it would only remake if svn.nobuild.tex.sh or 
the
perl script had a later timestamp than the target. It's important that
svn.nobuild.tex be updated every time I run the makefile.

Original comment by mojo...@gmail.com on 1 Feb 2010 at 5:48

[GoogleCodeExporter]

Okay, just to clarify in the midst of my Monday brain slump:

- you are using svn
- and every commit triggers a new version, e.g., a new expansion of $Id$
- and this new expansion should make it into svn.nobuild.tex
- so that you can \include it as part of a footer.

Did I get that right?

Original comment by shiblon on 1 Feb 2010 at 5:57

[GoogleCodeExporter]

Using an $Id$ expansion would be one way of doing it, I did experiment with it
briefly although it's be a bit tricky with latex code.

I find it easier to use the attached perl script.

Original comment by mojo...@gmail.com on 1 Feb 2010 at 6:24

Attachments:

• makesvntex.pl

[GoogleCodeExporter]

I see what you're doing, now.  I'm thinking about this, but can't see an 
obvious way of 
making this automatic without help from svn itself, and that help would almost 
certainly come in the form of a $Id$ tag (or similar), since that's a simple 
way of 
getting an in-file change from an svn action (and that change would trigger 
make's 
rebuild logic).

But, since you're just interested in making this happen every time, you might 
try 
adding a rule like the following to your Makefile.ini, just to see if it works 
(assuming 
your target document is called "mydoc"):

mydoc.aux: svn.nobuild.tex

svn.nobuild.tex: makesvntex.pl
  ./makesvntex.pl
  touch makesvntex.pl

Note that we "touch" the source file after building the nobuild file.  This 
will force 
make to rebuild it next time, even though it exists.  Also, we make the .aux 
file 
depend on the nobuild file, which *should* make the right thing happen in the 
dependency chain.  If not, you might try .dvi or something similar.

Original comment by shiblon on 1 Feb 2010 at 6:54

[GoogleCodeExporter]

Thanks for the suggestion, unfortunately it doesn't really seem to work unless
makesvntex is manually touched before rebuilding:

$ touch makesvntex.pl 
$ make
./makesvntex.pl
Automatically generating file svn.nobuild.tex
touch makesvntex.pl
= thesis.tex --> thesis.d thesis.dvi (1) =
$ touch thesis.tex
$ make
= thesis.tex --> thesis.d thesis.dvi (1) =
make: `thesis.aux' is up to date.

As you can see, it also seems to have changed the default build target from
thesis.pdf to thesis.aux:

Also I remember the principal problem with the $Id$ tag:
It's file specific and shows the $Id$ etc of latest update to that particular 
file. 
In other words If I put $Id$ into svn.nobuild.tex no amount of committing or 
updating
will change the tags inside it unless svn.nobuild.tex itself has been modified 
since
the last commit. That's why I had to go with the perl option.

One solution I can think of is that make actually modifies .

I might look into writing a wrapper script around svn commit, but I'm sure I'm 
going
to keep forgetting to type
./svncommit 
rather than
svn commit

Thanks once again for your help,

John

Original comment by mojo...@gmail.com on 3 Feb 2010 at 5:49

[GoogleCodeExporter]

Oh, right, the default build target is the first in the file.  *That* could 
probably be 
fixed.

I have been reading up on the "global workspace ID" issue in SVN, and there is 
just 
no easy way to do it without resorting to tricks like you've already employed.  
I *am* 
a bit surprised that the touch trick didn't work, though.  I wonder if it just 
happens 
too soon after the rebuild, so the timestamp doesn't change - on FAT 
filesystems, for 
example, the timestamp resolution is 2 seconds, so this could very well be the 
case.  
Adding "sleep 3" in between the build and the touch might fix it.

If you think I ought to make it harder to screw up the default .pdf build 
target, I can 
look into that in parallel.

Original comment by shiblon on 3 Feb 2010 at 6:37

[GoogleCodeExporter]

The filesystem is ext4. However it seems that the sleep trick works (although 
0.1
seconds seems sufficient).

I had spotted the default target "feature" before. I think it would be helpful 
to
document it in
$ make help
(it would also be nice if the make help trick were documented in the $ make 
--help text)

It's not a problem for me since I also wanted to implement a word count every 
time
the pdf is made. My makefile.ini now looks like this:

###########################
#Makefile.ini
thesis.pdf: wc.txt

thesis.aux: svn.nobuild.tex

svn.nobuild.tex: makesvntex.pl
        ./makesvntex.pl
        sleep 0.1
        touch makesvntex.pl

wc.txt: wcpdf
        ./wcpdf thesis.pdf | tee wc.txt
        sleep 0.1
        touch wcpdf

###########################

See also attached wcpdf file

Although a bit of a hack this seems to work very well. I have to say that this
Makefile and SVN have taken a lot of the effort and worry out of writing my 
thesis.
It would be really great to integrate them better. However I would say that the 
fact
that it's not easier is down to deficiencies in SVN . I'm gobsmacked that you've
managed to squeeze so much functionality out of a flat makefile.

Thanks again

Original comment by mojo...@gmail.com on 4 Feb 2010 at 1:06

Attachments:

• wcpdf

[GoogleCodeExporter]

Thanks for the compliments :)

Now, to address some of the deficiencies.

1) I unfortunately have no control over make --help output, since that's the 
help for 
the make *binary*, and doesn't even look at makefiles in the directory.   Blah.

2) I added to the Makefile.ini documentation that you can restore default 
target 
behavior by adding the following line at the beginning of Makefile.ini:

  %: %.pdf ;

3) I'm glad the sleep trick worked.  It's nice to have something figured out :)

Now, we get to decide what the next step is.  Do we try to generalize this in 
some 
clever way that will be useful to others, or do we leave it at this and add an 
entry to 
the codesite wiki?  I'm kind of in favor of the latter, but I also realize that 
the existing 
script support in the makefile (e.g., using blah.tex.sh to generate blah.tex 
and then 
build it) has a major failing where automatic dependency tracking is concerned.

Your solution to the problem is to always rebuild it.  I actually quite like 
that solution, 
so I think what I will do is add the following targets (or modify as needed) in 
Makefile 
(with appropriate values for $(SLEEP) and $(FS_SLEEP_TIME):

%.tex: %.tex.sh
  $(QUIET)$(call echo-build,$<,$@)
  $(QUIET)$(SHELL) $< $@
  $(QUIET)$(SLEEP) $(FS_SLEEP_TIME)
  $(QUIET)$(TOUCH) $<

%.tex: %.tex.pl
  $(QUIET)$(call echo-build,$<,$@)
  $(QUIET)$(PERL) $< $@
  $(QUIET)$(SLEEP) $(FS_SLEEP_TIME)
  $(QUIET)$(TOUCH) $<

And perhaps another similar one for Python.  This would allow you to create a 
file 
called svn.nobuild.tex.pl, which would basically do exactly what makesvntex.pl 
does.  
Would that work?

Original comment by shiblon on 4 Feb 2010 at 3:03

[GoogleCodeExporter]

I think the solution that you outline would certainly work for me but perhaps 
the
current behaviour is better as default -- automatic remaking could be very 
annoying
if you didn't specifically want that behaviour.

Maybe you could make it so that

svn.nobuild.tex.pl 
would make 
svn.nobuild.tex 
if the perl script has been modified 

but

svn.nobuild.tex.alwaysmake.pl
would make
svn.nobuild.tex
every time.

A python script one would be good as well.

Cheers

Original comment by mojo...@gmail.com on 4 Feb 2010 at 3:12

[GoogleCodeExporter]

I've commited ra29c75712f04 for your testing pleasure.

For now, I think the "always build scripted targets" rule is perfectly 
reasonable.  If 
there are complaints (this is not a widely used feature) then I'll be happy to 
reconsider it.  The makefile is pretty smart about when to go to the trouble of 
rebuilding stuff, so having it always regenerate scripted .tex files is not a 
bad thing; 
you wouldn't be calling "make" if there wasn't at least a change to the .tex or 
something that requires a latex invocation, after all.

So, try the following:

- Download ra29c75712f04 (have a gander at the diff, too, if you're morbidly 
interested)
- Move Makefile.ini out of the way (this means word counting won't work, but 
hey, 
this is a test!)
- Rename makesvntex.pl to svn.nobuild.tex.pl
- Try building!

I think this will do it.  I have added perl and python script support, and even 
fixed a 
long-standing limitation (you couldn't do scripted includes before, but I 
figured out 
how to detect them properly, so we're good, now).

Original comment by shiblon on 4 Feb 2010 at 3:55

• Changed state: Accepted

[GoogleCodeExporter]

Not quite there I'm afraid. It seems to get caught in an infinite loop of 
rebuilding 

= svn.nobuild.tex.pl --> svn.nobuild.tex =
Automatically generating file svn.nobuild.tex
= thesis.tex --> thesis.d thesis.dvi (1) =
= svn.nobuild.tex.pl --> svn.nobuild.tex =
Automatically generating file svn.nobuild.tex
= thesis.tex --> thesis.d thesis.dvi (1) =
= svn.nobuild.tex.pl --> svn.nobuild.tex =
Automatically generating file svn.nobuild.tex
= thesis.tex --> thesis.d thesis.dvi (1) =
= svn.nobuild.tex.pl --> svn.nobuild.tex =
Automatically generating file svn.nobuild.tex
= thesis.tex --> thesis.d thesis.dvi (1) =
^Cmake: *** [thesis.d] Deleting file `thesis.fls'

Another minor bug here would also be that as soon as you touch svn.nobuild.tex
svn.nobuild.tex.pl will no long be rerun until you touch svn.nobuild.tex.pl

I think this is acceptable though.

J

Original comment by mojo...@gmail.com on 4 Feb 2010 at 4:33

[GoogleCodeExporter]

Wow, that is really weird behavior.   It really shouldn't be possible to have 
an infinite 
loop.  I can't seem to reproduce that behavior on my end.  Can you by chance 
create 
a minimal example that has this failure mode and attach it to the bug?

Regarding the second bug you mentioned, I'm not sure that I understand.  I 
touch 
svn.nobuild.tex.pl, so it always has a newer timestamp than svn.nobuild.tex, so 
that 
should cause it to always be built.

Just thinking out loud.  If you can think of a minimal example that causes this 
problem, that would be really great.

Original comment by shiblon on 4 Feb 2010 at 4:40

[GoogleCodeExporter]

I think this may be related to the fact that I have used

onlysources.tex := thesis.tex

in the makefile

Original comment by mojo...@gmail.com on 4 Feb 2010 at 4:55

Attachments:

• minex.tar.gz

[GoogleCodeExporter]

Did you try doing a "make clean" before running the test?  I think that will 
make a 
difference.

Looking at the output, it appears to be hitting this target multiple times:

%.d %.aux %.aux.make %.fls: %.tex

It shouldn't even be *possible* to do this within the makefile, since this is a 
leaf in 
the dependency chain.  Yes, it will notice that the .d file has changed, 
reinclude it, 
and start over, but that's really it.  It should hit it once per source file, 
then the .dvi 
dependency kicks in, which is where the retry logic lives.  This particular 
target has 
no retry logic whatsoever.  It just builds once and gathers dependencies to put 
into 
the %.d file.

My output looks like this on my minimal test (using rae457f9e0dbc , which has 
the 
appropriate test files in it):

> make test-scripted-include
NOTE: You may ignore warnings about the following files:

     test-scripted-include.d

Makefile:1875: test-scripted-include.d: No such file or directory
= test-scripted-include.tex --> test-scripted-include.d 
test-scripted-include.dvi 
(1) =
= scripted-include-file.nobuild.tex.sh --> scripted-include-file.nobuild.tex =
= test-scripted-include.tex --> test-scripted-include.dvi (2) =
Success!  Wrote 1 page, 252 bytes

= test-scripted-include.dvi --> test-scripted-include.ps =
= test-scripted-include.ps --> test-scripted-include.pdf =

Original comment by shiblon on 4 Feb 2010 at 5:04

[GoogleCodeExporter]

I just tried the same thing (using onlysources.tex := 
test-scripted-include.tex) and it 
didn't loop for me.  Does it stop if you remove that?

Original comment by shiblon on 4 Feb 2010 at 5:06

[GoogleCodeExporter]

I'm afraid I don't really have time to test this right now. I'll try to give it 
a go
at some point and get back to you. I've reverted to the Makefile.ini I had 
before and
it seems to work very well :)

Thanks for all you help

Original comment by mojo...@gmail.com on 4 Feb 2010 at 5:19

[GoogleCodeExporter]

No problem.  I'm out of time for this today, as well.  When you get a chance, 
try this:

rd9670ac43004

It no longer runs touch on the script file, instead relying on a native make 
directive to 
force a build every time.  I think that might solve your problem.

Get back to me when you can, and I'll push the new version out if it's good.

Original comment by shiblon on 4 Feb 2010 at 5:33

[GoogleCodeExporter]

That sounds like a much cleaner solution. I'll give it a go when I have time.
Cheers

Original comment by mojo...@gmail.com on 4 Feb 2010 at 5:37

[GoogleCodeExporter]

Actually, scratch that - I was able to create an example that reproduces this 
issue.  I'm 
looking into it.

Original comment by shiblon on 4 Feb 2010 at 5:37

[GoogleCodeExporter]

OK.  I figured out what is going on, but that unfortunately doesn't allow me to 
fix it 
easily.  :-(

Here's the deal.  The makefile runs, then realizes that it doesn't have needed 
.d files 
included, so it builds those.  When finished, it *runs again*.  At that point, 
the 
dependencies of the .d files have been touched, so it does it again.  And 
again.  It 
appears to have a limit, though, because it eventually gives up running and 
goes on.  
It takes about 5 times for it to settle down.

This sucks, obviously, but as I thought about it, I realized that there is no 
easy way to 
indicate intent here: we want to build a .tex file every time it is needed, but 
not *too* 
often, because we don't want to build it every time the makefile is invoked, 
just once 
going into the root invocation, to be used for the final product.

Make doesn't allow that kind of nonlocal state (something that has driven me 
crazy 
in the past), so we have a couple of options here:

1) we just leave it as it is without sleeps, and files are only regenerated if 
their scripts 
are touched.  That has the disadvantage of not regenerating files every time 
they are 
needed, which is not what I'd like to see.  It is fortunately really easy to 
stick stuff into 
Makefile.ini that makes this work as needed, something like this:

  myfile.tex.pl: FORCE
  FORCE:

But I'm not really thrilled with that.

2) We could make builds unconditional (with the looping problem that we've 
already 
seen), but not actually perform the build steps unless the timestamp on various 
files 
is within a particular range.  This can be accomplished by writing out a cookie 
file 
with a timestamp in it for the script when the file is built, and checking for 
that 
cookie.  If the cookie doesn't exist, or the timestamp is too old, then the 
actual build 
steps are performed.  I like this solution, but hate that is such a dirty, 
dirty hack.

Thoughts welcome.  I'm working on 2) right now, just in case it works out.

Original comment by shiblon on 4 Feb 2010 at 6:59

[GoogleCodeExporter]

Your cookie suggestion reminds of a solution I had thought of involving the svn 
$Id$
tags.

What we could do is create a Makefile equivalent of the svn tags: You would put 
a
certain string in a comment in the script and the first time Make ran it would 
find
that string and add a serial number to it. i.e. the line
#$latex-makefile-serialno$
would expand to 
#$latex-makefile-serialno 00000000000000001$

Each time the makefile was run, this serial number would be incremented.

Now in our python script we add svn tags like this

svnid = "$Id$"[4:-1]

or whatever we need to get the svn tag into a variable, then we generate our 
latex code.

Here's the clever bit: subversion will now twig that the python script has 
changed
and increment the tags (provided that make has been run since the last commit.

Then, when we run make again, the Makefile will twig that the python script has
changed (because svn has updated the tags presumably updating the file 
timestamp) and
remake the latex code.

This would be a very elegant solution if we could get it working. The problem 
is: the
script changes on each make (serial number implemented) as well as on each svn 
update.

This might give you some more options on how to implement your cookie idea 
though.

J

Original comment by mojo...@gmail.com on 4 Feb 2010 at 7:18

[GoogleCodeExporter]

The nice thing about this is that in principal (if we can separate make editing 
the
file from svn editing the file) is that the svn.nomake.tex will really only be
regenerated after an svn commit.

Original comment by mojo...@gmail.com on 4 Feb 2010 at 7:20

[GoogleCodeExporter]

All of these ideas are great, but they all suffer from the same problem: how do 
you 
detect the "first time" make runs?  There is simply no way to do that.

Every solution I can think of requires detecting this, and that's a 
deal-killer.  There's 
just no way to be sure which invocation of make, or sub-invocation of make, we 
are 
currently in.  We can't reach out into the state of the process and ask it 
where we are 
in the dependency tree.

I think we've run into a fundamental limitation of make, here.  Unfortunately, 
that 
happens a lot with this Makefile.  It can only do so much, and we're already 
seriously 
pushing the limits of the technology with it.

What I have always done in the past in cases like this is create a simple 
wrapper 
script that calls make for me, e.g.:

#!/bin/bash
# build.sh
touch svn.nobuild.tex.pl
make

That would do just the right thing, since you really want to do something at 
the 
highest level of make invocation, a pre-processing step of ensuring that stuff 
changes, if you will.

So, I think that's my final recommendation for now.  We can use the new 
facility for 
scripts and the ability of the makefile to handle includes, coupled with a 
simple build 
wrapper script that touches the relevant script.

I really can't see any other way to do this.  The Makefile file is invoked so 
many times 
without any indication of which is the "root" invocation, that there is no 
clean way to 
do pre- and post-processing, which is at the heart of the problem here.

So, I think that r8542fa5b6691 is my final answer, here.  Scripted generation 
works 
fine, you just have to manage your dependencies outside of the makefile.  
Hopefully 
that's not too onerous.  We've just run headlong into a fundamental limitation 
of 
make (once again - sigh).

There is still the cookie option, but then I would need to drag in a whole 
bunch more 
technology, like a real scripting language (right now "sed" is the most 
advanced thing 
we use in this makefile, and BSD sed at that) that can get file timestamps and 
do 
numeric comparisons, etc.  Not something I would like to throw in there for 
this, 
honestly.

Another thought: we could send no parameters to the script indicating that we 
just 
want it to say "yes" or "no" to whether it needs to be run again, e.g.,

perl svn.nobuild.tex.pl

And it would return "yes" on stdout, or "no", as needed.  Then the timestamp 
computation would be done in the script.  Make would simply always try to 
rebuild 
the .tex file from it, but would invoke it once without parameters, then only 
execute 
the rebuild if it says "yes".

Thoughts?

Original comment by shiblon on 4 Feb 2010 at 7:49

• Changed state: Started

[GoogleCodeExporter]

Hey, I did it!  I got it working.  It turns out that make creates an internal 
variable 
called MAKE_RESTARTS, so I write that out to a cookie file when a script runs, 
and I 
only run the script if MAKE_RESTARTS is <= the value in that file.

It works like a charm, only executes the script once per make invocation, and 
doesn't 
require sleeping or touching stuff.  It does add a dependency on "expr", but 
that's a 
very basic system binary, so I don't feel too bad about including it in the 
binary 
dependencies.

Please try rd806d201457b for me, and let me know if this fixes things for you.  
You 
shouldn't see the loopy behavior anymore, and you should see your script run 
every 
time.

Original comment by shiblon on 5 Feb 2010 at 8:25

[GoogleCodeExporter]

As you say it seems to work perfectly! I've committed the new Makefile to my svn
archive ;)

Cheers,

J

Original comment by mojo...@gmail.com on 5 Feb 2010 at 10:12

[GoogleCodeExporter]

Excellent!  I'm uploading the official download version 2.1.43 now with that 
fix.  Thanks 
for the patience in working through this difficult issue.

Original comment by shiblon on 5 Feb 2010 at 11:11

• Changed state: Verified

[GoogleCodeExporter]

I have found a really serious issue with declaring the script file as PHONY.  
This is 
going to cause an infinite loop even with the fix I put it for the following 
reason:

.d file is imported
To create it, you need .tex
To get that, you need .tex.pl or whatever, which is PHONY
So .tex is built
Necessitating a rebuild of .d
Necessitating a rebuild of .tex
and so on forever until the OS gets things built fast enough that the 
timestamps 
don't differ enough, or until make breaks the recursion explicitly.

So, I took this out :(  The logic for avoiding infinite make level recursion is 
still there, 
but in many cases this just doesn't work properly.

The accepted approach to causing scripts to always be run is to create a 
Makefile.ini 
and put the following in it:

ALWAYS_RUN_SCRIPTS := my_critical_script.tex.pl

Or, if you want the old behavior, you can do

ALWAYS_RUN_SCRIPTS := $(all_files_scripts)

I don't recommend the latter, though.  Leaving bug closed.  rf4607b964265 has 
the 
new behavior.

Original comment by shiblon on 22 Mar 2010 at 2:17

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I take it back.  I thought about this longer and harder and fixed it properly 
this time.  It 
should work much better in r6607787dc0b5.

As a bonus, rst sources now work properly (I wasn't thinking of them as 
scripts, but 
they really are, since they generate .tex).

Original comment by shiblon on 23 Mar 2010 at 5:43