Run time performance needs improvement.

[xsgordon]

Run time performance is currently quite slow. Processing the CLI Guide, which is relatively trivial compared to some others, takes around 6 minutes. The vast majority of this time is spent performing the "all" and "common" XSL transforms. It seems likely the best two ways to improve the situation are:

• Parameterize and consolidate the two XSL files to avoid parsing twice, this will need to be done with care as there are overlapping XSL match conditions in the two files.
• Determine a way to identify pre-transform any XML files which are in the tree but are not included in the book itself, this is most commonly an issue with the files from the "common" XML directory which are currently all included in processing.

[xsgordon]

Running print_unused early in the process seems to be undesirable or even impossible, doing so requires that the product number and revision history transformations have already run, as well as the copying of publican specific XML files into the tree.

The problem is that copying these files in ahead of time means they will themselves be detected as being unused and/or included in transforms, resulting in breakage. Alternatives are either a more tolerant implementation of a routine for detecting unused files or being more stringent about determining which files are actually copied in from common.

[xsgordon]

A transform to extract all xi:includes in the document for basic-install takes nearly 2 minutes using Saxon, it's possible performance with XSL 1.0 processor and transform would be faster but there are other transforms in use that require 2.0 anyway.

[xsgordon]

From 10 runs against the basic-install guide with no arguments:

6m29.063s 6m10.887s 6m23.149s 6m23.076s 6m22.027s 6m28.256s 6m12.033s 6m19.699s 6m17.699s 6m24.703s

[xsgordon]

From 10 runs against the basic-install guide with no arguments (patch applied):

4m37.267s 4m37.941s 4m36.075s 4m36.730s 4m35.945s 4m36.457s 4m28.585s 4m26.324s 4m22.629s 4m22.703s

Significant improvement (~1.5 to 2 minutes on average) but still not what you would call "quick". Next step is to see if any particular transform is bogging things down or whether this is just the cost of doing business with the XSL processor.

[xsgordon]

Dropping the language transform buys another 20-30 seconds.

[xsgordon]

Dropping the os to condition transform had negligible impact.

[xsgordon]

Making the language selector more specific (programlisting[@language] V @language) didn't seem to make a difference. Actually on running the benchmarks again it actually seems to make the average run time about 10 seconds worse which is a tad surprising.

[xsgordon]

I've had to remove the call to publican print_unused as a result of a Publican bug so that has expanded build times somewhat.

[xsgordon]

Next step is to prune the tree pre-transform. This presents somewhat of a chicken and egg problem because at this time the tree isn't actually a valid XML document because the transforms haven't run. Some educated guessing can be done here though which will likely bring transformation times down quite a bit.

• List files in common (base name only).
• Grep for references to said files (base name only).
• Remove files for which no references are found.

This only addresses unused files in common content but this represents the vast majority of cases, most other files exist in the tree because an author explicitly but them there and linked to them whereas we blindly copy all of common up front.

When it's fixed the print unused files at the end can be reinstated and used to perform any final clean up of files missed by this basic check.

[xsgordon]

Unfortunately as the basic install guide no longer exists it's probably going to be necessary to run some new benchmarks which wont be comparable to the old ones.

[xsgordon]

Averages from 10 runs against the config-reference guide. Column groupings (M,S, S (Total)) are (in order):

• Real time.
• User time.
• System time.

Run	M	S	S (Total)	M	S	S (Total)	M	S	S (Total)
1	5	40.950	340.950	5	36.646	336.646	0	38.399	38.399
2	5	44.269	344.269	5	38.450	338.450	0	38.813	38.813
3	5	46.041	346.041	5	44.838	344.838	0	35.483	35.483
4	5	42.157	342.157	5	38.803	338.803	0	38.254	38.254
5	5	37.024	337.024	5	30.774	330.774	0	39.453	39.453
6	5	36.604	336.604	5	30.201	330.201	0	40.584	40.584
7	5	39.057	339.057	5	34.382	334.382	0	38.463	38.463
8	5	47.786	347.786	5	43.265	343.265	0	35.996	35.996
9	5	47.541	347.541	5	39.581	339.581	0	41.059	41.059
10	5	41.061	341.061	5	35.220	335.220	0	39.648	39.648
Averages	5	42.249	342.249	5	37.216	337.216	0	38.615	38.615

[xsgordon]

The biggest time sink is without a doubt the running of the XSL transformation, so what is required is further investigation to determine if there is any way to further optimize or consolidate the transforms in xsl/transform/all.xsl. These are run on each and every file in the tree.

[xsgordon]

Moving this out of the 0.3.0 milestone as I don't realistically see any significant changes occurring here for a while.