Performance issue when compiled template includes whole file multiple times

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Have very slow file system lookups
2. Have a compiled no-cache Smarty template with a lot of include statements
3. Get slow response

What is the expected output? What do you see instead?
The compiled template to use include_once for plugins to prevent redundant 
loads. Instead I get a compiled template that includes the same files multiple 
times.

What version of the product are you using? On what operating system?
3.1.11, LAMP stack

Please provide any additional information below.

From an email with a sysadmin:

-------
Note particularly, that the file

/.../uvm.edu/fs/a7/users/s/u/summer/public_html/enginece/3.0/lib/Smarty-3.1.11/l
ibs/plugins/shared.mb_str_replace.php
is looked up 2736 times in a single request!  There might be some optimizations 
you can make there..
-------

It appears lines 121 and 130 of smarty_internal_compile_function.php are the 
culprits, but I haven't tried anything yet.

Original issue reported on code.google.com by jrbea...@gmail.com on 23 May 2014 at 7:06

[GoogleCodeExporter]

For v3.1.11, the following lines contribute include statements:
smarty_internal_compile_function.php:121
smarty_internal_compile_function.php:130
smarty_internal_template.php:347
smarty_internal_template.php:358

I attempted to change the compiler output to include_once, but it didn't 
significantly affect output time. Ideally, the compiler would maintain a queue 
of includes and prepend them to the output rather than checking if they don't 
exist yet at that point in the output. 

I.e. a file shouldn't need to be looked up 2.7k times.

Original comment by jrbea...@gmail.com on 23 May 2014 at 7:41

[GoogleCodeExporter]

The lines you mentioned are not the problem.

I suspect that you are using the escape modifier plugin.

I does load shared.mb_str_replace.php by require once. 

It looks like that someone tried some optimizations to load 
shared.mb_str_replace.php only with for the options needed, but this could end 
up in calling require_once on each call of the modifier.

Try to change the code in modifier.escap.php to

if (!function_exists('smarty_mb_str_replace')) {
   require_once(SMARTY_PLUGINS_DIR . 'shared.mb_unicode.php');
}

Please give me some feedback.

Original comment by Uwe.Tews@googlemail.com on 24 May 2014 at 10:03

• Changed state: Accepted

[GoogleCodeExporter]

Verified that moving the include dependencies in the compiled template to the 
top of the file increased performance. Tried separately with your suggestion 
(while also commenting out all other require_once calls), and it didn't show a 
significant increase in performance. I'm going to go document the number of 
individual plugin calls in the template.

Original comment by jrbea...@gmail.com on 27 May 2014 at 1:36

[GoogleCodeExporter]

I'm using two modifiers: the escape modifier and the replace modifier.

Escape for obvious reasons.
Replace to insert primarily   (because browsers don't like UNICODE 
non-breaking space), but also <br> in the weekdays instance.

section-filter.tpl
==================

* subjects.length = 96
* instructors.length = 391

* escape (form action)
* escape `2 * subjects.length` (option names, values)
* escape (text field value)
* escape `instructors.length` (option values)

`1 + 96 * 2 + 1 + 391 = 585 escape`

section-offers.tpl
==================

* htmlOffers.length = 1
* urlOffers.length = 7

* escape `htmlOffers.length + 2 * urlOffers.length` (ui-tabs labels, urls)

`1 + 2 * 7 = 15 escape`
`585 + 15 = 600 escape`

section-table.tpl
=================

n/a, see child templates

section-tbody.tpl
=================

* sections.length = 578
* fullSections.length = ~50
* section.instructors.length = [0..5], usually 1 (per section)

* escape `sections.length` (short title)
* replace `sections.length` (short title)
* escape `sections.length` (url)
* escape `sections.length` (long title)
* escape `fullSections.length` (url)
* escape `section.instructors.length` (list-item, name)

`578 + 578 + 578 + 50 + 578 = 2362 escape`
`600 + 2362 = 2962 escape`

`578 replace`

section-tbody-meeting-pattern.tpl
=================================

* section.meetingPatterns.length = [1..13], usually < 4 (per section)

* replace `2 * section.meetingPatterns.length` (start, [end] date)
* replace `section.meetingPatterns.length` (meeting weekdays)

`2 * 578 * 2 + 578 * 2 = 3468 replace`
`3468 + 578 = 4046 replace`

TOTALS
======

~2962 escape calls
~4046 replace calls

Original comment by jrbea...@gmail.com on 27 May 2014 at 2:22

[GoogleCodeExporter]

Compiled section-table.tpl has 3 explicit PHP include statements, it's child 
templates, section-tbody.tpl and section-tbody-meeting-pattern.tpl are within 
Smarty inline includes.

And that's pretty much what I can provide you with for documentation of this 
particular case.

If I were to go about modifying Smarty, I would:

* Create a dependency list in the compiler
* Whenever a necessary PHP include is encountered while compiling, add it to 
the dependency list
* Unconditionally require all dependencies in the list at the top of the 
compiled template.

I see in the plugins directory files like modifiercompiler.escape.php that look 
like they're supposed to get around dependencies like that. I don't see 
modifiercompiler.replace.php though, which might be included in a more recent 
version?

Original comment by jrbea...@gmail.com on 27 May 2014 at 2:44

[GoogleCodeExporter]

I looked at the modifier.replace.php and shared.mb_str_replace.php. It appears 
that smarty_mb_str_replace is supposed to be a multibyte-safe wrapper for 
str_replace. However, in the [PHP 
documentation](http://php.net/manual/en/function.str-replace.php#refsect1-functi
on.str-replace-notes), ***it states that vanilla str_replace is binary safe,*** 
which is probably why it isn't implemented in the multibyte string library to 
begin with. Of course there might be some other reason for having the 
smarty_mb_string_replace function that I'm not aware of (though it certainly 
isn't the possible absence of the multibyte string library).

That being said, the added overhead of function logic in 
shared.mb_str_replace.php is probably why there isn't a 
modifiercompiler.replace.php.

Original comment by jrbea...@gmail.com on 27 May 2014 at 3:09

[GoogleCodeExporter]

I created a modifiercompiler.replace.php that simply uses str_replace and that 
has shaved off a lot of time, but the page is still slow to load. I'm going to 
see where the bottleneck might be.

Original comment by jrbea...@gmail.com on 27 May 2014 at 4:45

[GoogleCodeExporter]

Yea, it's a huge improvement for the Smarty component to just use str_replace 
and to also have a modifiercompiler.replace.php

Original comment by jrbea...@gmail.com on 27 May 2014 at 4:54

[GoogleCodeExporter]

Here's a comment on the multibyte string library page at PHP.net that might 
help understand what the scope is in regard to str_replace:

http://www.php.net/manual/en/ref.mbstring.php#109937

Original comment by jrbea...@gmail.com on 27 May 2014 at 8:29

[GoogleCodeExporter]

Note that you can use also any PHP function as a modifier. Because of the 
different paramter ordering the template code might look a bit strange like

{'search'|str_replace:'replace':$foo}

Here some other performace hints:

A)
As I understand you call the section_xxx  subtemplates many times. Creating the 
context of a subtemplate might be also a major bottleneck. I would implement 
template functions instead of subtemplates.
See http://www.smarty.net/docs/en/language.function.function.tpl and 
http://www.smarty.net/docs/en/language.function.call.tpl

You can put all template function into one subtemplate as a library which you 
include once in your main main template. I think this would be a hughe 
perfomance boost.

B)
Instead loading modifier from plungins you can register local functions as 
modifier.
See http://www.smarty.net/docs/en/api.register.plugin.tpl

C)
Check if instead running modifier on the individual variables an output filter 
which run once on the complete output would be an option.
See http://www.smarty.net/docs/en/plugins.outputfilters.tpl and 
http://www.smarty.net/docs/en/api.register.filter.tpl

I think option A) will brig the highest performance boost.

Original comment by Uwe.Tews@googlemail.com on 27 May 2014 at 10:06

[GoogleCodeExporter]

A is interesting. I use the inline on the subtemplates of section-table.tpl and 
believed that would take care of the overhead. What would account for the 
difference between template functions and inline includes with a provided 
context?

With B, I'm aware of that, however at this point I'm thinking of the Smarty 
code rather than my own code. I've already resolved this issue in my own code 
by altering the replace modifier as I've described, which resulted in the 
template code running about 5 times faster.

C wouldn't be an option.

I'm trying to wrap my mind around how the smarty_mb_str_replace function would 
be much different from the built-in str_replace function (aside from the 
performance hit). PHP's str_replace works with UTF-8 because UTF-8 is designed 
such that any byte sequence can only represent a single character.

Problems would arise when you have mixed character sets. ASCII and UTF-8 get 
along fine. Mixing ISO-8859-1 (i.e. extended ASCII) and UTF-8 is where you 
could potentially have issues. However, smarty_mb_str_replace doesn't even 
address those issues.

Here's code that demonstrates how both str_replace and smarty_mb_str_replace 
break identically in that exact situation. Running the same with a different 
mb_internal_encoding doesn't change anything, and with a different output 
encoding it will have different results:

$searchISO = mb_convert_encoding("¢", "ISO-8859-1");
$replaceISO = mb_convert_encoding("¢", "ISO-8859-1");
$subjectISO = mb_convert_encoding("¢£", "ISO-8859-1");

$searchUTF = mb_convert_encoding("£", "UTF-8");
$replaceUTF = mb_convert_encoding("£", "UTF-8");
$subjectUTF = mb_convert_encoding("¢£", "UTF-8");

echo str_replace($searchISO, $replaceUTF, $subjectUTF) . PHP_EOL;
echo str_replace($searchUTF, $replaceISO, $subjectUTF) . PHP_EOL;

echo smarty_mb_str_replace($searchISO, $replaceUTF, $subjectUTF) . PHP_EOL;
echo smarty_mb_str_replace($searchUTF, $replaceISO, $subjectUTF) . PHP_EOL;

Essentially, smarty_mb_str_replace was created for a misinterpreted problem. 
Making sure that character encodings are the same when performing str_replace 
will work fine unless you are using a multibyte character encoding where byte 
sequences are ambiguous (which isn't an issue for utf formats).

Original comment by jrbea...@gmail.com on 28 May 2014 at 3:30

[GoogleCodeExporter]

Issues with Shift_JIS encoding, which is still pretty popular for Japanese, 
Chinese, and Korean, would be (taken from Perl ShiftJIS::Regexp documentation): 
mismatching a single-byte character on a trailing byte of a double-byte 
character, or a double-byte character on two bytes before and after a character 
boundary.

For the replace modifier plugin, you'd still have UTF-8 compatibility if you 
used PHP's built-in str_replace function.

Original comment by jrbea...@gmail.com on 28 May 2014 at 3:53

[GoogleCodeExporter]

If you inline subtemplates it does merge the compiled code of all template into 
the compiled file of the main template. The performace gain is that only one 
compiled file has to be loaded. You can still use all attributes of the 
{include ....} tag. Each template has its own template object and they are 
completely isolated from each other.

Template functions are similar to subroutines. They run in the context of the 
calling template. The template functions can see all variables of the calling 
template (same as subtemplates). They have an own variable scope, so variables 
assigned within the function are not seen in the outside world. And you can 
pass variables as parameter.

I'm shure that it will boost your application a lot.

Original comment by Uwe.Tews@googlemail.com on 29 May 2014 at 8:48