Allow Email Reports to be sent in HTML

[julienmoumne]

See /trac/ticket/151#comment:11

Actions :

• Refactor PDFRenderer as an implementation of an abstract report renderer class (Piwik_ReportRenderer)
• Move PDFRenderer to core/ReportRenderer and rename it to Piwik_ReportRenderer_Pdf)
• Add an HTML implementation to ReportRenderer (Piwik_ReportRenderer_Html)
• Update PDFReport UI & database to allow sending reports in HTML

[julienmoumne]

Attachment: gmail-piwik-logo.jpg

[julienmoumne]

Attachment: gmail-browsers.jpg

[julienmoumne]

Attachment: msn-piwik-logo.jpg

[julienmoumne]

Attachment: msn-browsers.jpg

[julienmoumne]

Attachment: outlook2010-piwik-logo.jpg

[julienmoumne]

Attachment: outlook2010-browsers.jpg

[julienmoumne]

Attachment: thunderbird3piwik-logo.jpg

[julienmoumne]

Attachment: thunderbird3-browsers.jpg

[julienmoumne]

Attachment: yahoomail-piwik-logo.jpg

[julienmoumne]

Attachment: yahoomail-browsers.jpg

[julienmoumne]

Attachment: fixes#2318#2320.patch

[julienmoumne]

I have almost finished working on this ticket.

Images (Piwik logo, browser logos, ..) in emails have to link to a Piwik instance so they can be downloaded by email clients.

Should we add a config option with the URL of the piwik instance to use ? Or should we use

$_SERVER['SERVER_NAME']

or some sort?

[julienmoumne]

I understand changing file names of a plugin is virtually impossible. (PDFReports -> EmailReports).

However, Is it ok to change

'name' => 'Pdf Export Plugin',

to

'name' => 'Email Report Plugin',

?

Same goes for

'PDFReports_PluginDescription' => 'Create and download your custom PDF reports, and have them emailed daily, weekly or monthly.',

'PDFReports_DefaultPDFContainingAllReports' => 'Default PDF containing all available reports.',

and many others

?

What about translations ?

[robocoder]

Replying to JulienM:

Should we add a config option with the URL of the piwik instance to use ? Or should we use

$_SERVER['SERVER_NAME']

or some sort?

See Piwik_Url.

[robocoder]

re: comment:2 it's easy to rename a plugin but a bit messy for the update script, and a PITA for translators.

That said, since pdf reports can also be downloaded, I would prefer simply "Reports".

[julienmoumne]

Piwik_Url is what I was looking for thanks.

"Reports" is more correct indeed.

For translations, how about removing the translation for other languages so it appears as "not translated" in the translation ui ?

[julienmoumne]

I've read the code managing database updates. I understand there is a way for plugins to define database updates the same way core does.

I have to add 'format' column to the pdf table. Should this go in core or plugin DB updates ?

[robocoder]

core/Updates/

[mattab]

Excellent work mate...!

I tested report for komodo island and it worked well :)

Some feedback:

• could logos have padding/margin right in outlook?
• more padding in the first row
• reuse colors / fonts / font-sizes of current Piwik UI so it looks as close as possible to current HTML
• we could also modify PDF so they look the same as well
• OK to modify translations for sure, but when you modify translations just change the Index of the translation, otherwise other 'outdated' translated strings will be reused. For example 'PDFReports_PluginDescription' could become 'PDFReports_PluginDescriptionReports' eg.
• while "Reports" is maybe more correct, "Email reports" is better because it advertises the main purpose of the feature, which is to email reports. Downloading reports is rarely useful and mostly there for debugging
• OK to put Updates in core/Updates rather than plugins/ as this is how we do it for all core plugins (when we don't it's either to show how it works, or by mistake) I also vote for including in 1.4 as 1.3 is now final and will be released anytime.

[julienmoumne]

Now the ground work is done it's very easy to add new formats. I'm going to implement Piwik_ReportRenderer_Odt (using http://www.odtphp.com/) if that's ok.

[julienmoumne]

• logo padding ok for all clients
• TH padding ok for all clients
• new scheme applied to html and pdf as well
• added core/Updates/1.4-rc1.php

[robocoder]

It looks like the ODTPHP is license incompatible with the GPLv3. Also, development on the library seems to have tapered off -- what version of the Open Document Format does it support?

In the meantime, feel free to add support for odt, but check for the presence of the library in the event we can't include the library in the distribution.

[julienmoumne]

How about http://phpdocwriter.sourceforge.net/ ?

[robocoder]

phpdocwriter is LGPL (ok), but also hasn't been updated in a while.

If odtphp is a better library, you could try asking the devs to update the license (eg from "GPLv2" to "GPLv2 or above").

[julienmoumne]

phpdocwriter generates .sxw files [http://phpdocwriter.sourceforge.net/phpdocwriter/examples/ex6.php] so I'm going to send an email to the developers of ODTPHP

[julienmoumne]

On this page [they link to http://www.gnu.org/copyleft/gpl.html.

In [they also link to http://www.gnu.org/copyleft/gpl.html.

[http://www.gnu.org/copyleft/gpl.html] seems to be v3.

Do they only need to update the text in [http://www.odtphp.com/index.php?i=credits] from

OdtPHP is released under the terms of the GNU Public License. To get more informations about the GPL license, click here. 

to

OdtPHP is released under the terms of the GNU Public License. To get more informations about the GPLv3 license, click here. 

?

[robocoder]

It must have been the sleep deprivation -- I swear I saw "v2" somewhere. Thanks for double checking. Go head.

[julienmoumne]

I'm running into a bit of trouble with that library.

To zip/unzip odt data they use the PclZip library.

With the latest packaged version (1.0.1) it is not possible to override the temp directory used by PclZip.

Looking at the latest revision http://odtphp.svn.sourceforge.net/viewvc/odtphp?view=revision&revision=51 second point:

- Option to choose tmp directory by appplication works correctly now

shows they have fixed the issue.

However, it has not been packaged yet.

Should I update the file myself or drop the idea of having odt export for lack of quality libraries?

[robocoder]

grab the latest from their svn

[julienmoumne]

I should have investigated before. This library doesn't allow table generation with a variable number of columns.

They suggest to use http://pear.php.net/package/OpenDocument/

[julienmoumne]

No table support in this library.

[mattab]

As per discussion I think safer not to implement ODT at this point.

[julienmoumne]

Updates:

• generate report as inlined content (outputType=0) is dropped. This feature is not that useful, not future proof (can any format be inlined in the browser?) and necessitate a Null Renderer so the ResponseBuilder doesn't append the success message at the end of the content.
• toc is kept for HTML but removed for PDF (bookmarks are good enough)

[julienmoumne]

Here is the proposed patch

• @Review markers are remaining questions
• includes r4415, r4505, r4506 (PDFRenderer & API).
• styles are inlined because gmail doesn't support other means
• fixes #2320

[mattab]

Feedback

• @Review, should this be abstracted in a Piwik_EmailReport_x ? looks good as is
• rename _epilogue to _footer for consistency?
• function Piwik_Smarty() should be function __construct()
• great work on the Piwik_ReportRenderer refactoring!
• the comment " * To render a report as a pdf file, simply do: [...]" is not useful, because users will have to use the API to get any report, not use the low level classes. Only core will use these, and the code is clean & self documented as it is
  • I propose to change $outputFilename = PIWIK_INCLUDE_PATH . '/tmp/' . $filename; to use /tmp/assets instead. there have been permission problems with deleting files in tmp/ which I think will work better in tmp/assets since this directory is created by the webserver user, therefore there will be better umask there (or am I wrong...?)
• @Review : Code inspired from TCPDF Plugin Here I wouldn't put all the caching headers etc. just keep it simple (by design, none of the API is cached at present). There is currently one piece of code that allows download of a file, it is in Csv.php > renderHeader() which is a lot simpler than TCPDF code (and similar header to Csv.php is probably enough for serving HTML content)
  • don't forget to change to // $smarty->assign("currentHost", "http://".Piwik_Url::getCurrentHost()); ;)
• are both _epilogue.tpl and _header.tpl or could it just be _header.tpl?
• when I generate an HTML report, only the Logo/intro and TOC are displayed, no reports are displayed. But maybe this is because the patch didn't apply properly or something..?
• in the UI, when report is set to HTML, can you display a HTML logo, for example this one looks nice.
• also would it be possible to capitalize HTML and PDF in the UI
• and show the icons in the SELECT box as well as before the Download button?

After these modifications, OK to commit. Very good code, great work Julien!! :)

TODO post commit

• add integration tests to test HTML output
  • when 1 report: no TOC
  • with all reports that are metadata compatible

(I can do it when this is committed)

[julienmoumne]

(In [4512]) fixes #2318 #2320

[robocoder]

(In [4516]) refs #2318

[mattab]

(In [4517]) refs #2318 Prevent "Notice undefined index" for existing (PDF) reports

[mattab]

(In [4518]) refs #2318 xss protection + working if piwik installed in sub dir (eg. in http://localhost/piwik)

[mattab]

(In [4519]) Adding link on logo. refs #2318 I think this feature is done, kuddos Julien! I will just try and add integratio tests on the html output

[mattab]

Reopening, to not forget: HTML wont work correctly when sent from cron, since the hostname can't be read most of the time. We need to record the piwik hostname+path in a flag in the DB, that would be used in place in $smarty->assign("currentPath", Piwik_Url::getCurrentUrlWithoutFileName() );

[mattab]

(In [4552]) Fixes #2318, #2325 - URL will be checked for in each piwik request to UI and API (not tracker), and will be set if not already set. then, we use this URL for the HTML email images & link to the piwik reports

[mattab]

(In [4568]) Fixing Notice:</strong> <i>Undefined variable: reportFont</i> in <b>/home/www/dev5.piwik.org/core/ReportRenderer/Pdf.php</b> on line <b>95</b> Refs #2318

[mattab]

Regression in PDF: in Greek for example, letters now appear as ???? after 1.4-rc2 upgrade (reproducible on localhost). Before, it used to display OK. Maybe a refactoring miss out?

[julienmoumne]

(In [4577]) fixes #2318 refs #2320 : revert font to the one used before trying to make style uniform

[julienmoumne]

(In [4619]) refs #2318 fixing font-size incorrectly interpreted in gmail

[julienmoumne]

(In [4659]) refs #2318 font & size enhancements

[julienmoumne]

(In [5313]) refs #2318 fix for r4518, xss protection broke URLs in HTML Reports, applying same protection as in datatable_cell.tpl

[julienmoumne]

(In [5415]) fixes #2706

• refs #2318, #71 : Graphs now supported
• refs #2670, #898 : Restoring VisitsSummary report metadata before eCommerce
• refs #1721 : Additional security, filename fixes and new internal outputType

[julienmoumne]

(In [5582]) * fixes #2706, #2828, #2704, refs #1721, #2637, #2711, #2318, #71 : horizontal static graph implemented

• fixes #2788, refs #2670, #1721, #2637, #2711 : default graph type logic moved to ImageGraph API, improved date/period logic, new parameter graphs_default_period_to_plot_when_period_range
• fixes #2704, #2804, refs #1721 : pChart updated to 2.1.3, pChart code removed from Piwik code, OOP refactoring, support for unifont.ttf if present in ImageGraph/fonts, testAllSizes now uses report metadata ImageGraphUrl
• refs #5491 : space between report title and report table reduced to avoid page overflow
• refs #2829 : TODO display percentages
• r5544, r5547, r5549 merged