Thruk v3.02 - Report not rendering PDFs & Emails not working

[MattCrum1]

Describe the bug I am running Thruk & Naemon in an AlmaLinux container - trying to use the Report functionality to email an active Service problems report every day based on a URL and am having 2 problems:

1) Report does not render PDFs

I'm using the URL:

/thruk/cgi-bin/status.cgi?host=all&servicestatustypes=28

The report renders fine in HTML, but is about 15MB; not suitable for emails. I switched to PDFs and the generated file is 3 pages of white space with a couple of navigation elements (scrollbars etc); no service information.

I've tried using the absolute URL https://servername:port/thruk/cgi-bin/status.cgi?host=all&servicestatustypes=28 but the behaviour is the same.

2) Report is not emailed Emails from Naemon work fine, but the report emails do not. I have sendmail configured on the server to use our internal mail relay and testing it using 'echo "test" | sendmail -s subject my@email.address' works fine. Using the "Send Report By Email" feature results in a page hang for 1 minute, then returns to the reports page with no errors on screen. See Thruk log below.

Thruk Version 3.02

To Reproduce Steps to reproduce the behavior:

1. Create a report like this with the URL /thruk/cgi-bin/status.cgi?host=all&servicestatustypes=28
2. Create a report preview with PDF disabled and view - it works, but is around 15MB
3. Create a report preview with PDF enabled and view - it creates a blank page with navigation elements displayed
4. Click Send Report By Email and email the report - the UI loads for about 60 seconds, then returns to the Reports page - no email is received

Expected behavior Report PDF is generated correctly Email is sent

Additional context Thruk.log showing the long running task:

[2023/04/19 10:51:59][423247e9e57f][WARN] slow_page_log_threshold (15s) hit, page took 60.4s to load. [2023/04/19 10:51:59][423247e9e57f][WARN] page: https://xxxxxxxxxxxxxx/thruk/cgi-bin/reports2.cgi [2023/04/19 10:51:59][423247e9e57f][WARN] params: {'CSRFtoken' => '...','action' => 'email','cc' => '','desc' => 'Summary of xxxxxxxx','js' => 'no','language' => 'en','minimal' => 'yes','nav' => 'no','pdf' => 'yes','report' => '2','send' => '1','subject' => 'Report: Daily DDI M... [2023/04/19 10:51:59][423247e9e57f][WARN] user: xxxxxxxxxx [2023/04/19 10:51:59][423247e9e57f][WARN] address: xxxxxxxxxx [2023/04/19 10:51:59][423247e9e57f][WARN] +----------------------------------------------------------------------------------+-------------+ [2023/04/19 10:51:59][423247e9e57f][WARN] | total time | 60.34885s | [2023/04/19 10:51:59][423247e9e57f][WARN] +----------------------------------------------------------------------------------+-------------+ [2023/04/19 10:51:59][423247e9e57f][WARN] | _dispatcher: https://xxxxxxxxxxxxxxxl/thruk/cgi-bin/reports2.cgi | 60.34860s | [2023/04/19 10:51:59][423247e9e57f][WARN] | time: Wed Apr 19 10:50:59 2023 - host: 423247e9e57f - pid: 15947 - req: 1752 | | [2023/04/19 10:51:59][423247e9e57f][WARN] | Root begin | 0.00730s | [2023/04/19 10:51:59][423247e9e57f][WARN] | User::set_dynamic_attributes | 0.00077s | [2023/04/19 10:51:59][423247e9e57f][WARN] | Thruk::Controller::reports2::index | 60.33884s | [2023/04/19 10:51:59][423247e9e57f][WARN] | AddDefaults::add_defaults(ADD_CACHED_DEFAULTS) | 0.23508s | [2023/04/19 10:51:59][423247e9e57f][WARN] | AddDefaults::get_proc_info | 0.23036s | [2023/04/19 10:51:59][423247e9e57f][WARN] | AddDefaults::set_processinfo | 0.23002s | [2023/04/19 10:51:59][423247e9e57f][WARN] | AddDefaults::set_processinfo fetch | 0.22925s | [2023/04/19 10:51:59][423247e9e57f][WARN] | _do_on_peers(get_processinfo) | 0.22788s | [2023/04/19 10:51:59][423247e9e57f][WARN] | _get_result_parallel(328c5,6680b,00a4c,9bce3,5214a,5090a,eb527,b | 0.22737s | [2023/04/19 10:51:59][423247e9e57f][WARN] | slowest site: 5090a -> 0.2091 | | [2023/04/19 10:51:59][423247e9e57f][WARN] | _merge_answer() | 0.00019s | [2023/04/19 10:51:59][423247e9e57f][WARN] | Utils::Menu::read_navigation() | 0.00340s | [2023/04/19 10:51:59][423247e9e57f][WARN] | Utils::Reports::get_report_templates() | 0.02291s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00164s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00145s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00162s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00153s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00153s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00208s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00152s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00150s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00152s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00153s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00135s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00073s | [2023/04/19 10:51:59][423247e9e57f][WARN] | Utils::Reports::report_send() | ~60.07724s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00077s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00072s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00108s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: get_variable.tt | 0.00112s | [2023/04/19 10:51:59][423247e9e57f][WARN] | render: reports/report_from_url.tt | 0.00135s | [2023/04/19 10:51:59][423247e9e57f][WARN] | Root end | 0.00160s | [2023/04/19 10:51:59][423247e9e57f][WARN] | finalize_request | 0.00025s | [2023/04/19 10:51:59][423247e9e57f][WARN] +----------------------------------------------------------------------------------+-------------+ [2023/04/19 10:51:59][423247e9e57f][WARN] | total time waited on backends | 0.22717s | [2023/04/19 10:51:59][423247e9e57f][WARN] | total time waited on rendering | 0.00000s | [2023/04/19 10:51:59][423247e9e57f][WARN] | total time | 60.34885s | [2023/04/19 10:51:59][423247e9e57f][WARN] +----------------------------------------------------------------------------------+-------------+

[Napsty]

I also noticed a problem creating PDF reports since upgrading from Thruk 2.x to 3.04. thruk report 1 created a PDF before, now the output is [1.rpt] report calculated successfully in 0.4s. The file /var/lib/thruk/reports/1.rpt is not a PDF either, it contains JSON data.

[sni]

i guess that's something else. The first report takes exactly 60seconds, so i assume some kind of timeout.

[Napsty]

Agree, probably not related. However the text "report not rendering pdfs" also applies to this case.

For the PDF rendering I'm now using curl and the reports2.cgi as a workaround.

root@thruk:~# curl -s "http://localhost/thruk/cgi-bin/reports2.cgi?report=1" -u thrukadmin:secret -o /tmp/1.pdf

[sni]

i assume this is due to phantomjs not beeing able to parse recent html/css. I will deprecate phantomjs with the next release and switch to puppeteer for all png/pdf conversions.

[danshome]

Hello @sni,

I wanted to bring to your attention some high-level findings after doing some initial research, although I must admit that I may be making some assumptions due to time constraints.

In addition to the PDF issues you mentioned, I noticed some potential issues with sending HTML files. Specifically, when sending an email, the status.html file appears to have a calculated size of 14.7k in the send email dialog, but the email size is actually 14.7MB upon delivery. Upon further investigation, I found that all fonts are being embedded in the HTML as data URLs, causing an unnecessarily large file size. Is it possible that the status.html is being generated and then converted to a PDF using phantomjs? I could see this causing unexpected behavior due to the size of the source file caused by the embedded fonts. To improve efficiency, it may be worth considering including just font-awesome.min.css and allowing the client to render the necessary fonts.

Additionally, multiple language fonts seem to be embedded despite the language being set to English. While embedding fonts may be necessary to ensure the report is visible even without internet access, the estimated attachment size should include the fact that all the fonts are being embedded and calculate the size with the embedded font. It is important to note that a 14MB HTML report for a single-page HTML report seems excessive, and hopefully, fixing the language issue will significantly reduce the size.

Lastly, it would be useful to have a way to set the default reply-to address for sending emails without configuring map files on the local mail server, and it would be great if either sendmail or mail could consistently be used instead of both. Up until trying to test sending some reports I didn't need sendmail installed, maybe try mail and if it's not found try sendmail?

I hope this information is helpful to you. Cheers!

[sni]

thanks for bringing this up. There went at least two things wrong here. It seems like the css fonts have been included twice (fixed in dab7ba1c6805fcc8183e3b2732d4579302724320). This cuts the size in half already.

Then i removed all but one src url from those font-faces (fa96085569b72b33e2b7b031c4cc2504398aba7b). This saved another 5mb. So in total, a status.html report now takes ~2mb instead of the 15mb before.

But yes, you can always convert this in to a pdf which uses ~500kb. Simply set "Direct PDF" to yes in the report options.