search

Icinga / icingaweb2-module-pdfexport

PDF export functionality for Icinga Web 2
GNU General Public License v2.0
31 stars 16 forks source link

PDF does not generate #17

Closed speedrz closed 4 years ago

speedrz commented 5 years ago

Issue

PDF Export Module is installed in Icinga so I can download SLA reports. The binary is pointed to /usr/bin/google-chrome-stable.

When clicking the Download > PDF link in Icinga a new tab opens but nothing further happens (no PDF is generated or downloads, tab header just says 'Loading...' but never completes).

Expected Behavior

I would expect the PDF to be generated and download in Chrome. Generating a CSV or JSON works OK without issue.

Current Behavior

When clicking the Download > PDF link in Icinga a new tab opens but nothing further happens (no PDF is generated or downloads, tab header just says 'Loading...' but never completes).

On the server side I can see active processes, which never complete and remain open unless killed manually:

ps aux | grep chrome

www-data 5127 0.0 0.0 4276 740 ? S 12:23 0:00 sh -c '/usr/bin/google-chrome-stable' '--headless' '--disable-gpu' '--no-sandbox' '--remote-debugging-port=0'

www-data 5128 0.3 3.5 534056 72384 ? Sl 12:23 0:00 /usr/bin/google-chrome-stable --headless --disable-gpu --no-sandbox --remote-debugging-port=0

www-data 5141 0.1 2.2 400492 45740 ? S 12:23 0:00 /opt/google/chrome/chrome --type=zygote --no-sandbox --headless --headless --enable-crash-reporter

www-data 5155 0.1 3.1 442484 64524 ? Sl 12:23 0:00 /opt/google/chrome/chrome --type=gpu-process --field-trial-handle=409458055811527330,9041078600582879043,131072 --no-sandbox --headless --headless --enable-crash-reporter --gpu-preferences=KAAAAAAAAAAgAAAgAAAAAAAAYAAAAAAAEAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAA --use-gl=swiftshader-webgl --override-use-software-gl-for-tests --service-request-channel-token=10587834501044168516

www-data 5156 0.1 2.4 449796 50908 ? Sl 12:23 0:00 /opt/google/chrome/chrome --type=utility --field-trial-handle=409458055811527330,9041078600582879043,131072 --lang=en-US --service-sandbox-type=network --no-sandbox --use-gl=swiftshader-webgl --headless --enable-crash-reporter --service-request-channel-token=7961077103313469533 --shared-files=v8_context_snapshot_data:100,v8_natives_data:101

www-data 5159 0.1 3.0 623100 62192 ? Sl 12:23 0:00 /opt/google/chrome/chrome --type=renderer --no-sandbox --allow-pre-commit-input --use-gl=swiftshader-webgl --field-trial-handle=409458055811527330,9041078600582879043,131072 --disable-databases --disable-gpu-compositing --lang=en-US --headless --enable-crash-reporter --num-raster-threads=1 --service-request-channel-token=7049592006485112464 --renderer-client-id=4 --shared-files=v8_context_snapshot_data:100,v8_natives_data:101

Your Environment

chlb commented 4 years ago

Hi,

have you already found a workaround?

`2019-11-21T15:02:06+01:00 - ERROR - WebSocket\ConnectionException in /usr/share/icingaweb2/modules/pdfexport/vendor/textalk/websocket/lib/Base.php:202 with message: Empty read; connection dead? Stream state: {"timed_out":false,"blocked":true,"eof":true,"stream_type":"tcp_socket\/ssl","mode":"r+","unread_bytes":0,"seekable":false}

0 /usr/share/icingaweb2/modules/pdfexport/vendor/textalk/websocket/lib/Base.php(92): WebSocket\Base->read(Integer)

1 /usr/share/icingaweb2/modules/pdfexport/vendor/textalk/websocket/lib/Base.php(173): WebSocket\Base->receive()

2 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/HeadlessChrome.php(276): WebSocket\Base->close()

3 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/HeadlessChrome.php(215): Icinga\Module\Pdfexport\HeadlessChrome->printToPDF(String, String, Array)

4 /usr/share/icingaweb2/modules/reactbundle/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php(34): Icinga\Module\Pdfexport\HeadlessChrome->Icinga\Module\Pdfexport{closure}(String)

5 /usr/share/icingaweb2/modules/reactbundle/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php(70): Evenement\EventEmitter->Evenement{closure}(String)

6 /usr/share/icingaweb2/modules/reactbundle/vendor/react/stream/src/ReadableResourceStream.php(146): Evenement\EventEmitter->emit(String, Array)

7 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(244): React\Stream\ReadableResourceStream->handleData(Resource)

8 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(211): React\EventLoop\StreamSelectLoop->waitForStreamActivity(NULL)

9 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/HeadlessChrome.php(225): React\EventLoop\StreamSelectLoop->run()

10 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/ProvidedHook/Pdfexport.php(82): Icinga\Module\Pdfexport\HeadlessChrome->toPdf()

11 /usr/share/icingaweb2/modules/reporting/application/controllers/ReportController.php(151): Icinga\Module\Pdfexport\ProvidedHook\Pdfexport->streamPdfFromHtml(String, String)

12 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Reporting\Controllers\ReportController->downloadAction()

13 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch(String)

14 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))

15 /usr/share/php/Icinga/Application/Web.php(300): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))

16 /usr/share/php/Icinga/Application/webrouter.php(99): Icinga\Application\Web->dispatch()

17 /usr/share/icingaweb2/public/index.php(4): require_once(String)

18 {main}

(END) `

Icingaweb just throws this error message when i try to download a PDF file and i can't determine why. Have you an idea?

speedrz commented 4 years ago

No I haven't found a workaround, the PDF is still not generating for me. I'm not sure if we are seeing the same problem but hopefully someone can offer some help to us both.

If I find a workaround I'll be sure to post it here.

davidlawn commented 4 years ago

I am experiencing this issue also, hopefully a solution can be found.

chlb, you might be facing another issue I found which was that this module expects google-chrome to live in /bin but on my servers it got installed in /usr/bin so I had to fix it with a symlink

ln -s /usr/bin/google-chrome /bin/google-chrome

MarcusCaepio commented 4 years ago

Hey there, just testing the current modules (pdfexport, reporting, idoreports) in a dev container (before taking it in my live cluster). Same Issue here. Clicking on PDF download ends in a loading screen. Using the Container of jordan/icinga2 and installed everything like described

speedrz commented 4 years ago

@lippserd Seems like this is an issue affecting a few people. Any help you can offer?

andrewmiskell commented 4 years ago

I'm seeing this issue as well, trying to generate the report over and over will finally get a working result but it's still a 1 out of 10 chance it'll work properly.

dnsmichi commented 4 years ago

Git master holds a new implementation based on chrome's web api AFAIK. I don't know whether it is already finished, but would be worth a look up until someone is able to look into this issue again (currently everyone is busy with IcingaDB).

MarcusCaepio commented 4 years ago

I already cloned the master branch. Should we try the 0.9.1 releasae?

Obivatelj commented 4 years ago

Same issue here @lippserd, with kind of domino effect against other module:

If i step down from master to 0.9.1 release, PDFExport works as expected in diverse Icingaweb2 Pages, like overview menues, etc... But, still, i met one module which already uses features from PDFExport which are present only in Master, and that is Reporting Module. So i got into situation where i am able to fix only part of my problem, and still can't get those badly needed PDF Reporting exports. I would have to downgrade PDFExport and Reporting Module, and probably db schema which i am not sure that i can.

wolf-in-sheeps-clothing commented 4 years ago

Yeah, same issue here. So I used git checkout in PDFExport to v0.9.1 and in Reporting to v0.9.2 PDF now works in both (Monitoring Modul -> PDF Export and Reporting Modul), but no new features in Reporting Modul like Templates :-(

@lippserd : please have al look and give us a small info. Thanks.

durzo commented 4 years ago

I have it working, please try this patch: https://gist.githubusercontent.com/durzo/e5e7db5fe8f54f4eaf7ad69260400bb9/raw/ef53b73a0b4194fe3bfa640b62070d412d383f91/patch.diff

Firstly, there is a bug in textalk websocket that does not close the socket correctly, there is a dirty hack to fix that.

Secondly, executing google-chrome as the apache user has no $HOME and chrome cannot locate the profile directory so we set some environment variables to set that to /tmp.

Lastly: no matter what I tried, i could never get chrome to terminate by itself, so now set a 2 second timer to end the process loop. This is enough time on my m2.large aws ec2 to process 21 pages of PDF, if you need more just increment it.

wolf-in-sheeps-clothing commented 4 years ago

Hi @durzo I've testet your patch and it works only sometimes. If I use a Template in the Report Module the PDF cannot be loaded:

image

cristorrian commented 4 years ago

Same Issue.

nilmerg commented 4 years ago

@durzo Thanks for investigating this. I've applied it here and combined it with my own fixes. Works flawlessly now. Will push a PR tomorrow.

@wolf-in-sheeps-clothing That's probably due to a bug in one of pdfexport's dependencies. (https://github.com/tecnickcom/TCPDF/pull/123 If you're on PHP 7.4) Will push a PR with the latest version of tcpdf tomorrow as well.

cristorrian commented 4 years ago

Thanks @durzo for the fix, work it properly. :+1:

nilmerg commented 4 years ago

Created the following PRs:

Please test them if you have the chance.

@durzo Please create a PR over at https://github.com/Textalk/websocket-php for your other fixes there.

durzo commented 4 years ago

I don't think my patch to websocket-php is good to apply outside of the use case of Icinga as it may result in closed connections to otherwise healthy (non-chrome) websockets.

I will try to test your PR this week.

speedrz commented 4 years ago

I've tested the 2 PRs in 2 environments I have, the only major difference between the 2 is the version of Icinga Web 2:

In ENV1 the PDF loads as expected. In ENV2 I am getting the same 'Failed to load PDF Document' error as @wolf-in-sheeps-clothing.

image

Both environments run PHP v5.6 and Google Chrome v79

nilmerg commented 4 years ago

Are these the exact same pdfs on both envs you're trying to load? If not are there any size differences? (More data, more pages, more pictures, ...)

speedrz commented 4 years ago

The data is different as the two environments are seperate, but the amount of data is similar (i.e. not much data at all - 1 page only).

I have even created a test report with only 1 host and it fails to load with the same error.

nilmerg commented 4 years ago

I've updated PR #19 now to incorporate also the library upgrades. And added some more changes which also include more (debug) logging.

There's also https://github.com/Icinga/icingaweb2/pull/4044 for Icinga Web 2 which fixes how errors are shown. The dialog as shown above you keep getting should not occur anymore but an error which you can paste here. The patch should cleanly apply to an Icinga Web 2 v2.6.1 installation.

Download: https://patch-diff.githubusercontent.com/raw/Icinga/icingaweb2/pull/4044.patch Apply with: cd /usr/share/php/Icinga && patch -Np1 < 4044.patch

speedrz commented 4 years ago

I've applied the patch @nilmerg but I'm still getting the same error dialog box as before. I've checked the 2 files and can see the changes have been applied, and I've reloaded the Icinga2 service but no difference.

When applying the patch I had to specify the files to change, which in my environment are:

Is that correct?

nilmerg commented 4 years ago

Yes, these are correct. If you still get the dialog please make sure you also applied the latest version of #19 in your module installation and turn on debug logging in Icinga Web 2 and attach the messages about the CDP communication with the browser. Then we should see what's going wrong (or right).

speedrz commented 4 years ago

Ok, with https://github.com/Icinga/icingaweb2-module-pdfexport/pull/19 applied I get the following error message:

Undefined constant 'Icinga\Application\Hook\PdfexportHook'

#0 [internal function]: Icinga\Application\Web->Icinga\Application\{closure}()
#1 {main}

EDIT I'm not sure if I've applied this correctly as that error message now applies over all of the Icinga panels: image

I've had to disable the module.

nilmerg commented 4 years ago

Yes you should probably start from scratch. How did you apply the PR?

nilmerg commented 4 years ago

@speedrz Did you already have the chance to try it again? Would love to know what exactly is going (wrong) on your side to be able to fix it. :smile:

speedrz commented 4 years ago

I've removed the previous installation of pdfexport and just installed #19 which was giving me the following new error when I try to download a PDF:

Failed to start browser: Fontconfig warning: "/etc/fonts/fonts.conf", line 86: unknown element "blank" #0 [internal function]: Icinga\Module\Pdfexport\HeadlessChrome->Icinga\Module\Pdfexport\{closure}(String) #1 /usr/share/icingaweb2/modules/reactbundle/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php(34): call_user_func_array(Object(Closure), Array) #2 [internal function]: Evenement\EventEmitter->Evenement\{closure}(String) #3 /usr/share/icingaweb2/modules/reactbundle/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php(70): call_user_func_array(Object(Closure), Array) #4 /usr/share/icingaweb2/modules/reactbundle/vendor/react/stream/src/ReadableResourceStream.php(146): Evenement\EventEmitter->emit(String, Array) #5 [internal function]: React\Stream\ReadableResourceStream->handleData(Resource) #6 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(244): call_user_func(Array, Resource) #7 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(211): React\EventLoop\StreamSelectLoop->waitForStreamActivity(NULL) #8 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/HeadlessChrome.php(235): React\EventLoop\StreamSelectLoop->run() #9 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/ProvidedHook/Pdfexport.php(78): Icinga\Module\Pdfexport\HeadlessChrome->toPdf() #10 /usr/share/icingaweb2/modules/reporting/application/controllers/ReportController.php(118): Icinga\Module\Pdfexport\ProvidedHook\Pdfexport->streamPdfFromHtml(Object(Icinga\Module\Pdfexport\PrintableHtmlDocument), String) #11 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Reporting\Controllers\ReportController->downloadAction() #12 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch(String) #13 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response)) #14 /usr/share/php/Icinga/Application/Web.php(300): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response)) #15 /usr/share/php/Icinga/Application/webrouter.php(104): Icinga\Application\Web->dispatch() #16 /usr/share/icingaweb2/public/index.php(4): require_once(String) #17 {main}

I've since copied the config from my working Icinga server (which omits the offending blank config) and now it all works!

So I think it is looking good from here @nilmerg

nilmerg commented 4 years ago

It is! Glad it works for you now. But this is only a warning that shouldn't be an issue for the export to run. I probably can enhance this further so that such warnings aren't an issue anymore. Thanks for reporting back!!

MarcusCaepio commented 4 years ago

So is this fixed now?

nilmerg commented 4 years ago

Maybe? This was a combination of several issues/changes which are now hopefully fixed to an extent that PDF generation works 99% of the time. Though most of the changes were related to error reporting, so best would be trying #19 (if you didn't yet) and reporting back if you get errors (and how you possibly fixed them).

MarcusCaepio commented 4 years ago

It is not. A test in the container of jordan/icinga2 with the modules:

shows the same result. PDF is not exporting, just a loading screen

nilmerg commented 4 years ago

Hm, with a stable version of Icinga Web 2? The current master also has a fix to properly show errors for pdf exports (https://github.com/Icinga/icingaweb2/pull/4044, as mentioned above. Please apply that and try it again.

MarcusCaepio commented 4 years ago

Nope, doesn't work. Even not with the changes you mentioned. I'm gonna try it in my production system and will give feedback. My production is another OS with a different php version

nilmerg commented 4 years ago

Then please turn on debug logging (Icinga Web 2) and take a look there. You can also try running the browser in headless mode manually, sometimes it's reporting errors then.

MarcusCaepio commented 4 years ago

Your branch is working on my production system. I thought, it is already merged in master.... My Bad

MarcusCaepio commented 4 years ago

So, another problem occurs. After upgrading reporting module to 0.10.0 to have a nice template, export does not work anymore as soon as I use a template

Empty read; connection dead? Stream state: {"timed_out":false,"blocked":true,"eof":true,"stream_type":"tcp_socket\/ssl","mode":"r+","unread_bytes":0,"seekable":false}

#0 /usr/share/icingaweb2/modules/pdfexport/vendor/textalk/websocket/lib/Base.php(92): WebSocket\Base->read(Integer)
#1 /usr/share/icingaweb2/modules/pdfexport/vendor/textalk/websocket/lib/Base.php(173): WebSocket\Base->receive()
#2 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/HeadlessChrome.php(314): WebSocket\Base->close()
#3 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/HeadlessChrome.php(235): Icinga\Module\Pdfexport\HeadlessChrome->printToPDF(String, String, Array)
#4 /usr/share/icingaweb2/modules/reactbundle/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php(70): Icinga\Module\Pdfexport\HeadlessChrome->Icinga\Module\Pdfexport\{closure}(String)
#5 /usr/share/icingaweb2/modules/reactbundle/vendor/react/stream/src/ReadableResourceStream.php(146): Evenement\EventEmitter->emit(String, Array)
#6 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(244): React\Stream\ReadableResourceStream->handleData(Resource)
#7 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(211): React\EventLoop\StreamSelectLoop->waitForStreamActivity(Integer)
#8 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/HeadlessChrome.php(245): React\EventLoop\StreamSelectLoop->run()
#9 /usr/share/icingaweb2/modules/pdfexport/library/Pdfexport/ProvidedHook/Pdfexport.php(78): Icinga\Module\Pdfexport\HeadlessChrome->toPdf()
#10 /usr/share/icingaweb2/modules/reporting/application/controllers/ReportController.php(118): Icinga\Module\Pdfexport\ProvidedHook\Pdfexport->streamPdfFromHtml(Object(Icinga\Module\Pdfexport\PrintableHtmlDocument), String)
#11 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Reporting\Controllers\ReportController->downloadAction()
#12 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch(String)
#13 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#14 /usr/share/php/Icinga/Application/Web.php(300): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#15 /usr/share/php/Icinga/Application/webrouter.php(99): Icinga\Application\Web->dispatch()
#16 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#17 {main}

Double-checked it. The only working combination is currently: idoreports | 0.9.1 (Downloaded under "Releases") pdfexport | 0.9.1 (git clone branch feature/enhanced-export) reporting | 0.9.2 /Downloaded under "Releases)

reporting 0.10.0 where you can use templates, is not working.

nilmerg commented 4 years ago

The template, does it contain a rather large background image? a logo? If so, then please try #20. :sweat_smile: It includes the fixes from #19 but also slightly changes how a report's content is transferred to the browser. If it's then working for you, I don't see any further reason to wait any longer to get these changes into the master.

MarcusCaepio commented 4 years ago

I tested with a background image and non background image. Both failed. So I don't think it's #20 Anyway, I am doing all these things on the production system right now. There is not much space for testing different branches. I have some kind of reporting running right now and will wait, till everything is running with the newest versions

nilmerg commented 4 years ago

Then please take a look into the (debug) logs again. You should then also see something if you're at least on #19 :wink: I'm interested in logs such as shown here