Closed S1m0x81 closed 8 years ago
Hi @S1m0x81,
Sorry to hear you're having this issue. @grendel513 knows the reporting section better than I do. Could you change debug to true in dojo/settings.py and see if there are any errors?
Also what operating system are you running? I'll try to reproduce the issue.
HI @S1m0x81 - Do you have celery running? PDF reports depend on celery because they are asynchronous. See our documentation at http://defectdojo.readthedocs.org/en/latest/features.html#reports
Thanks everyone for your help. I'm running Kali Linux 2.0 which is based on Debian Jessie and I do have celery running. Also I think it's worth to mention that your demo site has the same issue (See screenshot below)
root@Kali:/var/www/html/defectdojo# uname -a Linux Kali 4.0.0-kali1-amd64 #1 SMP Debian 4.0.4-1+kali2 (2015-06-03) x86_64 GNU/Linux root@Kali:/var/www/html/defectdojo# whereis wkhtmltopdf wkhtmltopdf: /usr/bin/wkhtmltopdf /usr/local/bin/wkhtmltopdf /usr/share/man/man1/wkhtmltopdf.1.gz root@Kali:/var/www/html/defectdojo# grep wkhtmltopdf dojo/settings.py
WKHTMLTOPDF_PATH = '/usr/local/bin/wkhtmltopdf'
@S1m0x81 - the demo site is unable to run celery (constraint from pythonanywhere) so it will not be able to generate PDF reports. Can you please attach your celery logs?
We have the latest running in production and do not have any issues with report generation. We are running it on Ubuntu.
The only time I have seen tis type of error happen is when celery is not running or not configured correctly.
@S1m0x81 - additionally, SMTP has to be configured in the settings.py file, as the asynchronous report generation process depends on it. If it cannot send an email (timeout, no server, etc) it will throw and exception in the celery process. This can be changed...and I will look into making it optional.
Thanks, I have postfix running and configured in settings.py. I'm copying celery logs:
OperationalError: (OperationalError) attempt to write a readonly database u'BEGIN IMMEDIATE TRANSACTION' () [2016-04-15 10:28:50,427: INFO/MainProcess] Connected to sqla+sqlite:///dojo.celerydb.sqlite [2016-04-15 10:28:50,444: CRITICAL/MainProcess] Frequent restarts detected: RestartFreqExceeded('5 in 1s',) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 285, in start self._restart_state.step() File "/usr/local/lib/python2.7/dist-packages/billiard/common.py", line 130, in step raise self.RestartFreqExceeded("%r in %rs" % (R, self.maxT)) RestartFreqExceeded: 5 in 1s
Okay so i switched to running celery as root and it worked but there is an error with wkhtmltopdf
The report you requested has failed to generate. Here are the details:
Hello S,
wkhtmltopdf exited with non-zero code -6. error: The switch --header-font-size, is not support using unpatched qt, and will be ignored.The switch --header-spacing, is not support using unpatched qt, and will be ignored.The switch --header-right, is not support using unpatched qt, and will be ignored.The switch --footer-line, is not support using unpatched qt, and will be ignored.The switch --header-left, is not support using unpatched qt, and will be ignored.The switch --outline, is not support using unpatched qt, and will be ignored.The switch --footer-font-size, is not support using unpatched qt, and will be ignored.The switch --footer-center, is not support using unpatched qt, and will be ignored.The switch --outline-depth, is not support using unpatched qt, and will be ignored.The switch --viewport-size, is not support using unpatched qt, and will be ignored.QXcbConnection: Could not connect to display
Enjoy,
Security Engineering
This email was generated by DefectDojo.
@S1m0x81 looks like your sqlite db needs write permissions (http://stackoverflow.com/questions/21054245/attempt-to-write-a-readonly-database-django-w-selinux-error)
@S1m0x81 sorry, did not see your comment. Looks like you need to install wkhtmltopdf on your distro from source or use the provided scripts: https://github.com/wkhtmltopdf/wkhtmltopdf/blob/master/INSTALL.md#linux
does not look like installing from package manager includes all dependencies.
@S1m0x81 Once we confirm what your fix is we can update the docs to make sure this scenario is covered. Thanks for your time in trying to get this working!
True, i will change ownership in production, for now its working with root. However wkhtmltopdf is throwing errors, i'm recompiling from source as someone suggested and see if it will help.
Installing wkhtmltopdf from source is failing
sudo python scripts/build.py jessie-amd64
make[1]: Leaving directory '/home/simo/wkhtmltopdf2/static-build/jessie-amd64/app/src/image' Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn} ^Cfpm --force --prefix /usr/local --category utils -s dir -C dist --url "http://wkhtmltopdf.org/" --maintainer "Ashish Kulkarni kulkarni.ashish@gmail.com" --name "wkhtmltox" --license "LGPLv3" --description "convert HTML to PDF and various image formats using QtWebkit" --version "0.12.4-dev-3250954" --package ../wkhtmltox-0.12.4-dev-3250954_linux-jessie-amd64.deb -t deb --deb-compression xz --provides wkhtmltopdf --conflicts wkhtmltopdf --replaces wkhtmltopdf --depends libc6 --depends libstdc++6 --depends zlib1g --depends libpng12-0 --depends libjpeg62-turbo --depends libssl1.0.0 --depends libfreetype6 --depends libicu52 --depends fontconfig --depends libx11-6 --depends libxext6 --depends libxrender1 --depends libxcb1 --depends xfonts-base --depends xfonts-75dpi . command failed: exit code 2
Any suggestions? Thanks I appreciate your continuous help.
sorry @S1m0x81 i don't have experience installing it on Jessie, sounds like you may have to look for help from the wkhtmltopdf users group: http://wkhtmltopdf.org/support.html
I have added your issue here: https://groups.google.com/forum/#!topic/wkhtmltopdf-general/umPQDnbauMw hopefully we will see a response.
@S1m0x81, it might be easier just to run a Ubuntu VM? We've tested the Ubuntu dependencies very thoroughly, but haven't done much on Kali. Sorry hope this helps.
Thanks guys for your help, i will switch to Ubuntu for now.
Sorry to reopen this but as suggested, I reinstalled on Ubuntu 14.04.4 LTS and still having the same issue, below are celery the logs, thanks!
[2016-06-15 11:24:10,210: INFO/MainProcess] Received task: dojo.tasks.async_custom_pdf_report[10313579-0250-4f7a-bdbb-10142ba96783] [2016-06-15 11:24:10,608: ERROR/MainProcess] Task dojo.tasks.async_custom_pdf_report[10313579-0250-4f7a-bdbb-10142ba96783] raised unexpected: IOError('wkhtmltopdf exited with non-zero code 1. error:\nThe switch --header-font-size, is not support using unpatched qt, and will be ignored.The switch --header-spacing, is not support using unpatched qt, and will be ignored.The switch --header-right, is not support using unpatched qt, and will be ignored.The switch --footer-line, is not support using unpatched qt, and will be ignored.The switch --header-left, is not support using unpatched qt, and will be ignored.The switch --outline, is not support using unpatched qt, and will be ignored.Unknown long argument --enable-javascript\n\nName:\n wkhtmltopdf 0.9.9\n\nSynopsis:\n wkhtmltopdf [OPTIONS]... [More input files]
Name: wkhtmltopdf 0.9.9
Hi @S1m0x81 sorry to hear the issue is back :(. Let me see If I can reproduce and I will get back to you tomorrow.
Hi @S1m0x81, I think the problem is that you're using an old version of wkhtmltopdf. I didn't have that problem when I installed from source. Here is a short script @grendel513 created to do so:
sudo apt-get install -y openssl build-essential xorg libssl-dev wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.10.0_rc2-static-amd64.tar.bz2 tar xvjf wkhtmltopdf-0.10.0_rc2-static-amd64.tar.bz2 sudo chown root:root wkhtmltopdf-amd64 sudo mv wkhtmltopdf-amd64 /usr/bin/wkhtmltopdf
You will probably need to update your settings file to point to /usr/bin/wkhtmltopdf instead of usr/local/bin/wkhtmltopdf. @grendel513 After this though I ran into another error in trace.py related to the email settings.
I went through and commented out all the lines referencing the mailer but still got:
'[2016-06-16 15:24:48,031: ERROR/MainProcess] Pool callback raised exception: ValueError('too many values to unpack',) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 1700, in safe_apply_callback fun(_args, *_kwargs) File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 384, in on_success return self.on_failure(ret_value) File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 443, in on_failure self._log_error(exc_info, send_failed_event=send_failed_event) File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 514, in _log_error task.send_error_email(context, einfo.exception) File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 892, in send_error_email self.ErrorMail(self, **kwargs).send(context, exc) File "/usr/local/lib/python2.7/dist-packages/celery/utils/mail.py", line 190, in send fail_silently=fail_silently) File "/usr/local/lib/python2.7/dist-packages/celery/app/base.py", line 416, in mail_admins to = [adminemail for , admin_email in self.conf.ADMINS] ValueError: too many values to unpack ' Could this be fixed with a simple celery conf? do we have [ in settings.py when we mean ( ?
Thanks for your help! I installed the version you suggested but still not working, i m posting the logs below, hopefully it will help!
[2016-06-17 10:59:48,139: INFO/MainProcess] Received task: dojo.tasks.async_cust om_pdf_report[2cfb5f4b-cf3a-4860-b00d-510d26bda687] [2016-06-17 10:59:48,486: ERROR/MainProcess] Task dojo.tasks.async_custom_pdf_re port[2cfb5f4b-cf3a-4860-b00d-510d26bda687] raised unexpected: IOError('wkhtmltop df exited with non-zero code 1. error:\nUnknown long argument --viewport-size\n\ nName:\n wkhtmltopdf 0.10.0 rc2\n\nSynopsis:\n wkhtmltopdf [GLOBAL OPTION]... [OBJECT]...
Name: wkhtmltopdf 0.10.0 rc2
Synopsis: wkhtmltopdf [GLOBAL OPTION]... [OBJECT]...
Document objects: wkhtmltopdf is able to put several objects into the output file, an object is either a single webpage, a cover webpage or a table of content. The objects are put into the output document in the order they are specified on the command line, options can be specified on a per object basis or in the global options area. Options from the Global Options section can only be placed in the global options area
A page objects puts the content of a singe webpage into the output document.
(page)? <input url/file name> [PAGE OPTION]... Options for the page object can be placed in the global options and the page options areas. The applicable options can be found in the Page Options and Headers And Footer Options sections.
A cover objects puts the content of a singe webpage into the output document, the page does not appear in the table of content, and does not have headers and footers.
cover <input url/file name> [PAGE OPTION]... All options that can be specified for a page object can also be specified for a cover.
A table of content object inserts a table of content into the output document.
toc [TOC OPTION]... All options that can be specified for a page object can also be specified for a toc, further more the options from the TOC Options section can also be applied. The table of content is generated via XSLT which means that it can be styled to look however you want it to look. To get an aide of how to do this you can dump the default xslt document by supplying the --dump-default-toc-xsl, and the outline it works on by supplying --dump-outline, see the Outline Options section.
Description: Converts one or more HTML pages into a PDF document, using wkhtmltopdf patched qt.
Global Options:
--collate Collate when printing multiple copies (default)
--no-collate Do not collate when printing multiple copies
--copies
[2016-06-17 10:59:48,490: ERROR/MainProcess] Pool callback raised exception: Val ueError('too many values to unpack',) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 1700, in safe_apply_callback fun(_args, *_kwargs) File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 384, in on_success return self.on_failure(ret_value) File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 443, in on_failure self._log_error(exc_info, send_failed_event=send_failed_event) File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 514, in _log_error task.send_error_email(context, einfo.exception) File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 892, in send_error_email self.ErrorMail(self, **kwargs).send(context, exc) File "/usr/local/lib/python2.7/dist-packages/celery/utils/mail.py", line 190, in send fail_silently=fail_silently) File "/usr/local/lib/python2.7/dist-packages/celery/app/base.py", line 416, in mail_admins to = [adminemail for , admin_email in self.conf.ADMINS] ValueError: too many values to unpack
hi @S1m0x81 I did more digging on this and I was able to resolve all the errors I was getting, but it is now failing silently. Unfortunately I don't think I can fix this without @grendel513's help and I know lately he has been very busy. As a temporary work around you can do asciidoc to pdf. It's not as awesome, but if you change pdf to ascii doc on the report generator, you can then use a tool like asciidoctor to go to pdf http://asciidoctor.org/docs/convert-asciidoc-to-pdf/
@S1m0x81 so sorry for the delay. I finally fixed it. Give me just a minute and I'll give the details and update the instructions.
1: The old link the in script was deprecated please use: **Note: this need to be run using sudo otherwise you'll get a utime failure.
sudo apt-get install -y openssl build-essential xorg libssl-dev wget http://download.gna.org/wkhtmltopdf/0.12/0.12.3/wkhtmltox-0.12.3_linux-generic-amd64.tar.xz tar xvfJ wkhtmltox-0.12.3_linux-generic-amd64.tar.xz sudo chown root:root wkhtmltox/bin/wkhtmltopdf sudo cp wkhtmltox/bin/wkhtmltopdf /usr/bin/wkhtmltopdf
2: Comment out all the lines related to the emailer in dojo/tasks.py I put """ around the email_requester and then # escapes where the method was used with CTRL+F or /email in vim to find the calls. Are work on getting this fixed in the code. Please let me know if you have any problems or if this resolves the issue for you
Well done @devGregA! Thanks for picking me up ;)
@grendel513 I do what I can ;)
@grendel513 @S1m0x81 submitted PR with code changes. Will update docs in near feature. I have some other high priority stuff I need to get done this weekend or I'd knock it out now. Sorry :(.
@devGregA - from what i see the docs need to be updated to:
1. Mention the email functionality is commented out by default
2. Point to the new bash script within the project rather then the external one.
Anything else? I can take care of docs, just let me know if you need anything else included.
@devGregA - documentation has been updated
@grendel513 can we leave this open and give @S1m0x81 a couple days to validate?
Yes, the pull request comment auto closed it.
On Jun 26, 2016, at 4:59 PM, devGregA notifications@github.com wrote:
@grendel513 can we leave this open and give @S1m0x81 a couple days to validate?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
Thanks guys, I will try in a bit and let you know.
I confirm, reports are being generated with the suggested fixes. Thanks @devGregA and @grendel513 for your help.
Woot! Thanks @S1m0x81. Sorry for all the trouble. We're currently working on stability efforts to hopefully cut down on issues like these.
For those encountering this issue, below is a summary of possible solutions with celery, wkhtmltopdf and firewall rules:
#!/bin/sh
pip install celery sqlalchemy
sudo apt-get install -y openssl build-essential xorg libssl-dev
cd /tmp
wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
tar xvfJ wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
sudo chown root:root wkhtmltox/bin/wkhtmltopdf
sudo cp wkhtmltox/bin/wkhtmltopdf /usr/bin/wkhtmltopdf
# add to settings.py
WKHTMLTOPDF_PATH = '/usr/bin/wkhtmltopdf'
# kick off celery
celery -A dojo worker -l info --concurrency 3 &
celery beat -A dojo -l info &
manage.py runserver 0.0.0.0:80
Note also that your firewall rules need to allow the webserver to access itself on its (public) IP, as the cover is generated at a URL like:
https://yourdojoserver.com/reports/cover?info=c&subtitle=b&title=a
I'm having an issue generating a PDF report, once I click on "Save and Run" the report status remains "Running" for hours and the report is never generated.
Below is the debug log, any help would be appreciated. Thanks
Performing system checks...
System check identified no issues (0 silenced). April 14, 2016 - 11:23:03 Django version 1.8.10, using settings 'dojo.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C.
[14/Apr/2016 11:23:24] "POST /reports/custom HTTP/1.1" 302 0 [14/Apr/2016 11:23:24] "GET /reports HTTP/1.1" 200 27206 [14/Apr/2016 11:23:34] "GET /reports/7/status HTTP/1.1" 200 32 [14/Apr/2016 11:23:44] "GET /reports/7/status HTTP/1.1" 200 32 [14/Apr/2016 11:23:54] "GET /reports/7/status HTTP/1.1" 200 32 [14/Apr/2016 11:24:04] "GET /reports/7/status HTTP/1.1" 200 32 [14/Apr/2016 11:24:14] "GET /reports/7/status HTTP/1.1" 200 32 [14/Apr/2016 11:24:24] "GET /reports/7/status HTTP/1.1" 200 32 [14/Apr/2016 11:24:34] "GET /reports/7/status HTTP/1.1" 200 32 [14/Apr/2016 11:24:44] "GET /reports/7/status HTTP/1.1" 200 32