Basic pdf report with 5 records too heavy

JohnMaverick commented 5 years ago

One simple table and 5 records - pdf 1mb. It is a little bit too much. How to squeeze it whiles generating pdf?

vovpff commented 5 years ago

You are welcome!

bpabiszczak commented 5 years ago

750KB Font [2-3 days to fix] 200KB no compression [1-2 days to fix]

JohnMaverick commented 5 years ago

@bpabiszczak not sure I understood. I just reported issue... in fact bug, as in normal "world" no where no one allowing for generating such large files. Imagine PDF report with 1200 records?

bpabiszczak commented 5 years ago

@JohnMaverick - from the technical side this is not a mistake. That’s the default PDF file size.

Many companies that own libraries [Adobe, mPDF, TCPDF] perform optimization, e.g. when loading fonts, they only use the characters that are in the document. In addition, they use compression so that they can further reduce the file size. It can be reduced to 20-30 KB. You can generate a file with 1200 records and see that there is not much difference between these files because the font takes up most of the space. We have been generating 1MB PDF files for years and it has never bothered anyone. Nothing's for free - performing optimization and compression loads the processor.

Unfortunately for us this is a low priority upgrade.

TonyM15 commented 5 years ago

Unfortunately for us this is a low priority upgrade. Save paper - save bandwidth - both consuming something. It is all about resources/energy. Are Yetiforce green? Just saying....

gkArvindr commented 4 years ago

If you are in Linux platform, which you should be most likely, in Debian run this -$ sudo apt-get install ghostscript and after installing $ps2pdf Large.pdf Small.pdf in my case 3mb file was reduce 183kb

apcloic commented 4 years ago

@gkArvindr Thanks.

Quick and dirty solution :

add a ghostscript wrapper for php
import classes in app/PDF/YetiforcePDF.php

use YetiForcePDF\Document;

// Import classes
use GravityMedia\Ghostscript\Ghostscript;
use Symfony\Component\Process\Process;

then add the following code to function output in app/PDF/YetiforcePDF.php


public function output($fileName = '', $dest = '')
    {
        if (empty($fileName)) {
            $fileName = ($this->getFileName() ? $this->getFileName() : time()) . '.pdf';
        }
        if (!$dest) {
            $dest = 'I';
        }
        $this->writeHTML();
        $output = $this->pdf->render();
        if ('I' !== $dest) {
            return file_put_contents($fileName, $output);
        }
        header('accept-charset: utf-8');
        header('content-type: application/pdf; charset=utf-8');
        $basename = \App\Fields\File::sanitizeUploadFileName($fileName);
        header("content-disposition: attachment; filename=\"{$basename}\"");

        // Quick and dirty hack to reduce PDF size
        // Define input and output files

        $inputFile = tmpfile();
        fwrite($inputFile, $output);
        $pathInputFile = stream_get_meta_data($inputFile)['uri'];

        $outputFile = tmpfile();
        $pathOutputFile = stream_get_meta_data($outputFile)['uri'];
        // Create Ghostscript object
        $ghostscript = new Ghostscript();

        // Create and configure the device
        $device = $ghostscript->createPdfDevice($pathOutputFile);
        $device->setCompatibilityLevel(1.4);

        // Create process
        $process = $device->createProcess($pathInputFile);

        // Run process
        $process->run();

        $output = file_get_contents($pathOutputFile);

        echo $output;
        fclose ($inputFile);
        fclose ($outputFile);

    }

PDF size before : 1Mo PDF size after : 180ko

vovpff commented 4 years ago

@apcloic could you put code to code formatting?

apcloic commented 4 years ago

@vovpff Sorry, I've always had difficulties with code formatting on github. Is it better like this ?

vovpff commented 4 years ago

@apcloic this is looks cool. Thanks

vovpff commented 4 years ago

@apcloic I can't implement this. No experience....

apcloic commented 4 years ago

@vovpff

To implement, I think the only difficulty is to add ghostscript wrapper :

backup everything first
grab a copy of composer.phar latest version https://getcomposer.org/composer-stable.phar
put it at the root of yetiforce web folder
log in to your server and open a shell, go to the root of yetiforce and run the following command


php composer.phar require gravitymedia/ghostscript:dev-master

if all goes well, you can then modify app/PDF/YetiforcePDF.php


<?php

/**
 * Class using YetiForcePDF as a PDF creator.
 *
 * @package   App\Pdf
 *
 * @copyright YetiForce Sp. z o.o
 * @license   YetiForce Public License 3.0 (licenses/LicenseEN.txt or yetiforce.com)
 * @author    Rafal Pospiech <r.pospiech@yetifoce.com>
 * @author    Radosław Skrzypczak <r.skrzypczak@yetiforce.com>
 */

namespace App\Pdf;

use YetiForcePDF\Document;

// Import classes
use GravityMedia\Ghostscript\Ghostscript;
use Symfony\Component\Process\Process;

/**
 * YetiForcePDF class.
 */
class YetiForcePDF extends PDF
{
    const WATERMARK_TYPE_TEXT = 0;
    const WATERMARK_TYPE_IMAGE = 1;

    /**
     * Charset.
     *
     * @var string
     */
    protected $charset = '';

    /**
     * HTML content.
     *
     * @var string
     */
    public $html = '';

    /**
     * Page format.
     *
     * @var string
     */
    public $format = '';
    /**
     * @var string
     */
    public $orientation = 'P';
    /**
     * @var string
     */
    protected $header = '';

    /**
     * @var string
     */
    protected $footer = '';

    /**
     * @var string
     */
    protected $footerYetiForce = '';

    /**
     * @var string
     */
    protected $watermark = '';
    /**
     * @var int
     */
    protected $headerMargin = 10;
    /**
     * @var int
     */
    protected $footerMargin = 10;

    /**
     * Default margins.
     *
     * @var array
     */
    public $defaultMargins = [
        'left' => 30,
        'right' => 30,
        'top' => 40,
        'bottom' => 40,
        'header' => 10,
        'footer' => 10
    ];

    /**
     * Default font.
     *
     * @var string
     */
    protected $defaultFontFamily = 'DejaVu Sans';

    /**
     * Default font size.
     *
     * @var int
     */
    protected $defaultFontSize = 10;

    /**
     * @var \Vtiger_Module_Model
     */
    protected $moduleModel;

    /**
     * Additional params.
     *
     * @var array
     */
    protected $params = [];

    /**
     * Returns pdf library object.
     */
    public function getPdf()
    {
        return $this->pdf;
    }

    /**
     * Constructor.
     */
    public function __construct()
    {
        $this->setInputCharset(\App\Config::main('default_charset') ?? 'UTF-8');
        $this->pdf = (new Document())->init();
        if (!\App\Config::component('Branding', 'is_customer_branding_active')) {
            $this->footer = $this->footerYetiForce = "<table style=\"font-family:'DejaVu Sans';font-size:6px;width:100%; margin: 0;\">
                <tbody>
                    <tr>
                        <td style=\"width:50%;\"></td>
                    </tr>
                </tbody>
            </table>";
        }
    }

    /**
     * {@inheritdoc}
     */
    public function getInputCharset()
    {
        return $this->charset;
    }

    /**
     * {@inheritdoc}
     */
    public function setInputCharset(string $charset)
    {
        $this->charset = $charset;
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setTopMargin(float $margin)
    {
        $this->pdf->setDefaultTopMargin($margin);
        $this->defaultMargins['top'] = $margin;
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setBottomMargin(float $margin)
    {
        $this->pdf->setDefaultBottomMargin((float) $margin);
        $this->defaultMargins['bottom'] = $margin;
        return $this;
    }

    /**
     * Set left margin.
     *
     * @param float $margin
     */
    public function setLeftMargin(float $margin)
    {
        $this->pdf->setDefaultLeftMargin((float) $margin);
        $this->defaultMargins['left'] = $margin;
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setRightMargin(float $margin)
    {
        $this->pdf->setDefaultRightMargin((float) $margin);
        $this->defaultMargins['right'] = $margin;
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setHeaderMargin(float $margin)
    {
        $this->headerMargin = $margin;
        return $this;
    }

    /**
     * Gets header margin.
     */
    public function getHeaderMargin()
    {
        return $this->headerMargin;
    }

    /**
     * {@inheritdoc}
     */
    public function setFooterMargin(float $margin)
    {
        $this->footerMargin = $margin;
        return $this;
    }

    /**
     * Gets footer margin.
     */
    public function getFooterMargin()
    {
        return $this->footerMargin;
    }

    /**
     * {@inheritdoc}
     */
    public function setMargins(array $margins)
    {
        $this->setTopMargin($margins['top'] ?? $this->defaultMargins['top']);
        $this->setBottomMargin($margins['bottom'] ?? $this->defaultMargins['bottom']);
        $this->setLeftMargin($margins['left'] ?? $this->defaultMargins['left']);
        $this->setRightMargin($margins['right'] ?? $this->defaultMargins['right']);
        $this->setHeaderMargin($margins['header'] ?? $this->defaultMargins['header']);
        $this->setFooterMargin($margins['footer'] ?? $this->defaultMargins['footer']);
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setPageSize(string $format, string $orientation = null)
    {
        $this->pdf->setDefaultFormat($format);
        if ($orientation) {
            $this->pdf->setDefaultOrientation($orientation);
        }
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setFont(string $family, int $size)
    {
        $this->defaultFontFamily = $family;
        $this->defaultFontSize = $size;
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function parseParams(array $params)
    {
        foreach ($params as $param => $value) {
            switch ($param) {
                case 'margin-top':
                    if (is_numeric($value)) {
                        $this->setTopMargin($value);
                    } else {
                        $this->setTopMargin($this->defaultMargins['top']);
                    }
                    break;
                case 'margin-bottom':
                    if (is_numeric($value)) {
                        $this->setBottomMargin($value);
                    } else {
                        $this->setBottomMargin($this->defaultMargins['bottom']);
                    }
                    break;
                case 'margin-left':
                    if (is_numeric($value)) {
                        $this->setLeftMargin($value);
                    } else {
                        $this->setLeftMargin($this->defaultMargins['left']);
                    }
                    break;
                case 'margin-right':
                    if (is_numeric($value)) {
                        $this->setRightMargin($value);
                    } else {
                        $this->setRightMargin($this->defaultMargins['right']);
                    }
                    break;
                case 'header_height':
                    if (is_numeric($value)) {
                        $this->setHeaderMargin($value);
                    } else {
                        $this->setHeaderMargin($this->defaultMargins['header']);
                    }
                    break;
                case 'footer_height':
                    if (is_numeric($value)) {
                        $this->setFooterMargin($value);
                    } else {
                        $this->setFooterMargin($this->defaultMargins['footer']);
                    }
                    break;
                case 'title':
                    $this->setTitle($value);
                    break;
                case 'author':
                    $this->setAuthor($value);
                    break;
                case 'creator':
                    $this->setCreator($value);
                    break;
                case 'subject':
                    $this->setSubject($value);
                    break;
                case 'keywords':
                    $this->setKeywords(explode(',', $value));
                    break;
                default:
                    break;
            }
        }
        return $this;
    }

    // meta attributes

    /**
     * {@inheritdoc}
     */
    public function setTitle(string $title)
    {
        $this->pdf->getMeta()->setTitle($title);
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setAuthor(string $author)
    {
        $this->pdf->getMeta()->setAuthor($author);
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setCreator(string $creator)
    {
        $this->pdf->getMeta()->setCreator($creator);
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setSubject(string $subject)
    {
        $this->pdf->getMeta()->setSubject($subject);
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setKeywords(array $keywords)
    {
        $this->pdf->getMeta()->setKeywords($keywords);
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setHeader(string $headerHtml)
    {
        $this->header = trim($headerHtml);
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function setFooter(string $footerHtml)
    {
        $this->footer = trim($footerHtml) . $this->footerYetiForce;
        return $this;
    }

    /**
     * Wrap header content.
     *
     * @param string $headerContent
     *
     * @return string
     */
    public function wrapHeaderContent(string $headerContent)
    {
        $style = "padding-top:{$this->headerMargin}px; padding-left:{$this->defaultMargins['left']}px; padding-right:{$this->defaultMargins['right']}px";
        return '<div data-header style="' . $style . '">' . $headerContent . '</div>';
    }

    /**
     * Wrap footer content.
     *
     * @param string $footerContent
     *
     * @return string
     */
    public function wrapFooterContent(string $footerContent)
    {
        $style = "padding-bottom:{$this->footerMargin}px; padding-left:{$this->defaultMargins['left']}px; padding-right:{$this->defaultMargins['right']}px";
        return '<div data-footer style="' . $style . '">' . $footerContent . '</div>';
    }

    /**
     * Wrap watermark.
     *
     * @param string $watermarkContent
     *
     * @return string
     */
    public function wrapWatermark(string $watermarkContent)
    {
        return '<div data-watermark style="text-align:center">' . $watermarkContent . '</div>';
    }

    /**
     * Load custom fonts.
     *
     * @return $this
     */
    private function loadCustomFonts()
    {
        $fontsDir = 'layouts' . \DIRECTORY_SEPARATOR . 'resources' . \DIRECTORY_SEPARATOR . 'fonts' . \DIRECTORY_SEPARATOR;
        $resolvedDir = \Vtiger_Loader::resolveNameToPath('~' . $fontsDir, 'css');
        $customFonts = \App\Json::read($resolvedDir . 'fonts.json');
        foreach ($customFonts as &$font) {
            $font['file'] = $resolvedDir . $font['file'];
        }
        \YetiForcePDF\Document::addFonts($customFonts);
        return $this;
    }

    /**
     * Write html.
     *
     * @return $this
     */
    public function writeHTML()
    {
        $this->loadCustomFonts();
        $this->pdf->loadHtml($this->getHtml(), $this->charset);
        return $this;
    }

    /**
     * Gets full html.
     *
     * @return string
     */
    public function getHtml()
    {
        $html = $this->watermark ? $this->wrapWatermark($this->watermark) : '';
        $html .= $this->header ? $this->wrapHeaderContent($this->header) : '';
        $html .= $this->footer ? $this->wrapFooterContent($this->footer) : '';
        $html .= $this->html;
        return $html;
    }

    /**
     * Get template watermark.
     *
     * @param \Vtiger_PDF_Model $templateModel
     *
     * @return string
     */
    public function getTemplateWatermark(\Vtiger_PDF_Model $templateModel)
    {
        $watermark = '';
        if (self::WATERMARK_TYPE_IMAGE === $templateModel->get('watermark_type') && '' !== trim($templateModel->get('watermark_image'))) {
            if ($templateModel->get('watermark_image')) {
                $watermark = '<img src="' . $templateModel->get('watermark_image') . '" style="opacity:0.1;">';
            }
        } elseif (self::WATERMARK_TYPE_TEXT === $templateModel->get('watermark_type') && '' !== trim($templateModel->get('watermark_text'))) {
            $watermark = '<div style="opacity:0.1;display:inline-block;">' . $templateModel->get('watermark_text') . '</div>';
        }
        return $watermark;
    }

    /**
     * {@inheritdoc}
     */
    public function setWatermark(string $watermark)
    {
        $this->watermark = $watermark;
        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function loadHtml(string $html)
    {
        $this->html = $html;
        return $this;
    }

    /**
     * Output content to PDF.
     *
     * @param string $fileName
     * @param string $dest
     */
    public function output($fileName = '', $dest = '')
    {
        if (empty($fileName)) {
            $fileName = ($this->getFileName() ? $this->getFileName() : time()) . '.pdf';
        }
        if (!$dest) {
            $dest = 'I';
        }
        $this->writeHTML();
        $output = $this->pdf->render();
        if ('I' !== $dest) {
            return file_put_contents($fileName, $output);
        }
        header('accept-charset: utf-8');
        header('content-type: application/pdf; charset=utf-8');
        $basename = \App\Fields\File::sanitizeUploadFileName($fileName);
        header("content-disposition: attachment; filename=\"{$basename}\"");

        // Define input and output files

        $inputFile = tmpfile();
        fwrite($inputFile, $output);
        $pathInputFile = stream_get_meta_data($inputFile)['uri'];

        $outputFile = tmpfile();
        $pathOutputFile = stream_get_meta_data($outputFile)['uri'];

        // Create Ghostscript object
        $ghostscript = new Ghostscript();

        // Create and configure the device
        $device = $ghostscript->createPdfDevice($pathOutputFile);
        $device->setCompatibilityLevel(1.4);

        // Create process
        $process = $device->createProcess($pathInputFile);

        // Run process
        $process->run();

        $output = file_get_contents($pathOutputFile);

        echo $output;

        fclose ($inputFile);
        fclose ($outputFile);

    }

    /**
     * Export record to PDF file.
     *
     * @param int    $recordId   - record ID
     * @param int    $templateId - id of pdf template
     * @param string $filePath   - path name for saving pdf file
     * @param string $saveFlag   - save option flag
     */
    public function export($recordId, $templateId, $filePath = '', $saveFlag = '')
    {
        \Vtiger_PDF_Model::exportToPdf($recordId, $templateId, $filePath, $saveFlag);
    }
}

Hope it helps.

Regards,

vovpff commented 4 years ago

@apcloic wow. This works

> App\Installer\Composer::install
==================================================
Cleaned files: 16
--------------------------------------------------
Copy to public_html:
yetiforce/csrf-magic[move]: 0
maximebf/debugbar[move]: 0
yetiforce/yetiforcepdf/lib/Fonts[copy]: 0
ckeditor/ckeditor[move]: 0
--------------------------------------------------
Copy custom directories:
public_html/libraries/ckeditor-image-to-base  >>>  public_html/vendor/ckeditor/ckeditor/plugins/ckeditor-image-to-base | Files: 7
--------------------------------------------------
++++++++++++++++++++++++++++++++++++++++++++++++++
The problem occured when generating app_data/libraries.json file!!!
It is required to run yarn first and then the composer.
Example: https://github.com/YetiForceCompany/YetiForceCRM/blob/developer/tests/setup/dependency.sh
++++++++++++++++++++++++++++++++++++++++++++++++++
==================================================

Is it well?

apcloic commented 4 years ago

@vovpff Yes, I had the same warning about Yarn. I'm not familiar with yarn (which is a package manager), maybe YF could tell us if it's important ? If you have this new package in vendor/gravitymedia/ghostscript, it should work !

vovpff commented 4 years ago

@apcloic yes. This works. Now pdf size 6 times smaller. Do you have ideas how to roll back chandes if something going wrong?

apcloic commented 4 years ago

@vovpff To roll back, afaik you just have to :

restore original version of app/PDF/YetiforcePDF.php
use the following command to remove ghostscript package. php composer.phar remove gravitymedia/ghostscript

apcloic commented 3 years ago

FYI, this PDF mod no longer works with YF current version (6.1.23) and it breaks PDF generation. You could simply restore original version of app/PDF/YetiforcePDF.php Too bad but I will try to debug later.

apcloic commented 3 years ago

Ok, I've just re-add ghostscript dependencies which was removed by the upgrade process and now it's working again. php composer.phar require gravitymedia/ghostscript:dev-master Unfortunately I don't know how to make symfony external package upgrade-safe Regards

mariuszkrzaczkowski commented 3 years ago

To reduce the size of PDF, you can change the default font, add to the library the ability to change using a variable:

\YetiForcePDF\Objects\Font::$defaultFontFamily = 'Roboto';

Here is the result for the invoice

The default font in PDF is DejaVu Sans, but it is important that there is not a single instance of using the font, it will be added anyway

apcloic commented 3 years ago

@mariuszkrzaczkowski

After applying the required changes to add the ability to change default font, DejaVu Sans was still added because of the footer defined in app/Pdf/YetiforcePDF.php

So I've made the following extra modifications :

protected $defaultFontFamily = 'Roboto';

and

if (!\App\YetiForce\Shop::check('YetiForceDisableBranding')) { $this->footer = $this->footerYetiForce = "<table style=\"font-family:'".$this->defaultFontFamily."';font-size:6px;width:100%; margin: 0;\"><tbody><tr><td style=\"width:50%\">Powered by YetiForce</td></tr></tbody></table>"; }

File size with DejaVu Sans : 448ko File size with Roboto : 105ko

This is great improvement, thanks a lot.

YetiForceCompany / YetiForceCRM

Basic pdf report with 5 records too heavy #11756