Problems with character conversion for languages e.g Polish, Hungarian etc.

aiturria-spain commented 1 year ago

Good morning

I have just a question concerning the conversion of characters for different Eastern European languages like Polish, Hungarian, Czech etc.

It seems to be, that UTF-8 doesn't cover all available characters in mentioned languages. In generated PDF the unknown chars are displayed as ??..

Which other charsets are supported by your extension? I took a look inside the PHP file pol.php and there is also used the UTF-8.

A short extract from my Fluid template:

<pdf:document outputDestination="F" outputPath="{pdfOrderFolder}{generatedFileName}" title="Gipo-Services {quickOrderForm.orderType.orderName}" encoding="UTF-8"

What am I doing wrong here? How can I reach to convert this characters?

Thanks for your feedback.

Kind regards

Alberto

maechler commented 1 year ago

@aiturria-spain I doubt that UTF-8 does not cover these characters as it supports all Unicode characters. I think you should be able to leave the pdfviewhelpers encoding settings as they are set by default.

I rather suspect that you do not treat the text as UTF-8 in other places of your application (e.g. database, connection to database, file encoding, etc.). Please make sure that you use UTF-8 throughout.

But if you can provide a minimal code sample of a Fluid template to reproduce the issue, then I will have a further look at it.

aiturria-spain commented 1 year ago

Hi Markus

Thanks a lot for your fast response. I'm not sure where the problem is located, but the pdfviewhelper gets the table header text from language files (.xlf). As far I could investigate with the debugger, the LocalizationUtility::translate() reads the correct content from language files. The text in polish is ok. After the PDF was created, the text in table header is displayed with question marks (??). Therefore I don't know what the problem could be. The charset from database and connection are utf8mb4:

            'tableoptions' => [
                'charset' => 'utf8mb4',
                'collate' => 'utf8mb4_unicode_ci',
            ],
            'charset' => 'utf8mb4',

I give you another extract from my fluid template, maybe it helps:

<pdf:document outputDestination="F" outputPath="{pdfOrderFolder}{generatedFileName}" title="{quickOrderForm.orderType.orderName}" unicode="1" encoding="UTF-8"

{feUser.representation.companyName} {feUser.representation.address} {feUser.representation.zip} {feUser.representation.city} {feUser.representation.country}

                        T {feUser.representation.phoneNr}
                        F {feUser.representation.faxNr}
                        {feUser.representation.mailAddress}
                        {feUser.representation.companyWeb}
                    </pdf:text>
                </f:then>
                <f:else>
                    <pdf:text fontSize="8" alignment="left" color="#8C8C8C" paragraphSpacing="0" posY="15">

                    </pdf:text>
                </f:else>
            </f:if>
        </pdf:column>
    </pdf:multiColumn>
</pdf:header>

<pdf:footer>
    <pdf:graphics.line style="{color: '#AAA'}" padding="{top: 0, bottom: 2}"/>
    <pdf:text color="#AAA">{f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/layout.xlf:downloadportal.pdf.footer.page-from')} {pdf:getPageNumberAlias()} {f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/layout.xlf:downloadportal.pdf.footer.page-of')} {pdf:getTotalNumberOfPagesAlias()}</pdf:text>
</pdf:footer>

<pdf:page orientation="landscape" margin="{top:45}">
    <pdf:text posY="50" padding="{bottom: 10, top: 10}" color="#8C8C8C">
        {quickOrderForm.orderType.orderName} {f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/quickorder.xlf:quickorder.form.mailtext.from')} <f:format.date format="d.m.Y">now</f:format.date>
    </pdf:text>
    <pdf:html styleSheet="EXT:dogrel_downloadportal/Resources/Public/Css/layout.min.css">
        <table width="100%" cellpadding="4" cellspacing="2" id="pdfTable">
            <thead>
                <tr>
                    <th class="pdfHeader" style="width: 4%">#</th>
                    <th class="pdfHeader" style="width: 16%">{f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/locallang.xlf:tx_dogreldownloadportal_domain_model_article.serial_nr')}</th>
                    <th class="pdfHeader" style="width: 22%">{f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/locallang.xlf:tx_dogreldownloadportal_domain_model_article.product_name')}</th>
                    <th class="pdfHeader" style="width: 16%">{f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/locallang.xlf:tx_dogreldownloadportal_domain_model_article.article_nr')}</th>
                    <th class="pdfHeader" style="width: 16%">{f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/locallang.xlf:tx_dogreldownloadportal_domain_model_article.article_description')}</th>
                    <th class="pdfHeader" style="width: 10%">{f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/locallang.xlf:tx_dogreldownloadportal_domain_model_article.order_amount')}</th>
                    <th class="pdfHeader" style="width: 16%">{f:translate(key: 'LLL:EXT:dogrel_downloadportal/Resources/Private/Language/locallang.xlf:tx_dogreldownloadportal_domain_model_article.order_remarks')}</th>
                </tr>
            </thead>
            <tbody>
                {pc:ArticlesForTable(articles: quickOrderForm.articles)}
            </tbody>
        </table>
    </pdf:html>
</pdf:page>

Kind regards

Alberto

maechler commented 1 year ago

@aiturria-spain Thanks for sharing the template and checking the charset used for the database. Could you please also share some of the characters that fail to be displayed? Then I could try to reproduce your issue.

aiturria-spain commented 1 year ago

Hi Markus

I give you the text content (Polish) for the table header that will be filled during execution of the ViewHelper:

Numer seryjny
Produkt / komponent
Numer artykułu
Opis Predmeta
Ilość
Uwagi

Some of this characters will be displayed as a mark question "?" in the PDF document..

Kind regards

Alberto

maechler commented 1 year ago

@aiturria-spain Thanks for the examples! I could now reproduce the issue with the characters. I only had little time to have a look, but it might be that these characters are not supported by the default font. When I change the fontFamily to dejavusans the characters appear. Can you try changing the font and see if that works?

aiturria-spain commented 1 year ago

Hi Markus

Thanks a lot for your help. I'll try to change the fontFamily and see what happens..

Kind regards

Alberto

aiturria-spain commented 1 year ago

Hi Markus

I tried to set the fontFamily in the html-template. I got the following error:

TCPDF ERROR: Could not include font definition file: arial

After checking the documentation I couldn't find out how to set another font. Do I have to set the fontFamily through the TypoScript or directly in the html template e.g. fontFamiliy="arial".

I took a look into the font you mentioned DejaVusans. How do I have to set this font? Because with Arial the chars in Polish aren't displayed..

Kind regards

Alberto

aiturria-spain commented 1 year ago

Hi Markus

I have the following TypoScript configuration:

    fonts {
        subset = 1
        outputPath = typo3temp/pdfviewhelpers/fonts/

        addTTFFont {

            cardo {
                path = EXT:my_downloadportal/Resources/Public/Fonts/Cardo/Cardo-Regular.ttf
            }
            dejavu {
                path =  EXT:my_downloadportal/Resources/Public/Fonts/DejaVu/DejaVuSans.ttf
            }

            computermodern {
                path =  EXT:my_downloadportal/Resources/Public/Fonts/cmunrm.ttf
            }
        }
    }

I guess the configuration is ok. Nevertheless I get errors like:

TCPDF ERROR: Could not include font definition file: computermodern

TCPDF ERROR: Could not include font definition file: dejavu

The ttf font files were uploaded and are reachable for the pdf generation process. Do you have an idea what I'm doing wrong?

Kind regards

Alberto

aiturria-spain commented 1 year ago

Hi Markus

After testing some possibilities I was able to solve the problem. You were right concerning the font dejavusans. This font contains all the needed characters for Polish, Hungarian etc.

My configuration for the fonts a re now the following:

plugin.tx_pdfviewhelpers.settings {

config {
    language = ger

    fonts {
        subset = 1
        outputPath = typo3temp/pdfviewhelpers/fonts/

        addTTFFont {
            cardo-regular {
                path = EXT:my_downloadportal/Resources/Public/Fonts/Cardo/Cardo-Regular.ttf
            }
            dejavusans {
                path =  EXT:my_downloadportal/Resources/Public/Fonts/DejaVu/DejaVuSans.ttf
            }
            cmunrm {
                path =  EXT:my_downloadportal/Resources/Public/Fonts/cmunrm.ttf
            }
        }
    }
}

}

The variable name must correspond with font name.. ;-)

Now it works..

Thank you and have a nice day..

Kind regards

Alberto

maechler commented 1 year ago

@aiturria-spain Great, I am glad to hear that it works for you now!

However you don't need to load dejavusans as a TTF font yourself, because it is already shipped with TCPDF. You can set the default font in TypoScript and still override it for special text types or even individual ViewHelpers, please see the settings inheritance section: https://docs.typo3.org/p/bithost-gmbh/pdfviewhelpers/main/en-us/ConfigurationReference/TypoScriptReference/Index.html#settings-inheritance

So, actually all you would have to do is to set this:

plugin.tx_pdfviewhelpers.settings {
    generalText {
        fontFamily = dejavusans
    }
}

bithost-gmbh / pdfviewhelpers

Problems with character conversion for languages e.g Polish, Hungarian etc. #210