search

peiTeX / qrbill

LaTeX package to create qrbills based on the Swiss standard
13 stars 3 forks source link

Invalid encoding validation error when using umlauts in city #10

Closed The-Compiler closed 2 years ago

The-Compiler commented 2 years ago

With a trivial example such as this:

\documentclass{standalone}
\usepackage[ngerman]{babel}

\usepackage{xcolor}
\usepackage{qrbill}

\begin{document}

\QRbill[
creditor*={foobar LLC\\
    Postfach \\
    404\\
    2342\\
    Zürich\\
    CH},
Account=CH1280808005649899718,
% insert additional data here
]

\end{document}

(note the "Zürich" rather than "Zurich" as in the docs), the following document gets generated: ex.pdf

Uploading that to the Postfinance validator shows the following error:

Content 'Z.rich' includes at least one invalid character (replaced with a dot in this display). Either the character encoding is not UTF-8 or a character outside the permitted "Latin Character Set" was used (see Implementation Guideline QRR CustomerCreditTransfer).

Trying to enter (roughly) the same data into the QR generator results in QR_Rechnung_20220806.pdf which does validate correctly.

Looking at the two QR codes with Binary Eye reveals that "Zürich" is encoded as Latin1 UTF-8 in both cases (5A C3 BC 72 69 63 68), so I really don't understand what the difference is... It doesn't seem solely the Postfinance validator though - swiss-qr-invoice.org also displays it as "Z�rich"...

TeXhackse commented 2 years ago

Did you use lualatex for the compilation?

The-Compiler commented 2 years ago

Indeed - built with lualatex -pdf ex.tex, with LuaHBTeX, Version 1.15.0 (TeX Live 2022/Arch Linux).

TeXhackse commented 2 years ago

I cannot reproduce that. Though the archlinux package (as all not vanilla TeX Live packages) might cause it.

Can you add a \listfiles at the beginning of the document and give a log?

The-Compiler commented 2 years ago 
This is LuaHBTeX, Version 1.15.0 (TeX Live 2022/Arch Linux) 
 restricted system commands enabled.
(./ex.tex
LaTeX2e <2021-11-15> patch level 1
 L3 programming layer <2022-04-10>
(/usr/share/texmf-dist/tex/latex/standalone/standalone.cls
Document Class: standalone 2018/03/26 v1.3a Class to compile TeX sub-files stan
dalone
(/usr/share/texmf-dist/tex/latex/tools/shellesc.sty)
(/usr/share/texmf-dist/tex/generic/iftex/ifluatex.sty
(/usr/share/texmf-dist/tex/generic/iftex/iftex.sty))
(/usr/share/texmf-dist/tex/latex/xkeyval/xkeyval.sty
(/usr/share/texmf-dist/tex/generic/xkeyval/xkeyval.tex
(/usr/share/texmf-dist/tex/generic/xkeyval/xkvutils.tex
(/usr/share/texmf-dist/tex/generic/xkeyval/keyval.tex))))
(/usr/share/texmf-dist/tex/latex/standalone/standalone.cfg)
(/usr/share/texmf-dist/tex/latex/base/article.cls
Document Class: article 2021/10/04 v1.4n Standard LaTeX document class
(/usr/share/texmf-dist/tex/latex/base/size10.clo))
(/usr/share/texmf-dist/tex/generic/luatex85/luatex85.sty)
(/usr/share/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty
(/usr/share/texmf-dist/tex/generic/infwarerr/infwarerr.sty)
(/usr/share/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty)))
(/usr/share/texmf-dist/tex/generic/babel/babel.sty
(/usr/share/texmf-dist/tex/generic/babel/luababel.def)
(/usr/share/texmf-dist/tex/generic/babel/luababel.def)
(/usr/share/texmf-dist/tex/generic/babel-german/ngerman.ldf
(/usr/share/texmf-dist/tex/generic/babel-german/ngermanb.ldf
(/usr/share/texmf-dist/tex/generic/hyph-utf8/loadhyph/loadhyph-de-1996.tex
UTF-8 German hyphenation patterns (reformed orthography)
(/usr/share/texmf-dist/tex/generic/hyph-utf8/patterns/tex/hyph-de-1996.tex
German Hyphenation Patterns (Reformed Orthography, 2006) `dehyphn-x' 2021-02-26
 (WL)))))) (/usr/share/texmf-dist/tex/latex/xcolor/xcolor.sty
(/usr/share/texmf-dist/tex/latex/graphics-cfg/color.cfg)
(/usr/share/texmf-dist/tex/latex/graphics-def/luatex.def))
(/usr/share/texmf-dist/tex/latex/qrbill/qrbill.sty
(/usr/share/texmf-dist/tex/latex/l3kernel/expl3.sty
(/usr/share/texmf-dist/tex/latex/l3backend/l3backend-luatex.def))
(/usr/share/texmf-dist/tex/latex/l3packages/l3keys2e/l3keys2e.sty)
(/usr/share/texmf-dist/tex/latex/fontspec/fontspec.sty
(/usr/share/texmf-dist/tex/latex/l3packages/xparse/xparse.sty)
(/usr/share/texmf-dist/tex/latex/fontspec/fontspec-luatex.sty
(/usr/share/texmf-dist/tex/latex/base/fontenc.sty)
(/usr/share/texmf-dist/tex/latex/fontspec/fontspec.cfg)))
(/usr/share/texmf-dist/tex/latex/qrcode/qrcode.sty)
(/usr/share/texmf-dist/tex/latex/koma-script/scrbase.sty
(/usr/share/texmf-dist/tex/latex/koma-script/scrlfile.sty
(/usr/share/texmf-dist/tex/latex/koma-script/scrlfile-hook.sty
(/usr/share/texmf-dist/tex/latex/koma-script/scrlogo.sty))))
(/usr/share/texmf-dist/tex/latex/graphics/graphicx.sty
(/usr/share/texmf-dist/tex/latex/graphics/graphics.sty
(/usr/share/texmf-dist/tex/latex/graphics/trig.sty)
(/usr/share/texmf-dist/tex/latex/graphics-cfg/graphics.cfg)))
(/usr/share/texmf-dist/tex/latex/numprint/numprint.sty
(/usr/share/texmf-dist/tex/latex/tools/array.sty)
No configuration file `numprint.cfg' found.)
(/usr/share/texmf-dist/tex/latex/qrbill/swiss.qrbill-cfg.tex)) (./ex.aux
<Reading QR code for "SPC
0200
1
CH1280808005649899718
S
foobar LLC
Postfach
404
2342
Zürich
CH

CHF

NON

EPD
//S1

" at level 7-M from aux file.>
) (/usr/share/texmf-dist/tex/latex/base/ts1cmr.fd)
(/usr/share/texmf-dist/tex/context/base/mkii/supp-pdf.mkii
[Loading MPS to PDF converter (version 2006.09.02).]
) (/usr/share/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty
(/usr/share/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg))
Underfull \vbox (badness 10000) detected at line 19

Underfull \vbox (badness 10000) detected at line 19

LaTeX Font Warning: Font shape `OML/cmm/m/it' in size <4.21571> not available
(Font)              size <5> substituted on input line 19.

LaTeX Font Warning: Font shape `OML/cmm/m/it' in size <3.01125> not available
(Font)              size <5> substituted on input line 19.

LaTeX Font Warning: Font shape `OMS/cmsy/m/n' in size <4.21571> not available
(Font)              size <5> substituted on input line 19.

LaTeX Font Warning: Font shape `OMS/cmsy/m/n' in size <3.01125> not available
(Font)              size <5> substituted on input line 19.

LaTeX Font Warning: Font shape `OT1/cmr/m/n' in size <4.21571> not available
(Font)              size <5> substituted on input line 19.

LaTeX Font Warning: Font shape `OT1/cmr/m/n' in size <3.01125> not available
(Font)              size <5> substituted on input line 19.

Underfull \vbox (badness 10000) detected at line 19

Underfull \vbox (badness 10000) detected at line 19

<QR code requested for "SPC
0200
1
CH1280808005649899718
S
foobar LLC
Postfach
404
2342
Zürich
CH

CHF

NON

EPD
//S1

" in version 0-M.>

<Copying the QR code for "SPC
0200
1
CH1280808005649899718
S
foobar LLC
Postfach
404
2342
Zürich
CH

CHF

NON

EPD
//S1

" in version 7-M as previously calculated.>

Underfull \vbox (badness 10000) detected at line 19

Underfull \vbox (badness 10000) detected at line 19
[1{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map}] (./ex.aux
<Reading QR code for "SPC
0200
1
CH1280808005649899718
S
foobar LLC
Postfach
404
2342
Zürich
CH

CHF

NON

EPD
//S1

" at level 7-M from aux file.>
)

 *File List*
standalone.cls    2018/03/26 v1.3a Class to compile TeX sub-files standalone
shellesc.sty    2019/11/08 v1.0c unified shell escape interface for LaTeX
ifluatex.sty    2019/10/25 v1.5 ifluatex legacy package. Use iftex instead.
   iftex.sty    2022/02/03 v1.0f TeX engine tests
 xkeyval.sty    2020/11/20 v2.8 package option processing (HA)
 xkeyval.tex    2014/12/03 v2.7a key=value parser (HA)
standalone.cfg    2018/03/26 v1.3a Default configuration file for 'standalone' 
class
 article.cls    2021/10/04 v1.4n Standard LaTeX document class
  size10.clo    2021/10/04 v1.4n Standard LaTeX file (size option)
luatex85.sty    2016/06/15 v1.4 pdftex aliases for luatex
pdftexcmds.sty    2020-06-27 v0.33 Utility functions of pdfTeX for LuaTeX (HO)
infwarerr.sty    2019/12/03 v1.5 Providing info/warning/error messages (HO)
 ltxcmds.sty    2020-05-10 v1.25 LaTeX kernel commands for general use (HO)
   babel.sty    2022/02/26 3.73 The Babel package
 ngerman.ldf    2021/02/27 v2.13 German support for babel (post-1996 orthograph
y)
  xcolor.sty    2021/10/31 v2.13 LaTeX color extensions (UK)
   color.cfg    2016/01/02 v1.6 sample color configuration
  luatex.def    2021/06/01 v1.2c Graphics/color driver for luatex
  qrbill.sty    2021/05/10 v1.03  Template for QR-bills based on the Swiss Paym
ent Standards 
   expl3.sty    2022-04-10 L3 programming layer (loader) 
l3backend-luatex.def    2022-04-14 L3 backend support: PDF output (LuaTeX)
l3keys2e.sty    2022-01-12 LaTeX2e option processing using LaTeX3 keys
fontspec.sty    2022/01/15 v2.8a Font selection for XeLaTeX and LuaLaTeX
  xparse.sty    2022-01-12 L3 Experimental document command parser
fontspec-luatex.sty    2022/01/15 v2.8a Font selection for XeLaTeX and LuaLaTeX

 fontenc.sty
fontspec.cfg
  qrcode.sty    2015/01/08 v1.51 QR code generation
 scrbase.sty    2021/11/13 v3.35 KOMA-Script package (KOMA-Script-independent b
asics and keyval usage)
scrlfile.sty    2021/11/13 v3.35 KOMA-Script package (file load hooks)
scrlfile-hook.sty    2021/11/13 v3.35 KOMA-Script package (using LaTeX hooks)
 scrlogo.sty    2021/11/13 v3.35 KOMA-Script package (logo)
graphicx.sty    2021/09/16 v1.2d Enhanced LaTeX Graphics (DPC,SPQR)
graphics.sty    2021/03/04 v1.4d Standard LaTeX Graphics (DPC,SPQR)
    trig.sty    2021/08/11 v1.11 sin cos tan (DPC)
graphics.cfg    2016/06/04 v1.11 sample graphics configuration
numprint.sty    2012/08/20 v1.39 Print numbers (HH)
   array.sty    2021/10/04 v2.5f Tabular extension package (FMi)
   swiss.qrbill-cfg
  ts1cmr.fd    2019/12/16 v2.5j Standard LaTeX font definitions
supp-pdf.mkii
epstopdf-base.sty    2020-01-24 v2.11 Base part for package epstopdf
epstopdf-sys.cfg    2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Live
 ***********

LaTeX Font Warning: Size substitutions with differences
(Font)              up to 1.98875pt have occurred.

)
(see the transcript file for additional information)
 60133 words of node memory still in use:
   2252 hlist, 21 vlist, 2068 rule, 41 disc, 68 local_par, 16 math, 429 glue, 4
 kern, 80 penalty, 352 glyph, 4675 attribute, 50 glue_spec, 2339 attribute_list
, 2 write, 4 pdf_colorstack nodes
   avail lists: 2:45,3:170,4:18,5:41,6:138,7:1606,8:1,9:8692,11:82
</usr/share/fonts/liberation/LiberationSans-Bold.ttf></usr/share/fonts/liberati
on/LiberationSans-Regular.ttf>
Output written on ex.pdf (1 page, 27961 bytes).
Transcript written on ex.log.
TeXhackse commented 2 years ago

Thanks - I will have a closer look at that and get back to you.

TeXhackse commented 2 years ago

I found the reason. Locally I was using the lua converter for the qrcode, which is a preparation for #1 When i use the qrcode package I get the same result. This is caused by the qrcode package only supporting the binary format. But I will provide an update to use another qrcode mode. The standard encoding for the binary mode is ISO which ofc doesn't match the requirement of being a utf-8 string.

The-Compiler commented 2 years ago

That's great to hear, thanks for having a closer look! I'm assuming that work isn't pushed to a branch or anything so far? Would be happy to test.

TeXhackse commented 2 years ago

There are some small todos, but I try to finish both during the weekend.

TeXhackse commented 2 years ago

ups - referenced the wrong issue. Anyay …

I contacted the author of the qrcode package but sadly didn't receive any answer.

This now means that I will add an own implementation for the qrcode mechanism and publish it separatly. While waiting for this the lua mechanism can be considered stable the tests didn't show any more issues.

Tekki commented 2 years ago

The question is how big a problem this is. My company and several of our customers have printed qr invoices using the qrcode package since 2 years, ignoring such failing tests, and there was never a "real world issue" because of that (even the infamous PostFinance app accepts them).

It would work if you double encode the strings. With this:

  creditor*={...
    8866\\
    Ziegelbrücke\\
    CH},

the qr code passes the tests at SIX, with this:

  creditor*={...
    8866\\
    Ziegelbrücke\\
    CH},

it doesn't.

TeXhackse commented 2 years ago

Its already working with the luaqrcode backend. - see #1 for this. The issue is, that the qrcode package is reencoding the string into ISO. So you get a valid QRcode out of it but it's ISO-encoded.

Hmmm you option of reencoding the string in advance is a possible workaround. I will add a code snippet here to do so until I finish the work on the qrcode package reimplementation.

TeXhackse commented 2 years ago

So that would be the workaround using the reencoding approach of @Tekki. I still don't like it, because the qrcode itself is interpreted as iso for the most readers. So handle with care. 

\ExplSyntaxOn
\cs_set:Nn \qrcode_setup_QRcode: {
    \str_clear:N \l_qrbill_data_str
    \bool_set_false:N \l_tmpa_bool
    \seq_map_inline:Nn \g__qrbill_qrscheme_seq {
        \bool_if:NTF \l_tmpa_bool
        {\str_put_right:Nn\l_qrbill_data_str  {^^J}}
        {\bool_set_true:N \l_tmpa_bool}
        \exp_args:NNv \str_set_convert:Nnnn \l_tmpa_str {l_qrbill_data_##1_tl} {iso88591} {utf8} 
        \str_put_right:NV \l_qrbill_data_str \l_tmpa_str%{l_qrbill_data_##1_tl}
    }
    %necessary to preserve spaces!
    \str_replace_all:Nnn \l_qrbill_data_str  {~ } {\ }
    \tl_set_rescan:Nno \l_qrbill_data_str {\ExplSyntaxOff} {\l_qrbill_data_str}
}
\ExplSyntaxOff

Simply place this after loading the current release or use the lua aproach.

Tekki commented 2 years ago

So that would be the workaround using the reencoding approach of @Tekki. I still don't like it...

Thanks. The QR Code with Umlauts is valid now. But I don't like such workarounds either.

The-Compiler commented 2 years ago

This is way above my TeX skills I suppose, but the workaround indeed does work. I'm afraid I haven't gotten around to testing the Lua QR approach yet.

TeXhackse commented 2 years ago

I just was informed for the reason there are so many new users around here - I will patch the workaround tomorrow and release another version before 2.00. The intermediate version will use the double encoding approach. It's valid with the qr standard anyway because the readers expect iso encoded qrcodes. I will also add an option to make replacement mappings for umlauts so one can choose which variant should be used.

TeXhackse commented 2 years ago

moved hte reimplementation to a new issue because it's at least working stable with the double encoding.

The-Compiler commented 2 years ago

I just was informed for the reason there are so many new users around here

I'm assuming the QR bill now being mandatory in Switzerland? For me, that wasn't really the motivation, FWIW - I just got tired of using invoice.sty and creating a QR bill with Postfinance's online generator by hand.