Form fill font extra \x00 and font size

[pubpub-zz]

          **new issue created about the problem of extra \x00 due to mix up between utf8 and 8bit only charsets**

@orgmast5 created:

I'm trying to fill a form, I installed the latest build from source main tree (pip install git+https://github.com/py-pdf/pypdf.git@main) and this form is causing issues with the text being rotated (in some viewers spaced out) and on the wrong side. I think it is related to these recent issues #2636 and #2724 It is fixed when I set auto_regenerate=True and open and save again with Acrobat Reader but that's not what's intended in the docs.(https://pypdf.readthedocs.io/en/stable/user/forms.html#filling-out-forms)

Code + PDF

This is a minimal, complete example that shows the issue:

from pypdf import PdfReader, PdfWriter

reader = PdfReader("template.pdf")
writer = PdfWriter()

writer.append(reader)

writer.update_page_form_field_values(
    writer.pages[0], 
    {"Stellenbezeichnung_1": "some filled in text"},
    auto_regenerate=False
)

with open("filled-out.pdf", "wb") as output_stream:
    writer.write(output_stream)

I attached the template pdf and the filled-out pdf, you are free to use them in your tests. template.pdf filled-out.pdf

Chromium screenshot: Firefox screenshot: evince pdf reader:

I tried messing with the annotation widgets and setting my own Font and Rect but I couldn't make that work. I did look at the bit mask and it is a multiline + file select field. When i saw the multiline thing and #2636 code changes adding a "DEFAULT_FONT_HEIGHT_IN_MULTILINE = 12" I thought maybe the nightly version would be better than 4.20 release. The text is indeed smaller but it still is rotated 90 degrees clockwise.

Originally posted by @pubpub-zz in https://github.com/py-pdf/pypdf/issues/2731#issuecomment-2204404945

[orgmast5]

Thank you very much for looking into this so fast!

I see you already saw the next issue, I'm just adding more view points from my machine. The Matrix data does fix the orientation for the field we discussed. But as you noticed the text is mangled because of extra padding \x00 bytes and appears squished. Sadly the Matrix patch doesn't seem to fix all orientation errors, it is still messed up for some fields like this date in that same template. I went ahead and fresh installed from your 2731 patch (https://github.com/pubpub-zz/pypdf/tree/iss2731) and filled the template with this slightly adjusted code (it now also writes something into the date field "Tag-Monat 1" (don't be confused this one doesn't have an underscore like "Stellenbezeichnung_1" for some reason)).

New build code

from pypdf import PdfReader, PdfWriter

reader = PdfReader("template.pdf")
writer = PdfWriter()

writer.append(reader)

writer.update_page_form_field_values(
    writer.pages[0], 
    {
        "Stellenbezeichnung_1": "some filled in text",
        "Tag-Monat 1": "03.07"
    },
    auto_regenerate=False
)

with open("filled-out.pdf", "wb") as output_stream:
    writer.write(output_stream)

Now when I open this date field in some softwares it says theres a template that requires it to be in the format of "D D M M " notice the double space in the middle, I don't think orientation is related to that and I couldn't parse out anything about the pdf with PyPDF that would say thats needed but maybe it helps you.

I attached pictures from different viewers (evince, firefox, chromium) with the latest nightly version and with your even newer 2731 patch. Issue with the date happens in both versions.

Nightly PyPDF build

evince:  firefox (interestingly the date is oriented correctly here):  chromium: 

Your 2731 Patch

evince:  firefox (again the orientations are correct here but you can see the \x00):  chromium: 

I also attached the documents generated on my machine. filled-out-nightly.pdf filled-out-2731patch.pdf

Thanks again, it feels like Adobe tailored the PDF specification to be a messy pain in the ass on purpose...