Supporting RTL Languages

[mkhashoggi]

Feature request

• A description of the feature you would like to have Currently PDFMiner does not supports RTL languages such as Arabic. This is the output of the current version of PDFMiner: PDF File:

  

  Output:

  

• If relevant, the context that you are in. What are you trying to achieve? What is needed is to apply Unicode bi-directional rules to allow reordering characters in bi-directional text. http://unicode.org/reports/tr9/#Explicit_Levels_and_Directions

• If possible, an example of what you want to achieve. Include the PDF that you are working on. Include the output that you would like to have. The desired output should look like this:

  

[vaknin]

This is important for Hebrew as well.

[pietermarsman]

Hi @mkhashoggi, thanks for the suggestion and the PR. I need some time to familarize myself with the logic and rules of RTL languanges to be able to review the PR.

[pietermarsman]

I did some reading about bi-directional text to understand the changes that we need to make. Please correct me if I'm wrong. I've no experience with right-to-left text, so I might have an overly left-to-right inclined way of thinking.

To summarize:

• Bi-directional text contains text from left-to-right and right-to-left. Its not so much about the alignment on the page, but more about the order of the characters.
• At first I thought the order of characters is simple because pdfminer.six outputs them in the order they appear. But that's not the case. Pdfminer.six outputs unicode, and unicode orders characters logically, e.g. in the sequence they are meant to be interpreted. Currently, pdfminer.six outputs unicode characters from left-to-right, but for right-to-left character blocks this should change to first outputing the right-most character. The displayer of the unicode output (e.g. the terminal or browser) is then responsible for rendering the first character of a right-to-left character string on the right. So basically reversing the reverse (from a left-to-right point-of-view).
• It's not trivial to determine what characters should be left-to-right and which should be right-to-left. Some characters called "strong" because they are always one or the other. But there can also be characters that could appear in both left-to-right and right-to-left text. In this case the context determines how the character should be displayed.
• There are special control characters for unicode to indicate the directionality of the text.

So what we need is a interpreter for text lines (i.e. LTTextLine) that detects which part of the text is left-to-right and which is right-to-left and add the necessary unicode characters if necessary. This is precisely what PR #516 is about.

[mojivalipour]

I understand that there is a PR #516 in progress regarding this support request. However, I was confused if this feature is currently usable or not. I wondered if there is an example that shows how to use this feature if applicable?

[JStyle21]

Hi,

If this is implemented what is the hold up here? Can anyone update?

[pietermarsman]

Anyone willing to work on #516? It implements this feature but it has been inactive for a while. And it does need extra work.

[barhemo]

Hey, when it will be ready? right now when i try to read from a pdf, the hebrew characters are missing

[pietermarsman]

No one is working on this currently

[taneron]

I recently needed this and tried to hack around until I got something that worked for me Maybe its useful for someone https://pypi.org/project/pdfminer.rtl