Closed emshihab008 closed 3 years ago
Hola emshihab008,
I would suggest to transform your pdfs in pngs to extract via r.read() the content.
Hope it will help you, fi-do
Hi @fi-do According to your suggestion, I ran the code. But I didn't get the proper result. Is it possible with RPA Python? or I have to use python OCR module (opencv) for the accurate result. And is there any possibility to snap specific region of png file for OCR ?
import rpa as r r.init(True, True) r.wait(2)
r.url("https://pdf2png.com/de/") r.click("choose.png") r.dclick("b.png") r.wait(10) r.click("all_files.png") r.rclick("zip.png") r.click('extract.png') r.run('C:\Users\shihab\Downloads\pdf2png\books') print(r.read(100,87,889,702)) r.close()
Output:
C:\Python\Python39\python.exe C:/PzcharmProjects/Practice.py -y Paste T ââ.-. Fâ. Move Copy Del EL âlwht J» 619.609 uon. E] Pam momâ t°' t°' ' E' ââ1â - j w 15 ist, wcnn das Geld nur ï¬ir cincn Compute ï¬.â Clipboard Organize " l â'1â â , f' fu dzw â Sch" . r t . âW Russo \on W n anz1g ulem rucht? man
This PC > Downloads w [LâLâNcm .. meint Peggy Ratsch, diC Spczialistin far lnfor- «. mWWu-chnik an der Schulbehordc des Bezirks Balti- 55 vToday (9) â1;â. m, Ichrer bekommen das Geld ï¬ir mehr Germ «hr lcicht, wcnn sic nachwcisen, dass sie eincn Computer mmmll einsctzen. Mit andcrcn Wonen: Besorge dir zu- mchst cincn Computer, dann wirst du spiter mehr be- is knmmcn. Das Zicl ist vcrmutlich, dass ieder Schiller einen âW's b°°" Computer crhélt. Ich bezwciï¬e allerdings, dass Peggy nts Ratsch dann aufhércn wiirde, noch mehr Geld zu fordem. ds V Yesterday (1) Technologen schcn in def Technologie nut die Lésuns V0â â Prohlcmcn, nic cin Problem. . ' W Présidcnt Clinton hat verkiindet, duos ein Compute: 131 E jcdcm Klasscnzimmcr Ziel den Bildungsprosramm! â1 . . nicht als Mincl fï¬r die Ambildung, Iondem :18 1hr 210" 15mm HM" Was ist an dicscr Vorstdlung falsch? chn die Tedll'lo' 'ISK(E:) V Earlierthis week (1) logic die Krénung des Unterrichts damellt, weâdâ âCh 'ISK (E) natiirlich die Wcrtigkciten in den Klassenzimmem andern. 'ISK(G:) a An dcr Poly High School im kalif0mischen 1-008 M war schon langc Zcit das DaCh undidâ. â3d â3m PDF [,0 ft . . W. Was war P e das Regenwaw 1n die Ian! RECHWâ Zâ â"1? Zunichst mum die Com!"ltef W w Plâ. v Last week (17) DEâ SChulleitcr neg dmchsagen, die Lchref kc.)nnï¬Â¢n nkï¬ U Sickc m âmm. Kan : D m WM- 3â. vâ. â."âââ.. . . _ L... .n. M '09
Hi Ekram,
I believe RPA for Python's OCR (Tesseract) is similar to Python OCR since Tesseract is the industry standard. However, your scanned image is not a properly scanned image, it has deformation due to not being flat-scanned. Even using commercial paid RPA tools it is unlikely to get good accuracy.
Even if you scan it perfectly, to get the best results, I recommend using some 3rd party apps in below post - https://www.linkedin.com/posts/kensoh_rpa-tagui-activity-6780711825156775938-Wyt4
These companies spent all their resources getting it right to convert scanned PDFs into digital form. Their app takes care of pre-processing like aligning orientation, color correction etc. So it is likely you get the best results from there (almost 200 languages supported).
To snap specific region, you can consider taking a snapshot of your PDF viewer, use an image editor to remove all dynamic content to make it transparent. Then save as .png. When you do r.read('frame.png'), OCR will be performed on their PDF viewer for you to do further processing. For eg, for macOS PDF view, you can do something like below (try downloading the image and you can see many parts are made transparent) -
Hello Kensoh, Thank your for information. I know this document is quite unorganized. That's why I tried OCR using this document. If it is possible with that document, then every document can be processed. And thank you for your solution regarding snapping. I have another question. Can I read document using r.read('apple.png')? or I have to provide coordinate?
import rpa as r r.init(True, False) k=r.read('apple.png') Print(k) r.close()
Output:
[RPA][ERROR] - TagUI process ended unexpectedly [RPA][ERROR] - use init() before using send()
I see.. Yes that makes sense your thinking. For your error I have not come across this before or raised by other users.
Can you attach the log file for SikuliX here? From the installation location below, there is TagUI folder, then src\tagui.sikuli*.log
For Windows, location of installation is %APPDATA%\tagui (%APPDATA% is usually C:\Users\Username\Appdata\Roaming)
It is strange because your previous example using coordinates can capture but using image fails.
Other than the logs, can you also attach the apple.png file here? I'm assuming the apple.png is in the same folder as where you run the python script.
Hi Kensoh, I ran the code today again and got the below output.
[RPA][ERROR] - cannot find apple.png Process finished with exit code 0
I have attached log file and original apple png file. Please find.
Hi Ekram,
When I run read() on an image that can't be found on the screen, I get similar errors too in the log like below -
[tagui] START - listening for inputs
[tagui] INPUT - [1] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [1] ERROR
[tagui] INPUT - [2] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [2] ERROR
[tagui] INPUT - [3] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [3] ERROR
[tagui] INPUT - [4] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [4] ERROR
That is correct and expected. But in your case you are saying that the Python program crashes is it?
Can you create the following file and put ocr.tag and apple.png in ~/.tagui/src folder?
ocr.tag
read apple.png to ocr_text
echo `ocr_text`
Then try running the following to run the RPA using TagUI directly.
cd ~/.tagui/src
./tagui ocr.tag
Above steps will help to isolate the problem whether program is crashing at TagUI / SikuliX or something wrong in Python.
Hi Ken, Initially, when I ran the code it was showing the following output: [RPA][ERROR] - TagUI process ended unexpectedly
But last time, when i ran again, it showed: [RPA][ERROR] - cannot find apple.png
According to your instruction, i created ocr.tag file and copy the code and put it recommended folder.
ocr.tag.txt
But I am confused about last line (cd ~/.tagui/src ./tagui ocr.tag). How should I write?
Oh sorry I see you are on Windows. You can do the following from command prompt -
cd c:\Users\shihab\AppData\Roaming\tagui\src
tagui ocr.tag
If the same error happens, then issue is on TagUI side. See if you can join the weekly call at Thursday 4-5pm (UTC+8), that should be 10-11am time in Germany. Another possible issue is when the image resolution for Retina displays. Let me search that issue solution and paste here.
See these 2 issues related to Retina display. The gist is that the image captured actually is a different density and much larger / smaller than actual screen. So users do some settings change to make capturing image work correctly.
https://github.com/kelaberetiv/TagUI/issues/896 https://github.com/kelaberetiv/TagUI/issues/818
Hi Ken,
Using command prompt, I tried to open ocr.tag. Bu it is showing cannot fine ocr.tag.
And i have also checked the following issue. kelaberetiv/TagUI#896 kelaberetiv/TagUI#818 But these are all about retina display isuue. I am using windows, ASUS Laptop, LED display, pixel size (1366& * 768).
I see.. This is a really strange issue. There doesn't seem to be anything else we can check, other than meeting on a Zoom call so that I can directly test on your computer. Will coming Thursday 10am Germany time be good for you to meet?
Since this issue is an upstream TagUI issue, we can meet on the weekly Zoom call to troubleshoot.
URL for the Zoom meeting is https://github.com/kelaberetiv/TagUI/issues/914
By the way, just to be sure, on your screen there is really the image apple.png showing but it does not detect correct?
Hi Ken, Yes. I can attend the zoom meeting on Thursday 10 am. yes. it cannot find the png file.
Ok Ekram, see you then!
Hi Ekram, sorry for dropping from the call earlier due to urgent company meeting.
Can you try the following?
Use the snipping tool to take a screenshot of your start button at bottom left of your screen. Save it as start.png
After that, in your script, see if you can do r.click('start.png') on the start button. If yes then the computer vision is ok.
And the next step will be you are trying to read using OCR of an image on the screen, how the read() works is you must really have it on the screen. It is not opening a file and performing OCR on it in the background. For that, you can use other tools on Python or 3rd-party tools.
For RPA for Python, and TagUI, the read() to capture OCR is only for text that is really now visible on the screen. It usually works by you providing a window frame of the PDF viewer but with the contents deleted to be transparent and save as a GIF. For example, what I share above - https://github.com/tebelorg/RPA-Python/issues/228#issuecomment-810551911
Hi Ken, I am sorry for late reply. I was a little bit busy. I could not concentrate on my project. According to your suggestion, I tested visual automation. It works perfectly. Currently, I am using picasa photo viewer. If i use another photo viewer, is there any possibility to work?
Hi Ekram, there are 2 things to check -
Hi Ken,
1. Tried to read image documents:
import rpa as r
r.init(True, False)
r.wait(2)
print(r.read('apple.png'))
r.close()
3951!?
Process finished with exit code 0
import rpa as r
r.init(True, False)
r.wait(2)
r.dclick('apple.png')
r.wait(2)
print(r.read('apple.png'))
r.close()
output: C:\Python\Python39\python.exe C:/PzcharmProjects/Practice.py g
Process finished with exit code 0
2. Tried to read pdf document
import rpa as r
r.init(True, False)
r.wait(2)
print(r.read('png2pdf.png'))
r.close()
output: C:\Python\Python39\python.exe C:/PzcharmProjects/Practice.py
Process finished with exit code 0
import rpa as r
r.init(True, False)
r.wait(2)
r.dclick('png2pdf.png')
r.wait(2)
print(r.read('png2pdf.png'))
r.close()
output: C:\Python\Python39\python.exe C:/PzcharmProjects/Practice.py [RPA][ERROR] - cannot find png2pdf.png
Process finished with exit code 0
To use read() to perform PDF to OCR, the PDF file must be opened already. And after opening, the input to read() should be same image resolution as the PDF viewer on the screen. For Retina display the snapshot tool will need manual downsizing.
What I'll recommend next is you try other tools like below, which can convert scanned PDF to digitalised PDF. There is free version with watermark but it should hopefully work for your use case.
https://www.tracker-software.com/product/pdf-xchange-editor
After you convert your PDFs into digital PDFs, then you can run RPA for Python to do other next steps. Or if converting to digital PDF is the goal, then perhaps no need to use RPA tools at all. Some of these PDF tools have functionality to auto-convert new files added to a folder.
Let me know how it goes!
Thank you for your suggestion. I have changed scanned pdf to editable pdf using pdf exchange editor. Still I am facing the same problem. I have attached the editable pdf so that you can also try. But if i use coordinate, then it works. png2pdf.pdf
import rpa as r
r.init(True, False)
r.wait(2)
r.dclick('png2pdf.png')
r.wait(2)
**print(r.read('png2pdf.png'))**
r.close()
**output:**
C:\Python\Python39\python.exe C:/PzcharmProjects/Practice.py
[RPA][ERROR] - cannot find png2pdf.png
Process finished with exit code 0
import rpa as r
r.init(True, False)
r.wait(2)
r.dclick('png2pdf.png')
r.wait(2)
**print(r.read(42,209,1082,644))**
r.close()
**output:**
C:\Python\Python39\python.exe C:/PzcharmProjects/Practice.py
The ship drew on and had safely passed the strait, which some volcanic shock
has made between the Calasareigne and Jams islands; had doubled Pomégue,
and approached the harbor under topsails, jib, and spanker, but so slowly and
sedately that the idlers, with that instinct which is the forerunner of evil, asked
one another what misfortune could have happened on board. However, those
experienced in navigation saw plainly that if any accident had occurred, it was
not to the vessel herself, for she bore down with all the evidence of being
skilfully handled, the anchor a-cockbill, the jib-boom guys already eased off, and
standing by the side of the pilot, who was steering the Pharaon towards the
narrow entrance of the inner port, was a young man, who, with activity and
vigilant eye, watched every motion of the ship, and repeated each direction of
the pilot.
Process finished with exit code 0
Next time see if you can join the weekly Zoom call again to look at it together. I don't think the following lines make any sense. Double-clicking on an icon image is to open the PDF file. But doing OCR by looking for the PDF icon is strange.
r.dclick('png2pdf.png')
r.wait(2)
**print(r.read('png2pdf.png'))**
But since you already have the editable PDF, what you need to do is automate opening the PDF, then do the following
r.dclick('pdficon.png')
r.wait()
r.keyboard('[ctrl]a')
r.keyboard('[ctrl]c')
pdf_text = r.clipboard()
Yeah sure. I will attend. I have also a same question regarding double clicking.
And regarding editable PDF, you are right . But that was non editable pdf. I made it editable using OCR tool of pdf exchange editor.
Hi. I would like to inform you that I need to process Scanned invoice PDF format or invoice JPEG/PNG format. In this regard, can I use any RPA python tool to extract the necessary information?
Regards, Ekram