search

aisingapore / TagUI

Free RPA tool by AI Singapore
Apache License 2.0
5.58k stars 580 forks source link

How do I print screen a web browser tab to a file? - here are a few ways to do it #916

Closed marcelocecin closed 3 years ago

marcelocecin commented 3 years ago

Hi there ! I have the following scenario there is a login form and after the credentials are validated a new tab is opened with a web RDP system how can I get a screen snap of this new tab? thanks !

kensoh commented 3 years ago

Hi Marcelo,

Base on your scenario, I assume that you are doing this from a web browser. Normally for webpages, there is a popup step which can set the context for which tab you are trying to automate on. More details here - https://tagui.readthedocs.io/en/latest/reference.html#popup

But in this case, doing a simple snap page to screenshot.png within the popup step might not work because the remote desktop object may not be something that is renderable directly. But you can test it out.

If that doesn't work, you can use the computer vision engine to do it with the following. That will do a full screen shot of your computer. Be sure to first using a click tab_identifier.png to activate the new tab if it is not already on the screen.

snap page.png to screenshot.png

You could also do the screenshot by specifying the boundaries of what to capture, for example below.

snap (x1,y1)-(x2,y2) to screenshot.png

To get the (x,y) coordinates of your mouse cursor position, move your mouse to the upper left boundary and do the following. Then repeat for the bottom left boundary. You could also use some 3rd-party apps that show your mouse (x,y) coordinates on the screen.

echo `mouse_xy()`

Note that you can run TagUI in live mode with tagui live from the command prompt for easy testing of such commands. Alternatively, you could insert live step in your workflow to tell TagUI to pause and enter live mode for all these experimentation.

Lastly, if you don't mind capturing the whole desktop, you could also try the keyboard step to printscreen -

keyboard [win][printscreen]
kensoh commented 3 years ago

Adding on, for macOS, I can do the print screen with keyboard [cmd][shift]3. I'm assuming for Windows the [win][printscreen] key combination will automatically save the screen to a file.

Also, to switch to the new tab, you could consider doing something like keyboard [ctrl][tab]

kensoh commented 3 years ago

Adding on, more on how this solution is found on this YouTube video - https://youtu.be/M4JbL-Mr93Y

marcelocecin commented 3 years ago

thanks @kensoh ! the snap page.png to screenshot.png works perfectly !

marcelocecin commented 3 years ago

hi @kensoh how are you doing ? I'm trying to get OCR using this command: save (456,287)-(592,304) to text.txt but nothing happens where can I check the log for this ? thanks !

kensoh commented 3 years ago

Hi @marcelocecin, the file should be save to the same folder as your automation flow file.

Hmmm, can you see what is inside the log file tagui\src\tagui.sikuli\tagui.log ?

(copying @ruthtxh for info)

marcelocecin commented 3 years ago

[error] script [ tagui ] stopped with error in line 584 [error] java.lang.UnsatisfiedLinkError ( java.lang.UnsatisfiedLinkError: Error looking up function 'TessPDFRendererCreateTextonly': /lib/x86_64-linux-gnu/libtesseract.so.4: undefined symbol: TessPDFRendererCreateTextonly ) [error] --- Traceback --- error source first line: module ( function ) statement 354: main ( text_read ) temp_text = region_layer.text() 331: main ( save_intent ) return text_read(raw_intent) 526: main ( parse_intent ) return save_intent(script_line) [error] --- Traceback --- end -------------- IDE terminated: returned: 1

kensoh commented 3 years ago

Hi Marcelo, it looks like you are running on Linux. Because the OpenCV and Tesseract libraries can't be packaged directly, can you look through below guide to install them manually? https://sikulix-2014.readthedocs.io/en/latest/newslinux.html

Let us know how it goes!

marcelocecin commented 3 years ago

hello Ken, do you know in which Linux folder do I need to put the tess4j-4.5.4.jar file? https://sikulix-2014.readthedocs.io/en/latest/textandocr.html?highlight=tess4j thank you

tesseract 4.1.1 leptonica-1.79.0 libgif 5.1.4 : libjpeg 8d (libjpeg-turbo 2.0.3) : libpng 1.6.37 : libtiff 4.1.0 : zlib 1.2.11 : libwebp 0.6.1 : libopenjp2 2.3.1 Found AVX Found SSE Found libarchive 3.4.0 zlib/1.2.11 liblzma/5.2.4 bz2lib/1.0.8 liblz4/1.9.2 libzstd/1.4.4

Python 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. import cv2 cv2.version '4.2.0'

kensoh commented 3 years ago

Hi @marcelocecin I don't have the answer to this question specific to the setup on Linux. I've raised an issue on the SikuliX project so that its maintainer can advise - https://github.com/RaiMan/SikuliX1/issues/417 (copying @ruthtxh)

marcelocecin commented 3 years ago

I downgraded tesseract to version 3.05.02 the error has now changed: Failed loading language 'eng' Tesseract couldn't load any languages!

kensoh commented 3 years ago

Hi Marcelo,

From the SikuliX guide above - https://sikulix-2014.readthedocs.io/en/latest/newslinux.html

It looks like recommended to have v4 (It must be a version 4.x). That might be the likely reason why this error message appears when you are trying to use the OCR capabilities of the packaged SikuliX engine in TagUI.

kensoh commented 3 years ago

Adding on, Raimund on his SikuliX issue for this asked me for the architecture of how TagUI uses SikuliX.

I'll send that over and see if he could make recommendations on how TagUI should incorporate newer SikuliX engines.

marcelocecin commented 3 years ago

hi Ken because of this comment, I downgraded

I already tried to define in the TAG code in Python the TESSDATA_PREFIX environment variable but even so the error persisted

I also tried to edit the tagui.py file, including that same definition, however the error persists

marcelocecin commented 3 years ago

hey Ken now is partially working! I need help in just one detail! after placing the traineddata files in the /home/user/.Sikulix/SikulixTesseract/tessdata/ folder the above error has stopped occurring now what is happening is something related to special characters I am OCR the phrase Versão 4.82.01.4:

however the generated file is bringing ã as ã - Versão: 4.82.01.4

kensoh commented 3 years ago

Hi Marcelo, I've seen something like that before, it seems something to do with the OCR engine. See this issue - https://github.com/kelaberetiv/TagUI/issues/905

We might have to raise this to Raimund at his SikuliX's page to see if he has similar feedback from other users.

kensoh commented 3 years ago

I don't have a solution yet for this OCR issue, but would also like to figure out a solution tgt with Raimund.

marcelocecin commented 3 years ago

hello Ken just some instructions to install tesseract-3.05.02 on Ubuntu 18.04.5

wget https://github.com/tesseract-ocr/tesseract/archive/refs/tags/3.05.02.tar.gz
tar xzf 3.05.02.tar.gz
cd tesseract-3.05.02
./autogen.sh
./configure
make
sudo make install
sudo ldconfig

then download the tessdata from https://github.com/tesseract-ocr/tessdata and put at /usr/local/share/tessdata and check it running the command tesseract --list-langs