Unable to get CaptureWebPage[] function to work

[deepcoder]

I am able to navigate to a web page using the WebUnit package, however when I execute the function:

CaptureWebPage[]

it returns the error:

ImportString[{"message" -> "unknown error: cannot get automation extension from unknown error: page could not be found: \ chrome-extension://aapnijgdinlhnhlmodcfapnahmbfebeb/generated\ background_page.html (Session info: chrome=57.0.2987.133) (Driver info: chromedriver=2.25.426935 \ (820a95b0b81d33e42712f9198c215f703412e1a1),platform=Mac OS X 10.10.5 \ x86_64)"}, "Base64"]

---

If I have to supply an argument to CaptureWebPage[], what is the correct syntax and data type? Picture of my test below. Thanks!

[arnoudbuzing]

I can reproduce your issue, but do not yet have a fix for it. Thanks for reporting it.

[arnoudbuzing]

Probably because the web driver that I include is too old: http://stackoverflow.com/questions/42204586/chromedriver-error-unknown-error-cannot-get-automation-extension

I will try updating it and make a new release

[arnoudbuzing]

@deepcoder try the new v0.0.3 release: https://github.com/arnoudbuzing/WebUnit/releases and let me know if that works better or not.

[deepcoder]

It seems to be working now (see attached). I could not follow your install instructions as there was no .paclet file for v0.0.3, so I just replaced the WebUnit directory in my ~/Library/Mathematica/Applications folder with same from v0.0.3 zip file. I'm a noob, so I do not understand the MMA directory structure and paclet very well. Prior to the copy, I tried to execute your build.nb file, not sure what it did.

At this point I realized that you are providing the chromedriver binaries with your package. I have one installed in my /usr/local/bin directory, but I now see that this is not the one you launch. I also see you have upgraded to the most recent version of the chromedriver, which was the version I had install outside MMA:

dproffer@Davids-Mac-Pro-MacOSX-x86-64> ./chromedriver --version ChromeDriver 2.29.461585 (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b)

FYI, at one point in the chromedrivers browser I did see the following message, but not sure if this is properly reflected in my most current configuration:

StartWebSession[]

Chrome session launches with following message:

You are using an unsupported command-line flag: --ignore-certificate-errors. Stability and security will suffer.

Thanks for your work! P.S. do you have any further example of calling your various functions? I am not well enough versed to grok some of the ways to use your package?

See picture below of successful results:

[arnoudbuzing]

There should be .paclet files now here (v0.0.3 and v.0.0.4): https://github.com/arnoudbuzing/WebUnit/releases

I have added a usage.nb notebook, with the beginnings of a few examples. (Still needs a lot of work).

Thanks for using my package!

[deepcoder]

Was able to install 0.0.4 successfully via paclet install instructions and run my basic tests also successfully. I was not able to find a file usage.nb in either the paclet directory or in the contents of the .zip file.

[deepcoder]

Just found usage.nb in root of github directory for WebUnit, thanks!

[Ali99947]

Hi can someone please help me.

I have written the following code: OpenWebPage["http://sqlzoo.net/"] ClickElement[Id["n-SELECT-basics"]] TypeElement[Id["txtar_1"], "SELECT * FROM world"] this all works but how do i press the Submit SQL button underneath the textbox?

[deepcoder]

Try:

JavascriptExecute["document.querySelector('button.submitSQL').click();"]

[Ali99947]

Thank you it works! Can you explain how do you arrive at the answer? I don't know Javascript and i can't find the documentation on how to use this WebUnit package.

[Ali99947]

Thank you can you also tell me how do i get a signal when the page is updated after pressing the button. I have written a program which pulls the result from the website table. The problem is it sometimes pull the result before the page is updated.

[arnoudbuzing]

Another option, using ClickElement directly:

ClickElement[XPath["//*[@id=\"frm__1\"]/div/button"]]

To get the xpath:

1. In Chrome, open the developer tools on the page of interest.
2. Select the "Elements" tab.
3. Click the element selector icon (top left, the icon with an arrow pointing into a square).
4. Click on the element of interest (here, the blue "Submit SQL" button)
5. In the developer tools, right click on the highlighted element and click Copy > Copy XPath (submenu)
6. In WebUnit, type ClickElement[XPath[""]] and paste the copied xpath inside the double quotes.

[arnoudbuzing]

For the second question, you can use LocateElement, e.g. this returns something different before and after you click submit:

LocateElement[XPath["//*[@id=\"mw-content-text\"]/div[3]/div[2]/h1"]]

[Ali99947]

Thank you, is there a way to select an item from a drop down list after clicking the item with id?