Add support for Selenium's captureNetworkTraffic option

[spooning]

Originally submitted to Google Code by mangaroo on 2 Oct 2011

Noticed recently that Selenium has a capture network traffic option that's available for Python and Java. It works with Firefox, and for other browsers, they need to be started under proxy mode to work (ieproxy, safariproxy). Not sure if SeleniumLibrary supports the proxied browser versions though, didn't see mention of it on Open Browser keyword documentation.

Found out per these references:

http://jira.openqa.org/browse/SRC-758

https://code.google.com/p/selenium-profiler/source/browse/trunk/web_profiler.py

I've supplied a partial patch to enable support for this option and a test that demonstrates it, by calling the Selenium Python API method for that option.

It is partial because I'm not sure where to appropriately implement new keyword in this (dynamic?) library that relays the SeleniumLibrary call to the actual Selenium Python API, so as to not need to use Call Selenium API keyword.

If you can point me to where I should make new keyword for this, or how to create new Python class/file for this dynamic library to call, I can submit a follow up patch.

Otherwise, it should be relatively simple for the team to add a wrapper to Selenium Python API method "captureNetworkTraffic" that's in selenium.py.

FYI, patch to enable option is to simply add another optional argument to open browser keyword for browser configuration options, and pass that option to _connect_to_selenium_server() to then pass to selenium.start(). Incidentally, selenium.py already had fix for Jira SRC-758 mentioned above, so I didn't have to patch that.

On a side note, not sure if Robot Framework natively supports XML and JSON data, but I guess users can write additional Python library to parse XML and JSON data as needed that can be returned from captureNetworkTraffic. For average user, perhaps easiest to just use the plain text output option.

[spooning]

Originally submitted to Google Code by @pekkaklarck on 3 Oct 2011

1) Capturing the traffic looks pretty interesting. Do you know does capturing slow down execution or potentially cause some other problems? If not, then it could be enabled automatically. Alternatively, if it's possible to configure that dynamically after starting the browser, we could implement keywords like Start/Stop Network Traffic Capture.

2) SeleniumLibrary isn't actually a dynamic library but gets keywords from helper libraries that it inherits. You can see that if you look at how SeleniumLibrary class is created in init.py. In this case it would probably be a good idea to create a new helper library for network traffic monitoring.

3) I think that in addition to returning the data as plain, XML, or JSON, the library should support parsing the returned data into Python dictionary or some other object. This would make it easier to verify the contents of the returned data without needing another library.

[spooning]

Originally submitted to Google Code by mangaroo on 3 Oct 2011

1) I don't know, just recently learned of the feature and have only used it for a little. I would assume there may be a performance penalty for the option, though it isn't likely to be big or cause issues. Probably need to do some testing or check with Selenium developers to confirm. Based on available articles on the internet, I doubt the option is configurable for start/stop. It is started with selenium.start(options) and ends when you call stop, which I assume correspond to browser open & close.

2) Thanks for the clarification Pekka, I'll look at that more closely to see what I can do.

3) That's a great idea, I'll look into that as well.

Also, an FYI, captureNetworkTraffic works after some activity occurs so can only be called after some other keywords, like after opening browser and navigating to some URL. And once called, it clears the captured traffic cache, so the next call requires more web browsing first, and it won't contain contents of previous call (unless you navigate back, etc.).

[spooning]

Originally submitted to Google Code by mangaroo on 5 Oct 2011

It looks like we would want the feature to be configurable - enable/disable as needed when the library starts up (imported) rather than auto-enabled.

On recent repeated use of CoreyG's web_profiler.py (though a slightly minor modified version by me), running once for single URL vs running repeatedly cycling over several URLs (but opening & closing browser for each URL visited rather than reuse same session - quick hack to test stuff), noticed that Firefox was less stable than IE or Safari in running the test throwing the error below, haven't investigated the details. Was using Firefox 3.6 and Selenium RC server. Didn't yet notice exception using IE or Safari.

Traceback (most recent call last): File "web_profilerM.py", line 264, in <module> main() File "web_profilerM.py", line 77, in main run(site, path, browser, details, host, port) File "web_profilerM.py", line 107, in run raw_xml = sel.captureNetworkTraffic('xml') File "C:\Python27\lib\site-packages\selenium-2.6.0-py2.7.egg\selenium\selenium .py", line 2010, in captureNetworkTraffic return self.get_string("captureNetworkTraffic", [type,]) File "C:\Python27\lib\site-packages\selenium-2.6.0-py2.7.egg\selenium\selenium .py", line 223, in get_string result = self.do_command(verb, args) File "C:\Python27\lib\site-packages\selenium-2.6.0-py2.7.egg\selenium\selenium .py", line 217, in do_command raise Exception, data Exception

Note that the Selenium driver is the default one, not the SeleniumLibrary version.

[spooning]

Originally submitted to Google Code by mangaroo on 16 Oct 2011

Development note, suggested keyword to implement:

Capture Network Traffic with argument name of "type" taking values of: plain, xml, json, and object.

Object would be a Python dictionary object, likely similar in structure to JSON output.

[spooning]

Originally submitted to Google Code by @pekkaklarck on 4 May 2012

Selenium RC has been deprecated and developing this library is thus ending. Selenium 2 (a.k.a. WebDriver) and Selenium2Library are the way forward. If you feel this issue is important, consider submitting it to the issue tracer of Selenenium2Library project.