mimic browser state broken by HttpOnly cookies

[arthurblake]

This is a great technique for downloading files with selenium, but if the server is setting it's session cookies with HttpOnly, it does not work.

Under the hood, driver.manage().getCookies() is getting the cookies via javascript which does not have access to HttpOnly cookies. Do you have any good ideas for solving this problem?

[Ardesco]

I don't have any reliable ways of doing it. We are in effect doing a CSRF to mimic the session and gain access to content from what is effectively a different browser. Understandably a lot of work has been put into stopping this sort of thing.

You could try explicitly defining the cookie locally (that would assume you knew in advance what the cookie looked like, or you knew enough about the structure to dynamically build an expected cookie with the right information).

Another option would be to get the devs to code in a switch that turns on/off httponly so that you can just use normal cookies in your test environment (Testability is a feature of good code).

Otherwise it will be a case of looking for vulnerabilities with HTTPOnly implementations e.g. http://www.natexim.com/how-to-bypass-httponly/

This is not really a reliable way to do things though.

[arthurblake]

Unfortunately I'm not in control of the server so I can't change the way cookies are being set (I'm not using Selenium as a test platform but instead to automate a routine download from a remote site that I'm not in control of.) Ideally Selenium needs a way to get the cookies other than using javascript (like directly from the automation DLL in IE for example.) I don't know if that is possible right now.

[Ardesco]

That functionality isn't built into Selenium (unless something has changed recently). You would need to hook directly into the automation DLL yourself and use that information to create your cookies (assuming it can give you the information you need to do that).

[daluu]

As a workaround, can't one use a proxy (or similar tool) to capture the httpOnly cookie that's sent from the server? So if you put Selenium/etc. behind a proxy, you can then search the proxy log for the response from the server domain you are expecting to receive the cookie from and parse out the value of that cookie to then create externally to download file, etc. This does require some extra setup and knowing the format/syntax of the cookie you're expecting (though not its value assuming that can be dynamic like a session ID) and the domain/server it's coming from for some particular request.

[arthurblake]

Yes, that's the same conclusion I came to and that's exactly what I'm working on.

[arthurblake]

Actually you don't need to know anything about the cookies. I'm making my proxy simply save all the cookies (like a browser would) and then creating a side interface for the selenium program to query the proxy for the cookie store before proceeding. It's extremely easy to slap together a custom proxy like this using NodeJS.

[Ardesco]

Sounds like a good idea :)

[daluu]

If possible, it would be great if you could share the code of your work Arthur. Save others from having to create their own for the same thing.

[arthurblake]

I was under the gun to get this working quickly. I ended up figuring out a way to download with the browser by using a combination of java.awt.Robot and some vbs scripts to make sure the browser always has focus and to automate clicking the save button. I think using the proxy would be a really good general purpose solution, but it becomes more challenging with SSL - perhaps someone will pick up the torch later on down the road. Thanks for your comments and thoughts.