search

selenide / selenide

Concise UI Tests with Java!
http://selenide.org
MIT License
1.83k stars 578 forks source link

Add file downloading mechanism without "href" #196

Closed asolntsev closed 8 years ago

asolntsev commented 9 years ago

Sometimes file download starts without clicking link with "href" attribute. Selenide could provide way to download those files too.

I guess it's only possible to do using proxy server (like browsermob-proxy).

leslyarun commented 9 years ago

@asolntsev Can u pls provide me with an example of using browsermob-proxy in Selenide ???

leslyarun commented 9 years ago

@asolntsev I'm using a corporate proxy. How to make browsermob to use the same proxy server ? please provide me with an example

asolntsev commented 9 years ago

I don't know exactly. I guess BrowserMobProxy could be able to intercept all http requests and forward them to your corporate proxy. But I don't know if it actually can do this.

If it cannot, we need to find or create such a proxy server.

Andrei Solntsev

2015-08-05 14:31 GMT+03:00 leslyarun notifications@github.com:

@asolntsev https://github.com/asolntsev I'm using a corporate proxy. How to make browsermob to use the same proxy server ? please provide me with an example

— Reply to this email directly or view it on GitHub https://github.com/codeborne/selenide/issues/196#issuecomment-127963117.

leslyarun commented 9 years ago

I'm using a proxy address and port no. for accessing the internet. But when I use BrowserMob, it creates it's own proxy server. If there is an way to make browsermob to use a particular proxy address and port no. , it would be very helpful. Do u have any idea ? @asolntsev

asolntsev commented 9 years ago

@leslyarun I don't know exactly if there is a way. This requires some investigation. If you get it sooner, please inform me.

ddemin commented 8 years ago

Hi all. It's simple! You can tune Browser Mob for auto downloading of files based on Content-Type header. Test case: 1) Navigate to some URL 2) Click on button which lead to response with Content-Type = application/pdf (downloading of PDF file after click) 3) Browser Mob Proxy catch Content-Type=application/pdf, save file to temp folder and replace response body with full path to PDF file

First of all you must add&start BMob (You can do it in @BeforeSuite in TestNG as example):

        // proxyServer  - field in class with type BrowserMobProxyServer
        proxyServer = new BrowserMobProxyServer(port);
        proxyServer.start();
        WebDriverRunner.setProxy(ClientUtil.createSeleniumProxy(proxyServer));

Then you must to create response filter (thanks @barancev)

public class FileDownloader implements ResponseFilter {

    private Set<String> contentTypes = new HashSet<String>();
    // ExConfiguration.CONFIG_DEFAULT_DOWNLOAD_PATH - my own field with absolute path to temporary folder
    private File tempDir = new File(ExConfiguration.CONFIG_DEFAULT_DOWNLOAD_PATH);
    private File tempFile = null;

    public FileDownloader addContentType(String contentType) {
        contentTypes.add(contentType);
        return this;
    }

    public static FileDownloader withContent(String contentType)
    {
        return new FileDownloader().addContentType(contentType);
    }

    public static FileDownloader withContents(String ... contentType)
    {
        FileDownloader downloader = new FileDownloader();
        for (int i=0; i < contentType.length; i++)
        {
            downloader.addContentType(contentType[i]);
        }
        return downloader;
    }

    @Override
    public void filterResponse(io.netty.handler.codec.http.HttpResponse response, HttpMessageContents contents, HttpMessageInfo messageInfo) {
        String contentType = response.headers().get("Content-Type");
        if (contentTypes.contains(contentType)) {
            try {
                String postfix = contentType.substring(contentType.indexOf('/') + 1);
                tempFile = File.createTempFile("downloaded", "." + postfix, tempDir);
                tempFile.deleteOnExit();

                FileOutputStream outputStream = new FileOutputStream(tempFile);

                outputStream.write(contents.getBinaryContents());
                outputStream.close();

                response.headers().remove("Content-Type");
                response.headers().remove("Content-Encoding");
                response.headers().remove("Content-Disposition");

                response.headers().add("Content-Type", "text/html");
                response.headers().add("Content-Length", "" + tempFile.getAbsolutePath().length());
                contents.setTextContents(tempFile.getAbsolutePath());
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

then set-up content in your test (PDF as example):


    @Test
    public void someTest() {
        proxyServer.addResponseFilter(FileDownloader.withContent("application/pdf"));
        Selenide.open("http://someurl.com");
        $(By.id("btn_for_start_pdfdownloading_without_href")).click();

        File downloadedPdf = new File($(By.tagName("body")).getText()));
        // You can use downloadedPdf as you wish
    }

Do not forget to stop BMob (in @AfterSuite as example)

        proxyServer.stop();
ddemin commented 8 years ago

@asolntsev can you add this into Selenide doc ?

asolntsev commented 8 years ago

@dimand58 yes, I am going to add this functionality to Selenide, so that user will not need to copy-paste this code.

ddemin commented 8 years ago

@asolntsev Will be great! Can I help with it?

asolntsev commented 8 years ago

@dimand58 Sure! Pull request would be very helpful. P.S. It seems that BrowserMobProxy doesn't work with latest Jetty versions. So, if I use latest Jetty in my project, I cannot use BrowserMobProxy :( I am right? Is there any solution?

ddemin commented 8 years ago

@asolntsev I will investigate it

ddemin commented 8 years ago

@asolntsev Try with

      <dependency>
           <groupId>net.lightbody.bmp</groupId>
           <artifactId>browsermob-core-littleproxy</artifactId>
           <version>2.1.0-beta-4</version>
       </dependency>

It works for me (with Selenide 3.0)

ddemin commented 8 years ago

@asolntsev

Sure! Pull request would be very helpful.

I have one problem - I don't know what kind of solution will be most appropriate and clear. Can you provide your 'high-level' vision of using BMob in Selenide?

ddemin commented 8 years ago

@asolntsev please investigate PR https://github.com/codeborne/selenide/pull/267

asolntsev commented 8 years ago

I have described my vision in comments for PR #267

asolntsev commented 8 years ago

Selenide 3.9.1 will be release in few hours. It will implement downloading files via BrowserMobProxy.

@dimand58 Huge thanks for your contribution to this functionality!!!

asolntsev commented 8 years ago

Now Selenide method $.download() can download any files using proxy server.

How it works:

  1. Test executes command $("selector").download();
  2. Selenide activates proxy-server
  3. Selenide clicks the element
  4. Proxy server tracks all server responses that contain http header Content-Disposition. Proxy server extracts file name from this header, and saves such a file in folder build/reports/tests.
  5. Selenide waits (up to 4 seconds) until some file(s) is downloaded.
  6. Method $.download() returns the first downloaded file.
  7. If some new windows have been opened at this time, Selenide closes all of them. It's needed when PDF is opened in a new browser window (inline).
thasherwin commented 8 years ago

Wow. That's awesome

thasherwin commented 8 years ago

I tested the new version and currently I don’t have a way to make download work with my own capabilities setup connected to a VPN with proxy server.

  1. Is there a way to do this with Selenide using your own capabilities when you are connected to VPN with a proxy server?
  2. What is the best practice to create a Selenide webdriver instance together with Selenium grid? Currently i do this by creating my own capabilities and tell selenide to use my webdriver instance.

Below a few scenarios that I used to test the download functionality: Scenario 1: Test download with no VPN connection and proxy Code:

@Test
public void testIfFileCanBeDownloadedWithOutVPNConnection(){
    open(url);
    try {
            File downloadedFile = $(".btn.btn-default").download();
            assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual: It works!!

Scenario 2: Test download with VPN connection and proxy configured using Selenide to create the webdriver instance

@Test
public void testIfFileCanBeDownloadededOnVPNWithProxy(){
    Proxy p=new Proxy();
    p.setHttpProxy("xxxx.local:8080");
    WebDriverRunner.setProxy(p);
    open (url);
    try {
        File downloadedFile = $(".btn.btn-default").download();
        assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual: It works!!

Scenario 3: Test download with my own capabilities together with a proxy server and connected to a VPN using Selenide. 

@Test
public void testIfFileCanBeDownloadedWithVPNConnectionAndProxyAndOwnCap(){
    DesiredCapabilities cap = DesiredCapabilities.firefox();
    cap.setBrowserName("firefox");
    Proxy p=new Proxy();
    p.setHttpProxy("xxxx.local:8080");
    WebDriverRunner.setWebDriver(new FirefoxDriver(cap));
    WebDriverRunner.setProxy(p);
    open (url);
    try {
            File downloadedFile = $(".btn.btn-default").download();
            assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual: it does not work. Page does not load. Probably my code is wrong. I am not configuring the proxy properly

Scenario 4: Test selenide with my own capabilities setup together Selenide connected to VPN. I did not mention any proxy. I guess i do need to mention the proxy to prevent an exception 

@Test
public void testIfFileCanBeDownloadedWithVPNCOnnectionAndOwnCap(){
    DesiredCapabilities cap = DesiredCapabilities.firefox();
    cap.setBrowserName("firefox");
    Proxy p=new Proxy();
    p.setHttpProxy("xxxx.local:8080");
    WebDriverRunner.setWebDriver(new FirefoxDriver(cap));
    WebDriverRunner.setProxy(p);
    open (url);
    try {
            File downloadedFile = $(".btn.btn-default").download();
            assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", org.hamcrest.Matchers.equalTo(downloadedFile.getName()));
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }
}

Actual: It does not work. Page does load but I get the following exception. I guess it's related to the proxy. 

java.lang.NullPointerException
    at com.codeborne.selenide.commands.DownloadFile.clickAndInterceptFileByProxyServer(DownloadFile.java:46)
    at com.codeborne.selenide.commands.DownloadFile.execute(DownloadFile.java:37)
    at com.codeborne.selenide.commands.DownloadFile.execute(DownloadFile.java:25)
    at com.codeborne.selenide.commands.Commands.execute(Commands.java:137)
    at com.codeborne.selenide.impl.SelenideElementProxy.dispatchAndRetry(SelenideElementProxy.java:85)
    at com.codeborne.selenide.impl.SelenideElementProxy.invoke(SelenideElementProxy.java:61)
    at com.sun.proxy.$Proxy5.download(Unknown Source)
    at DownloadFile.testDownload.testIfFileCanBeDownloadedWithVPNCOnnectionx(testDownload.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:643)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
    at org.testng.TestRunner.privateRun(TestRunner.java:782)
    at org.testng.TestRunner.run(TestRunner.java:632)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
    at org.testng.SuiteRunner.run(SuiteRunner.java:268)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1244)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1169)
    at org.testng.TestNG.run(TestNG.java:1064)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:74)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
asolntsev commented 8 years ago

@thasherwin Wow! Thank you for such a detailed report.

Yes, currently Selenide does download files through its own proxy server. If you create a browser instance by yourself, Selenide cannot add own proxy to it. It's not technically possible.

What we could do is provide public method like WebDriverRunner.getSelenideProxy(), so that everyone who creates his own browser can use Selenide proxy for it.

But are you sure you really need to create browser by yourself? In both Scenario 3 and Scenario 4 it's not really needed - you just create the same browser as Selenide would create automatically.

PS. This line of your code has wrong order of arguments: assertThat("Check if file: mailmerge.xls is downloaded", "mailmerge.xls", equalTo(downloadedFile.getName()));

The correct order is: assertThat("Check if file: mailmerge.xls is downloaded", downloadedFile.getName(), equalTo("mailmerge.xls"));

And actually the message is not really useful, so that I would just leave short line: assertThat(downloadedFile.getName(), equalTo("mailmerge.xls"));

thasherwin commented 8 years ago

@asolntsev Thanks for the quick reply. Well the only reason now to create my own webdriver instance is for Selenium grid. Please see the below example.

        WebDriver driver = null;
        DesiredCapabilities capability = DesiredCapabilities.firefox();
        capability.setBrowserName("firefox");
        capability.setPlatform(Platform.WINDOWS);
        capability.setCapability("machineName", "Open_Machine_FF");
        try {
            driver = new RemoteWebDriver(new URL("http://test.local:4444/wd/hub"), capability);
            WebDriverRunner.setWebDriver(driver);
        }catch(Exception e){

        }

Is it possible to create a Selenide RemoteWebDriver instance for selenium grid? If this is possible, than there is not a big need for me to have my own browser incance.

Also thanks for your comments on my assertion. I will update my code :)

asolntsev commented 8 years ago

it's easy. You just need to set both properties: Configuration.browser = "chrome"; // or "firefox" Configuration.remote = "http://localhost:4444/wd/hub";

It's enough. Selenide will create remote webdriver with needed capabilities. (Well, I am not sure about "machineName" capability. Can you try?)

Andrei Solntsev

2016-08-28 11:26 GMT+03:00 thasherwin notifications@github.com:

@asolntsev https://github.com/asolntsev Thanks for the quick reply. Well the only reason now to create my own webdriver instance is for Selenium grid. Please see the below example.

    WebDriver driver = null;
    DesiredCapabilities capability = DesiredCapabilities.firefox();
    capability.setBrowserName("firefox");
    capability.setPlatform(Platform.WINDOWS);
    capability.setCapability("machineName", "Open_Machine_FF");
    try {
        driver = new RemoteWebDriver(new URL("http://test.local:4444/wd/hub"), capability);
        WebDriverRunner.setWebDriver(driver);
    }catch(Exception e){

    }

Is it possible to create a Selenide RemoteWebDriver instance for selenium grid? If this is possible, than there is not a big need for me to have my own browser incance.

Also thanks for your comments on my assertion. I will update my code :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/codeborne/selenide/issues/196#issuecomment-242962858, or mute the thread https://github.com/notifications/unsubscribe-auth/AARE3VjddbnOTb9h9TKCWPREcGKOEjbIks5qkUYcgaJpZM4FXEwI .

thasherwin commented 8 years ago

@asolntsev I tested it on the grid using;

 Configuration.browser = "chrome"; // or "firefox"
  Configuration.remote = "http://localhost:4444/wd/hub";

and it works. I can now close my issue #317 related to downloads using jenkins :) I do mis the capabilty option. With that i can tell selenium to use a specific machine connected the grid. Maybe something for a pull request if this option is not yet built?

Thanks

asolntsev commented 8 years ago

Yes, definitely. It's a perfect candidate for pull request. Let's do it!

On Aug 29, 2016 11:59 AM, "thasherwin" notifications@github.com wrote:

@asolntsev https://github.com/asolntsev I tested it on the grid using;

Configuration.browser = "chrome"; // or "firefox" Configuration.remote = "http://localhost:4444/wd/hub";

and it works. I can now close my issue #317 https://github.com/codeborne/selenide/issues/317 related to downloads using jenkins :) I do mis the capabilty option. With that i can tell selenium to use a specific machine connected the grid. Maybe something for a pull request if this option is not yet built?

Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/codeborne/selenide/issues/196#issuecomment-243070320, or mute the thread https://github.com/notifications/unsubscribe-auth/AARE3WXYiSWkyd93MnxjLWnFaA7PYOZ6ks5qkp9ZgaJpZM4FXEwI .

asolntsev commented 8 years ago

Yes, definitely. It's a perfect candidate for pull request. Let's do it!

On Aug 29, 2016 11:59 AM, "thasherwin" notifications@github.com wrote:

@asolntsev https://github.com/asolntsev I tested it on the grid using;

Configuration.browser = "chrome"; // or "firefox" Configuration.remote = "http://localhost:4444/wd/hub";

and it works. I can now close my issue #317 https://github.com/codeborne/selenide/issues/317 related to downloads using jenkins :) I do mis the capabilty option. With that i can tell selenium to use a specific machine connected the grid. Maybe something for a pull request if this option is not yet built?

Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/codeborne/selenide/issues/196#issuecomment-243070320, or mute the thread https://github.com/notifications/unsubscribe-auth/AARE3WXYiSWkyd93MnxjLWnFaA7PYOZ6ks5qkp9ZgaJpZM4FXEwI .