mattsse / chromiumoxide

Chrome Devtools Protocol rust API
Apache License 2.0
744 stars 77 forks source link

How to download picture like downloading other file types? #110

Open Hecatoncheir opened 1 year ago

Hecatoncheir commented 1 year ago

I can't understand how to download images, need some example like this:

https://github.com/chromedp/chromedp/issues/660#issuecomment-758121762

hsyan2008 commented 1 year ago

Something like this:

    let (mut browser, mut handler) = Browser::launch(
        BrowserConfig::builder()
            .enable_request_intercept()
            .disable_cache()
            .build()?,
    )
    .await?;

    let browser_handle = async_std::task::spawn(async move {
        while let Some(h) = handler.next().await {
            if h.is_err() {
                break;
            }
        }
    });

    // Setup request interception
    let page = Arc::new(browser.new_page("about:blank").await?);

    let mut response_received = page.event_listener::<EventResponseReceived>().await.unwrap();
    let intercept_page = page.clone();
    let intercept_handle = async_std::task::spawn(async move {
        while let Some(event) = response_received.next().await {
            if event.type == ResourceType::Image || event.response.url.contains(".jpg") {  // filter by type,or url 
                if let Ok(res)  = intercept_page.execute(GetResponseBodyParams::new(event.request_id.clone())).await{
                         //this res should you want
                }
            }
    });
makorne commented 12 months ago
        while let Some(event) = response_received.next().await {
            if event.type == ResourceType::Image || event.response.url.contains(".jpg") {  // filter by type,or url 
                if let Ok(res)  = intercept_page.execute(GetResponseBodyParams::new(event.request_id.clone())).await{
                         //this res should you want
                }
            }
    });

This leads to :

    Error {
        code: -32602,
        message: "Invalid InterceptionId.",
    },

How to download picture?

hsyan2008 commented 12 months ago
        while let Some(event) = response_received.next().await {
            if event.type == ResourceType::Image || event.response.url.contains(".jpg") {  // filter by type,or url 
                if let Ok(res)  = intercept_page.execute(GetResponseBodyParams::new(event.request_id.clone())).await{
                         //this res should you want
                }
            }
    });

This leads to :

    Error {
        code: -32602,
        message: "Invalid InterceptionId.",
    },

How to download picture?

Sometimes I get the same error and try again a few times

makorne commented 12 months ago

Sometimes I get the same error and try again a few times

I set it to the loop and got an endless error :(

May be request_id really wrong?

The first such: request_id: RequestId("8EF6FAB20B5051A95D84DE9D0AD9000D")

But others looks like: request_id: RequestId("201296.6")

makorne commented 12 months ago

What Chrome and ChromeDriver versions do you use???

I just tried with old 107.0.5296 without success.

hsyan2008 commented 11 months ago

What Chrome and ChromeDriver versions do you use???

I just tried with old 107.0.5296 without success.

ChromeDriver 114.0.5735.198 Chromium 114.0.5735.198 Arch Linux

hsyan2008 commented 11 months ago

Sometimes I get the same error and try again a few times

I set it to the loop and got an endless error :(

May be request_id really wrong?

The first such: request_id: RequestId("8EF6FAB20B5051A95D84DE9D0AD9000D")

But others looks like: request_id: RequestId("201296.6")

These request_id all are right.

hsyan2008 commented 11 months ago

@makorne

    let page = std::sync::Arc::new(page);                                                                                                                                                                                            
    let mut loading_finished = page.event_listener::<EventLoadingFinished>().await?;                                   
    let hm = Arc::new(Mutex::new(std::collections::HashMap::new()));
    let intercept_hm = hm.clone();                         
    let intercept_handle = tokio::task::spawn(async move {                                                             
        while let Some(event) = loading_finished.next().await {
            intercept_hm                                                                                               
                .lock()          
                .await                                                                                                 
                .insert(event.request_id.clone(), ());                                                                 
        }                                            
    });                              

    let mut request_paused = page.event_listener::<EventRequestPaused>().await?.fuse();
    let mut response_received = page.event_listener::<EventResponseReceived>().await?.fuse();                          
    let intercept_page = page.clone();                                                                                 
    let hm = hm.clone();                                                                                 
    let intercept_handle2 = tokio::task::spawn(async move { 
        loop {                                
            select! {                
                event = request_paused.next() =>{                                                                      
                    if let Some(event) = event {                                                                       
                        if let Err(e) = intercept_page                                                                 
                            .execute(ContinueRequestParams::new(event.request_id.clone()))
                                .await                                                                                 
                                {                                                                                      
                                    log::warn!("Failed to continue request: {e}");
                                }                          
                    }                                                                                                  
                },                                                                                                     
                event = response_received.next() => {                                                                  
                    if let Some(event) = event {
                        if !event.response.url.contains(".jpg?challenge=") {                                           
                            continue;                                                                                  
                        }                                                                                              
                        loop {                                                                                         
                            if let None =  hm.lock().await.get(&event.request_id) {
                                tokio::time::sleep(std::time::Duration::from_millis(100)).await;                       
                                continue;                                                                              
                            }             
                            if let Ok(res) = intercept_page
                                  .execute(GetResponseBodyParams::new(event.request_id.clone()))
                                 .await
                            {
                                         //this res should you want
                            }
                            break;                         
                        }                                  
                    }                                      
                }
                complete => break,
            }
        }
    });

It works fine with me