How do you bypass cookie consent?

[DoYouKnowTheAnswer]

The website I'm attempting to scrape has a pop-up modal on the initial page load. It requires you to click the "Accept All Cookies" button before allowing access to the page.

How do I bypass this?

Current code:

    c := colly.NewCollector(colly.MaxBodySize(math.MaxInt32))
    timeout := 120 * time.Second
    c.SetRequestTimeout(timeout)

        c.OnHTML("tbody", func(e *colly.HTMLElement) {
        log.Println("Scraping ASX page...")
        e.ForEach("tr", func(_ int, elem *colly.HTMLElement) {
        log.Println(elem.ChildAttr("a:first-child", "href"))
                })
         })

         c.Visit("https://www2.asx.com.au/markets/trade-our-cash-market/announcements.apt")

The OnHTML function never runs because it either needs a cookie in the context or that button needs to be clicked. How do I resolve this?

[sonu27]

This is probably out of scope for this. I'd suggest using a headless web browser instead.

[WGH-]

Most cookie consent dialogs are implemented with JavaScript, and thus are invisible to simple crawlers.

The real problem with this particular page is not the cookie consent modal. The problem is that the data is loaded with Ajax. Check your browser's network tab, you likely want to request that JSON-serving endpoint directly instead.