search

sparklemotion / mechanize

Mechanize is a ruby library that makes automated web interaction easy.
https://www.rubydoc.info/gems/mechanize/
MIT License
4.39k stars 473 forks source link

Issue 403 when connecting to page with redirects #645

Closed MarcoPrins closed 1 month ago

MarcoPrins commented 1 month ago

I'm getting a 403 forbidden with mechanize when visiting this page, however the same page displays fine when using RestClient What is the issue here? Seems like mechanize is not handling redirects properly?

To replicate, please try the same url as I used below; It's a public url

Version 2.10.1 and 2.8.5 (tried both)

  def agent
    @agent ||= begin
      agent = Mechanize.new
      agent.user_agent = user_agent
      agent
    end
  end

  def user_agent
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) ' \
      'Chrome/92.0.4515.159 Safari/537.36'
  end

  # This does not work
  agent.get('https://right-to-work.service.gov.uk/rtw-view')

  # This works
  RestClient.get('https://right-to-work.service.gov.uk/rtw-view')