gottfrois / link_thumbnailer

Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.
MIT License
512 stars 106 forks source link

Twitter url ERROR respond: "We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter? #141

Closed Paolo454 closed 1 year ago

Paolo454 commented 4 years ago

Hello. If I try to use link_thumbnailer with a Twitter link I get this error:

"We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?

I think I get this error because Twitter remove legacy version from 1 June 2020. I get this error in the Demo version as well.

How can I solve it? Thank you

gottfrois commented 4 years ago

By default the gem passes its own UserAgent which is probably used by some website to check whether JS is available or not. Can you try changing the user agent to something like Chrome or Firefox would use?

https://github.com/gottfrois/link_thumbnailer/blob/master/lib/link_thumbnailer/configuration.rb#L42

You can change the config globally or per request.

msmiller commented 4 years ago

Setting the agent doesn't seem to help:

2.6.3 :008 > object = LinkThumbnailer.generate('https://twitter.com/elonmusk/status/1284307886319919104', user_agent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36')
ETHON: started MULTI
ETHON: performed MULTI
 => #<LinkThumbnailer::Models::Website:0x00007fafaad41910 @images=[], @videos=[], @url=#<URI::HTTPS https://twitter.com/elonmusk/status/1284307886319919104>, @title="", @description="We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?", @favicon="https://abs.twimg.com/responsive-web/web/icon-svg.9e211f64.svg">
1dolinski commented 4 years ago

Hi, thanks for raising this issue, I'm seeing the same. I ran through the different agents found here: https://www.whatismybrowser.com/guides/the-latest-user-agent/chrome

For each, Twitter returns We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter? Google returns nil for the description

@gottfrois do you see the same?


    agents = [
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
      "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
      "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
      "Mozilla/5.0 (iPhone; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/83.0.4147.71 Mobile/15E148 Safari/604.1",
      "Mozilla/5.0 (iPad; CPU OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/83.0.4147.71 Mobile/15E148 Safari/604.1",
      "Mozilla/5.0 (iPod; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/83.0.4147.71 Mobile/15E148 Safari/604.1",
      "Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
      "Mozilla/5.0 (Linux; Android 10; SM-A205U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
      "Mozilla/5.0 (Linux; Android 10; SM-A102U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
      "Mozilla/5.0 (Linux; Android 10; SM-G960U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
      "Mozilla/5.0 (Linux; Android 10; SM-N960U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
      "Mozilla/5.0 (Linux; Android 10; LM-Q720) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
      "Mozilla/5.0 (Linux; Android 10; LM-X420) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
      "Mozilla/5.0 (Linux; Android 10; LM-Q710(FGN)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36"
    ]

    links = ["https://www.twitter.com", "https://www.google.com"]

    links.each do |link|
      agents.each do |agent|
        puts "#{link} - #{agent} - #{LinkThumbnailer.generate(link, redirect_limit: 5, user_agent: agent).description}"
      end
    end
gottfrois commented 4 years ago

I haven't tested myself but I don't see an easy solution for this. Is there a way to transform twitter-like URLs into their legacy website which supports non-js agents? Otherwise, it would mean adding JS rendering capability to the gem which is quite complex and adds a lot of deps.

msmiller commented 4 years ago

There may be a way to trick their nojs_router but I tied a few things and it seems to reject any attempts.

gottfrois commented 4 years ago

When you get the twitter response:

We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?

They don't include a redirection link in some custom HTTP header maybe? Is there a way to know the legacy URLs before and use them instead?

msmiller commented 4 years ago

It does provide a URL, but I had no luck trying to spoof it. I didn't really put much time into it though.

  <noscript>
    <form action="https://mobile.twitter.com/i/nojs_router?path=%2FAwardsDarwin%2Fstatus%2F1288731048868708352" method="POST" style="background-color: #fff; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 9999;">
      <div style="font-size: 18px; font-family: Helvetica,sans-serif; line-height: 24px; margin: 10%; width: 80%;">
        <p>We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?</p>
        <p style="margin: 20px 0;">
          <button type="submit" style="background-color: #1da1f2; border-radius: 100px; border: none; box-shadow: none; color: #fff; cursor: pointer; font-size: 14px; font-weight: bold; line-height: 20px; padding: 6px 16px;">Yes</button>
        </p>
      </div>
    </form>
  </noscript>
gottfrois commented 1 year ago

I'm closing this issue as I have no plans to add JS rendering support to this gem at the moment.