Closed Paolo454 closed 1 year ago
By default the gem passes its own UserAgent which is probably used by some website to check whether JS is available or not. Can you try changing the user agent to something like Chrome or Firefox would use?
https://github.com/gottfrois/link_thumbnailer/blob/master/lib/link_thumbnailer/configuration.rb#L42
You can change the config globally or per request.
Setting the agent doesn't seem to help:
2.6.3 :008 > object = LinkThumbnailer.generate('https://twitter.com/elonmusk/status/1284307886319919104', user_agent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36')
ETHON: started MULTI
ETHON: performed MULTI
=> #<LinkThumbnailer::Models::Website:0x00007fafaad41910 @images=[], @videos=[], @url=#<URI::HTTPS https://twitter.com/elonmusk/status/1284307886319919104>, @title="", @description="We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?", @favicon="https://abs.twimg.com/responsive-web/web/icon-svg.9e211f64.svg">
Hi, thanks for raising this issue, I'm seeing the same. I ran through the different agents found here: https://www.whatismybrowser.com/guides/the-latest-user-agent/chrome
For each, Twitter returns We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?
Google returns nil for the description
@gottfrois do you see the same?
agents = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36",
"Mozilla/5.0 (iPhone; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/83.0.4147.71 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (iPad; CPU OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/83.0.4147.71 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (iPod; CPU iPhone OS 13_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/83.0.4147.71 Mobile/15E148 Safari/604.1",
"Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; SM-A205U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; SM-A102U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; SM-G960U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; SM-N960U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; LM-Q720) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; LM-X420) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36",
"Mozilla/5.0 (Linux; Android 10; LM-Q710(FGN)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36"
]
links = ["https://www.twitter.com", "https://www.google.com"]
links.each do |link|
agents.each do |agent|
puts "#{link} - #{agent} - #{LinkThumbnailer.generate(link, redirect_limit: 5, user_agent: agent).description}"
end
end
I haven't tested myself but I don't see an easy solution for this. Is there a way to transform twitter-like URLs into their legacy website which supports non-js agents? Otherwise, it would mean adding JS rendering capability to the gem which is quite complex and adds a lot of deps.
There may be a way to trick their nojs_router
but I tied a few things and it seems to reject any attempts.
When you get the twitter response:
We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?
They don't include a redirection link in some custom HTTP header maybe? Is there a way to know the legacy URLs before and use them instead?
It does provide a URL, but I had no luck trying to spoof it. I didn't really put much time into it though.
<noscript>
<form action="https://mobile.twitter.com/i/nojs_router?path=%2FAwardsDarwin%2Fstatus%2F1288731048868708352" method="POST" style="background-color: #fff; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 9999;">
<div style="font-size: 18px; font-family: Helvetica,sans-serif; line-height: 24px; margin: 10%; width: 80%;">
<p>We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?</p>
<p style="margin: 20px 0;">
<button type="submit" style="background-color: #1da1f2; border-radius: 100px; border: none; box-shadow: none; color: #fff; cursor: pointer; font-size: 14px; font-weight: bold; line-height: 20px; padding: 6px 16px;">Yes</button>
</p>
</div>
</form>
</noscript>
I'm closing this issue as I have no plans to add JS rendering support to this gem at the moment.
Hello. If I try to use link_thumbnailer with a Twitter link I get this error:
"We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?
I think I get this error because Twitter remove legacy version from 1 June 2020. I get this error in the Demo version as well.
How can I solve it? Thank you