search

imperatrona / twitter-scraper

Scrape the Twitter frontend API without authentication with Golang.
MIT License
86 stars 12 forks source link

Get real URLs from t.co links in profile bio #22

Open thewh1teagle opened 5 days ago

thewh1teagle commented 5 days ago

While fetching followers or following, the user profile includes a bio, which contains the user's description. When the bio has URLs, they are often shortened to t.co links instead of displaying the full URLs. Typically, twitter provides the expanded URLs in other fields in the response, but I couldn't find any related field in the profile struct. Could you check if it's possible to add the missing metadata, if such fields exist?

Thanks!

P.S: I'm pretty sure I used that fields previously in the library x-api-rs. the metadata was stored in legacy object field.

cmj commented 5 days ago

Just adding to this, it seems they are now expanded in a character range of indices for account description (bio).

Example for CNN: https://0x0.st/X6Of.json

thewh1teagle commented 5 days ago

@cmj

Exactly. where did you get this json from? Does twitter scraper get the same response internally?

cmj commented 5 days ago

This is from the v1.1 users/lookup.json, which is the basic user lookup. I'm not seeing this internally, but I'm just noticing it is possible to expand.

You can play with it using this script https://github.com/cmj/twitter-tools/blob/main/user.sh