search

taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 581 forks source link

Twitter Javascript Update #281

Closed tpsvld closed 4 years ago

tpsvld commented 4 years ago

Hello Twitter has updated their software so that only when javascript is enabled it shows the data. This will make any attempt to scrape over HTTP requests not possible.

image

I guess the project is dead.

youssefsaqr commented 4 years ago

I have just used the module one day ago and it is retrieving all the data without issues I don't think the project is dead.

tpsvld commented 4 years ago

Not sure what you are downloading lol but you should look into it, you might also check that all the class and ID from the HTML DIVs are different and look compleatly random.

I guess it wouldn't be long until twitter took seriously the scrapping issue, especially when a US court provided legal coverage for scrapping with the ruling of "hiQ Labs, Inc. v. LinkedIn Corp". (But anyway i was hoping to do this for a university project)

Plus you know how many "progressive" governments police units are working with twitter due to alleged "misinformation campaigns" on corona-virus. They are literally going full police nazi in a witch hunt mode to see who is behind twitter campaigns. (in many cases to censor left wing government dissident points of view )

So yeah, expect twitter making near impossible to get information without them tightly regulating according to their totalitarian policy in their API

lapp0 commented 4 years ago

origin/master has a fix from https://github.com/taspinar/twitterscraper/pull/304

There is also a js implementation in PR https://github.com/taspinar/twitterscraper/pull/302