Why I am getting other than english language tweets??

taspinar / twitterscraper

Scrape Twitter for Tweets

MIT License

2.39k stars 581 forks source link

Why I am getting other than english language tweets?? #308

Open VAC2018 opened 4 years ago

VAC2018 commented 4 years ago

Even after mentioning lang as english I am getting Hindi tweets

lapp0 commented 4 years ago

What command are you using to run?

erb13020 commented 3 years ago

I haven't used the lang attribute yet but instead have been using the langdetect library. It looks like something kind of like this.

`from langdetect import detect

def detector(s): try: return detect(s) except: None

tweets = query_tweets(query, begindate=begin_date, enddate=end_date, poolsize=1)

df = pd.DataFrame(t.__dict__ for t in tweets)

df['lang'] = df['text'].apply(lambda x: detector(x))
df = df[df['lang'] == 'en']

Sorry for the formatting