twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.79k stars 2.72k forks source link

[QUESTION] Integration with Flask web framework #240

Open aaeissa opened 6 years ago

aaeissa commented 6 years ago

I was aiming to provide a simple UI in an existing Flask app I have for performing a search with Twint. The view function is quite simple - I get user input (a Twitter username) and run the search:

@app.route('/search')
def search_twitter():        
    username = request.form['username']
    config = twint.Config()
    config.Username = username
    config.Limit = 100
    config.Store_object = True

    # This is the async search
    twint.run.Search(config)

    tweets = twint.output.tweets_object
    return render_template('twitter_results.html', tweets=tweets) . 

When I submit the form, i get RuntimeError: There is no current event loop in thread 'Thread-2'.

At first I wasn't exactly sure what was going on under the hood with the twint.run.Search() method, but I did some digging and saw a lot of async code being used for searching. Some further reading seemed to suggest that Flask was not compatible with async code.

I tested this by including the following snippet at the start of my main app.py file, which immediately threw the same RuntimeError.

import asyncio 

loop = asyncio.get_event_loop()
app = Flask(__name__) 

I don't want to let go of using this library just yet, nor do I want to scrap the rest of my app. Since the code obviously works fine in a python interpreter or stand alone script, I've considered some hack-ish workarounds like calling a separate script with os.process(), but then I know I'm introducing security and performance concerns.

I was wondering if you had any ideas on integrating with Flask or generally how one might perform searches through a GUI?

ryanakuhl commented 6 years ago

EDIT: I'm finally up and running with Quart instead of Flask and Hypercorn instead of Gunicorn in the Google App environment. I inadvertently was running Twint 1.3 instead of the latest version, which caused a lot of unnecessary headaches. Another thing that caught me up was trying to get the function to run within app = Quart(name). I put it above and things went smoother from there.

ikario404 commented 5 years ago

Thanks for info. I was hesitating to use Quart too but I will definitely take a look now.

devanshuDesai commented 5 years ago

For anyone who doesn't want to go through the hassles of moving to Quart. I was able to fix this exact same error by changing line 213 in twint/run.py from:

get_event_loop().run_until_complete(Twint(config).main(callback)) to: loop = new_event_loop() set_event_loop(loop) loop.run_until_complete(Twint(config).main(callback))

Also, don't forget to add relevant imports at the top of the file: from asyncio import new_event_loop, set_event_loop

I have tested this with Flask and works as expected.

aaeissa commented 5 years ago

@devanshuDesai Thanks for the idea - did you make any additional changes? I implemented your solution and got NameError: name 'new_event_loop' is not defined

devanshuDesai commented 5 years ago

Sorry. Forgot to add the the imports. I have edited my message. You will need to add: from asyncio import new_event_loop, set_event_loop

70shakeel commented 4 years ago

For anyone who doesn't want to go through the hassles of moving to Quart. I was able to fix this exact same error by changing line 213 in twint/run.py from:

get_event_loop().run_until_complete(Twint(config).main(callback)) to: loop = new_event_loop() set_event_loop(loop) loop.run_until_complete(Twint(config).main(callback))

Also, don't forget to add relevant imports at the top of the file: from asyncio import new_event_loop, set_event_loop

I have tested this with Flask and works as expected.

Hi! Can you share your project where you have integrated flask to work with TWINT?

neon-ninja commented 4 years ago

Hi @70shakeel - I built a web based UI to twint with bottle (similar to flask) - you can see it at https://github.com/UoA-eResearch/twint-ui

yutaroishiwata commented 3 years ago

@neon-ninja Is this still working ??

neon-ninja commented 3 years ago

@neon-ninja Is this still working ??

Probably - try run it yourself