[QUESTION] Integration with Flask web framework

[aaeissa]

I was aiming to provide a simple UI in an existing Flask app I have for performing a search with Twint. The view function is quite simple - I get user input (a Twitter username) and run the search:

@app.route('/search')
def search_twitter():        
    username = request.form['username']
    config = twint.Config()
    config.Username = username
    config.Limit = 100
    config.Store_object = True

    # This is the async search
    twint.run.Search(config)

    tweets = twint.output.tweets_object
    return render_template('twitter_results.html', tweets=tweets) . 

When I submit the form, i get RuntimeError: There is no current event loop in thread 'Thread-2'.

At first I wasn't exactly sure what was going on under the hood with the twint.run.Search() method, but I did some digging and saw a lot of async code being used for searching. Some further reading seemed to suggest that Flask was not compatible with async code.

I tested this by including the following snippet at the start of my main app.py file, which immediately threw the same RuntimeError.

import asyncio 

loop = asyncio.get_event_loop()
app = Flask(__name__) 

I don't want to let go of using this library just yet, nor do I want to scrap the rest of my app. Since the code obviously works fine in a python interpreter or stand alone script, I've considered some hack-ish workarounds like calling a separate script with os.process(), but then I know I'm introducing security and performance concerns.

I was wondering if you had any ideas on integrating with Flask or generally how one might perform searches through a GUI?

[ryanakuhl]

EDIT: I'm finally up and running with Quart instead of Flask and Hypercorn instead of Gunicorn in the Google App environment. I inadvertently was running Twint 1.3 instead of the latest version, which caused a lot of unnecessary headaches. Another thing that caught me up was trying to get the function to run within app = Quart(name). I put it above and things went smoother from there.

[ikario404]

Thanks for info. I was hesitating to use Quart too but I will definitely take a look now.

[devanshuDesai]

For anyone who doesn't want to go through the hassles of moving to Quart. I was able to fix this exact same error by changing line 213 in twint/run.py from:

get_event_loop().run_until_complete(Twint(config).main(callback)) to: loop = new_event_loop() set_event_loop(loop) loop.run_until_complete(Twint(config).main(callback))

Also, don't forget to add relevant imports at the top of the file: from asyncio import new_event_loop, set_event_loop

I have tested this with Flask and works as expected.

[aaeissa]

@devanshuDesai Thanks for the idea - did you make any additional changes? I implemented your solution and got NameError: name 'new_event_loop' is not defined

[devanshuDesai]

Sorry. Forgot to add the the imports. I have edited my message. You will need to add: from asyncio import new_event_loop, set_event_loop

[70shakeel]

For anyone who doesn't want to go through the hassles of moving to Quart. I was able to fix this exact same error by changing line 213 in twint/run.py from:

get_event_loop().run_until_complete(Twint(config).main(callback)) to: loop = new_event_loop() set_event_loop(loop) loop.run_until_complete(Twint(config).main(callback))

Also, don't forget to add relevant imports at the top of the file: from asyncio import new_event_loop, set_event_loop

I have tested this with Flask and works as expected.

Hi! Can you share your project where you have integrated flask to work with TWINT?

[neon-ninja]

Hi @70shakeel - I built a web based UI to twint with bottle (similar to flask) - you can see it at https://github.com/UoA-eResearch/twint-ui

[yutaroishiwata]

@neon-ninja Is this still working ??

[neon-ninja]

@neon-ninja Is this still working ??

Probably - try run it yourself