vladkens / twscrape

2024! X / Twitter API scrapper with authorization support. Allows you to scrape search results, User's profiles (followers/following), Tweets (favoriters/retweeters) and more.
https://pypi.org/project/twscrape/
MIT License
1.12k stars 133 forks source link

Error when try break for iteration #30

Closed carlosdcuba1 closed 9 months ago

carlosdcuba1 commented 1 year ago

Hi, I got this error when I try to break a for iteration:

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/home/carlos/.local/lib/python3.11/site-packages/aiosqlite/core.py", line 109, in run
    get_loop(future).call_soon_threadsafe(set_result, future, result)
  File "/usr/lib/python3.11/asyncio/base_events.py", line 806, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/home/carlos/.local/lib/python3.11/site-packages/aiosqlite/core.py", line 117, in run
    get_loop(future).call_soon_threadsafe(set_exception, future, e)
  File "/usr/lib/python3.11/asyncio/base_events.py", line 806, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object QueueClient.__aexit__ at 0x7fee85f73680>
Traceback (most recent call last):
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/queue_client.py", line 41, in __aexit__
    await self._close_ctx()
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/queue_client.py", line 51, in _close_ctx
    await self.pool.unlock(ctx.acc.username, self.queue, ctx.req_count)
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/accounts_pool.py", line 181, in unlock
    await execute(self._db_file, qs, {"username": username})
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/db.py", line 17, in wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/db.py", line 123, in execute
    async with DB(db_path) as db:
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/db.py", line 104, in __aenter__
    await check_version()
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/db.py", line 38, in check_version
    ver = await get_sqlite_version()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/carlos/.local/lib/python3.11/site-packages/twscrape/db.py", line 31, in get_sqlite_version
    async with aiosqlite.connect(":memory:") as db:
  File "/home/carlos/.local/lib/python3.11/site-packages/aiosqlite/core.py", line 153, in __aexit__
    await self.close()
  File "/home/carlos/.local/lib/python3.11/site-packages/aiosqlite/core.py", line 171, in close
    await self._execute(self._conn.close)
  File "/home/carlos/.local/lib/python3.11/site-packages/aiosqlite/core.py", line 125, in _execute
    future = asyncio.get_event_loop().create_future()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/events.py", line 677, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'MainThread'.

Here is my code:

try:
    async with aclosing(api.search(query, limit=limit_tweets)) as gen:
        async for tweet in gen:
            if tweet.id < 123:
                break    
except Exception as err:
    print('ERROR downloading')

Thanks for all.

vladkens commented 1 year ago

Hi, @carlosdcuba1. Can you provide example how you exactly run this lines:

try:
    async with aclosing(api.search(query, limit=limit_tweets)) as gen:
        async for tweet in gen:
            if tweet.id < 123:
                break    
except Exception as err:
    print('ERROR downloading')

All asyncio code should be run by asyncio.run

carlosdcuba1 commented 1 year ago

`import asyncio from twscrape import AccountsPool, API from contextlib import aclosing

class TwScrap:

async def get_user_timeline_test(self, user_id, last_tweet=0):  
    pool = AccountsPool()
    api = API(pool)
    list_tweets = []
    limit_tweets = -1 if last_tweet != 0 else 50
    more_tweets = -1 if last_tweet == 0 else 50
    try:
        async with aclosing(api.user_tweets_and_replies(user_id, limit=limit_tweets)) as gen:
            async for tweet in gen:
                if tweet.user.id == user_id:
                    list_tweets.append(tweet)                   
                    if tweet.id < last_tweet: 
                        more_tweets -= 1          
                if more_tweets <= 0 and limit_tweets == -1:
                    break    
    except Exception as err:
        print('ERROR downloading data for user:', user_id, err, flush=True)
    return list_tweets

user=44196397 #elon musk tws=TwScrap() tweets=asyncio.run(tws.get_user_timeline_test(user, 1676784259359490048)) print(tweets) exit() `

Thanks for all

github-actions[bot] commented 9 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 9 months ago

This issue was closed because it has been stalled for 5 days with no activity.