taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 579 forks source link

OSError: [Errno 12] Cannot allocate memory #230

Open lukaspistelak opened 4 years ago

lukaspistelak commented 4 years ago

when i run query_tweets in 100000 loop:

Exception in thread -1: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/twitterscraper/query.py", line 233, in query_tweets pool = Pool(poolsize) File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 1046, in init self._create_worker_process(i) File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 1158, in _create_worker_process w.start() File "/usr/local/lib/python3.6/dist-packages/billiard/process.py", line 124, in start self._popen = self._Popen(self) File "/usr/local/lib/python3.6/dist-packages/billiard/context.py", line 333, in _Popen return Popen(process_obj) File "/usr/local/lib/python3.6/dist-packages/billiard/popen_fork.py", line 24, in init self._launch(process_obj) File "/usr/local/lib/python3.6/dist-packages/billiard/popen_fork.py", line 72, in _launch self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory

`

   list_of_tweets = query_tweets(q_url, limit=99999, begindate=dt.date(2019, 3, 21),poolsize=2)

    #print the retrieved tweets to the screen:  begindate = args.begindate, enddate = args.enddate,poolsize = args.poolsize, lang = args.lang)

#=======================================================================
    for tweet in list_of_tweets:            
       #             
        if(tweet.fullname in self.set_all_unuque_journalist and tweet.fullname!=journalist):            
            output.append(tweet.tweet_id)            

            #tweet.encode('utf-8')

        #print("tweet.fullname in self.set_all_unuque_journalist and tweet.fullname!=journalist",tweet.fullname in self.set_all_unuque_journalist ,tweet.fullname!=journalist)

        print("tweet.fullname,journalist",tweet.fullname,journalist)

    list_of_tweets=None`

EDIT:

i found POSSIBLE sollution:

to use:

maxtasksperchild=

Can somebody super fast improve it? or can i?