search

ziplokk1 / scrapy-amazon-robot-middleware

Scrapy middleware module which uses image parsing to submit a captcha response to amazon.
The Unlicense
11 stars 12 forks source link

Error: Too many open files #2

Open NicholasGarside opened 7 years ago

NicholasGarside commented 7 years ago

Hi,

I get this error when I try to run it with scrapy:

2017-03-08 10:01:23 [twisted] CRITICAL: Traceback (most recent call last): File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks result = g.send(result) File "c:\python27\lib\site-packages\scrapy\crawler.py", line 90, in crawl six.reraise(*exc_info) File "c:\python27\lib\site-packages\scrapy\crawler.py", line 72, in crawl self.engine = self._create_engine() File "c:\python27\lib\site-packages\scrapy\crawler.py", line 97, in _createengine return ExecutionEngine(self, lambda : self.stop()) File "c:\python27\lib\site-packages\scrapy\core\engine.py", line 68, in init self.downloader = downloader_cls(crawler) File "c:\python27\lib\site-packages\scrapy\core\downloader__init.py", line 88, in init self.middleware = DownloaderMiddlewareManager.from_crawler(crawler) File "c:\python27\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler return cls.from_settings(crawler.settings, crawler) File "c:\python27\lib\site-packages\scrapy\middleware.py", line 34, in from_settings mwcls = load_object(clspath) File "c:\python27\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object mod = import_module(module) File "c:\python27\lib\importlib__init__.py", line 37, in import_module import(name) File "c:\scrapes\scrapy-amazon-robot-middleware-master\captchabuster__init__.py", line 180, in images = load_images() File "c:\scrapes\scrapy-amazon-robot-middleware-master\captchabuster\init__.py", line 174, in load_images i = Image.open(os.path.join(letter_dir, img)) File "c:\python27\lib\site-packages\PIL\Image.py", line 2312, in open fp = builtins.open(filename, "rb") IOError: [Errno 24] Too many open files: 'c:\scrapes\scrapy-amazon-robot-middleware-master\captchabuster\iconset\z\981ca6fe34eaf28511810a47f7465c5e.gif'

I'm not sure how I can fix this, I'm using scrapy 1.2.1 with Windows 10

Cheers

ziplokk1 commented 7 years ago

I will take a look. I'm sorry I'm just seeing this. I have been looking for a better way to do the character recognition rather than iterating over image files due to this exact issue. I will get back to you soon with a work around.