search

iron-io / docs

Iron.io Dev Center documentation.
dev.iron.io
30 stars 48 forks source link

Article: How to use Scrapy with IronWorker #93

Closed treeder closed 3 years ago

turian commented 11 years ago

I would love to see this.

turian commented 11 years ago

Or any Python distributed crawler.

paddycarver commented 11 years ago

I'll bump it up in priority. :) Watch this thread--when I clear off some of the more pressing stuff I'm doing, I'll work on this.

paddycarver commented 11 years ago

@turian Can I ask where you're running into difficulty? What would you like to see in an article? Trying to figure out how to approach this that won't end with us writing an example for every web scraper ever, because that's not really tenable.

rajivseelam commented 10 years ago

Installing scrapy and lxml should do for now. When I test my workers locally they work, when i upload it the build fails with errors:

 No package 'libffi' found

removing: _configtest.c _configtest.o

c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory

compilation terminated.