search

khpeek / scraper-compose

Scrapy example project using Tor (through Privoxy) in a Docker Compose multi-container application
10 stars 6 forks source link

error "no such file or directory" #1

Closed lorenzogm closed 7 years ago

lorenzogm commented 7 years ago

Hi @khpeek, I have tried your code and it looks so good.

But I'm getting an error when I run docker-compose up:

scraper_1  | standard_init_linux.go:185: exec user process caused "no such file or directory"

I had to update the ./scraper/requirements.txt: scrapy-fake-useragent==1.0.2 to scrapy-fake-useragent==1.0.2.1 because the build was failing

I'm running Windows 10 and I just did the following:

git clone this-repo
cd this-repo
docker-compose build
docker-compose up

Probably it's just something minor and stupid but I don't know a lot about Docker... Could you help me?

khpeek commented 7 years ago

Hi Lorenzo,

Thanks for pointing out the failing build, I updated the Dockerfile for the scraper service such that it simply contains RUN pip install scrapy scrapy-fake-useragent etc. without specifying specific versions.

As for your error message, it seems like the problem is germane to machines running on Windows - see https://github.com/docker/labs/issues/215. I'm afraid I can't really help you there, the scraper does work for me on Linux and Mac.

Finally, just a few general notes on this example project: it contains a Scrapy extension which periodically changes Tor's IP address after a randomized number of items is scraped. This is intended to confound traffic analysis and prevent Tor exit nodes from getting banned; however, if you're just interested in using Tor, you could use a simpler container such as https://github.com/rdsubhas/docker-tor-privoxy-alpine which does not feature access to Tor's control port.

lorenzogm commented 7 years ago

Thank you for your help khpeek. Finally I'm using linux :)