Run docker in a non-interactively way

commoncrawl / news-crawl

News crawling with StormCrawler - stores content as WARC

Apache License 2.0

323 stars 35 forks source link

Run docker in a non-interactively way #47

Closed scarcenine closed 2 years ago

scarcenine commented 2 years ago

Hi, I've been trying to run docker in a non-interactively using the following command

docker run -d \
   -p 127.0.0.1:9200:9200 -p 5601:5601 -p 8080:8080 \
   -v .../data/warc:/data/warc \
   -v .../data/elasticsearch:/data/elasticsearch \
   -t newscrawler:1.18 /home/ubuntu/news-crawler/bin/run-crawler.sh

the complete logs can be found here When running it interactively I have no problem. Any idea what the problem is?

sebastian-nagel commented 2 years ago

could you share also the worker.log ? The linked log file is only that of the process launching the topology and then exits.
maybe you want to use docker-compose to launch all components in separate containers? See warc-crawler. Note: the crawler topologies provided there are to consume WARC files. You'd need to write one for real crawling.