search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.17k stars 323 forks source link

Cannot turn off DEBUG level log #223

Closed YanzhongSu closed 5 years ago

YanzhongSu commented 5 years ago

In Scrapy cluster, I have the following log settings: SC_LOG_LEVEL=INFO, SC_LOG_STDOUT=False

But for some reason, I still get DEBUG level log being printed out in the console. Here is the sample log [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.google.com> [sc-crawler] INFO: crawled processed successfully [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.google.com>

Does anyone how to turn it off?

madisonb commented 5 years ago

Scrapy Cluster's logger is not the same as Scrapy's logger, they are two different things. If you would like to change Scrapy's logger, use the setting LOG_LEVEL as outlined here.

If you would like you adjust the Scrapy Cluster project's logger, you can do that with the log prefixes that are SC_* (so they do not conflict) as found here.

If this resolves your issue please close the ticket.

YanzhongSu commented 5 years ago

Thank you @madisonb . It worked.