istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 324 forks source link

LogFactory rolling log doesnt actually roll #54

Closed madisonb closed 8 years ago

madisonb commented 8 years ago

The LogFactory logger instance doesn't actually roll the log when it hits the max size, completely negating any added benefit of the ConcurrentRotatingFileHandler we use under the hood.

from scutils.log_factory import LogFactory

logger = LogFactory.get_instance(
        json=False,
        stdout=False,
        level='DEBUG',
        name='test',
        dir='/tmp/',
        file='test.log',
        bytes='5MB',
        backups=2
)

while True:
    logger.info("test log")

Should write 3 log files total, 2 backups of ~5MB each and the main log file. It does no such thing and continues to write to the same log file, growing as big as the system can handle.

madisonb commented 8 years ago

This is due to an incorrect use of the parameter bytes which is not a string but a numeric value indicating the number of bytes. Docs need to be updated to reflect this.