istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 324 forks source link

redis.exceptions.ResponseError: unknown command 'PFCOUNT' #133

Closed mohit0749 closed 7 years ago

mohit0749 commented 7 years ago

getting this error when i run the following script. python kafka_monitor.py run

Traceback (most recent call last):
  File "kafka_monitor.py", line 559, in <module>
    sys.exit(main())
  File "kafka_monitor.py", line 543, in main
    kafka_monitor.run()
  File "kafka_monitor.py", line 401, in run
    self._main_loop()
  File "kafka_monitor.py", line 260, in _main_loop
    self._dump_stats()
  File "kafka_monitor.py", line 376, in _dump_stats
    extras[final] = self.stats_dict['total'][key].value()
  File "/usr/local/lib/python2.7/dist-packages/scutils/stats_collector.py", line 517, in value
    return self.redis_conn.execute_command("PFCOUNT", self.final_key)
  File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 573, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 585, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 582, in read_response
    raise response
redis.exceptions.ResponseError: unknown command 'PFCOUNT'
madisonb commented 7 years ago

According to the documentation, PFCOUNT has been avaiable in Redis since 2.8.9 (released April 2014). I would recommend running at least 3.0, but older versions without PFCOUNT may still work if you set the stats-based flags to False.

Kafka Monitor:

Redis Monitor:

Crawler:

This should disable stats collection on all the different components, thus remove the need to execute the redis command that is failing.

If this answers your question please close this ticket.

mohit0749 commented 7 years ago

thnks it solved my problem.