KeyError: 'scheduler'

[karpitsky]

Hello,

I've got the following error message:

# mrq-worker --scheduler

  File "/usr/local/bin/mrq-worker", line 11, in <module>
    load_entry_point('mrq', 'console_scripts', 'mrq-worker')()
  File "/usr/local/src/mrq/mrq/bin/mrq_worker.py", line 53, in main
    w.work()
  File "/usr/local/src/mrq/mrq/worker.py", line 456, in work
    self.work_loop(max_jobs=self.max_jobs, max_time=self.max_time)
  File "/usr/local/src/mrq/mrq/worker.py", line 510, in work_loop
    if self.config["scheduler"] and not self.greenlets["scheduler"]:
KeyError: 'scheduler'
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/src/mrq/mrq/processes.py", line 128, in watch
    self.watch_processes()
  File "/usr/local/src/mrq/mrq/processes.py", line 135, in watch_processes
    self.watch_process(process)
  File "/usr/local/src/mrq/mrq/processes.py", line 146, in watch_process
    status = process["psutil"].status()
  File "/usr/local/lib/python2.7/site-packages/psutil/__init__.py", line 669, in status
    return self._proc.status()
  File "/usr/local/lib/python2.7/site-packages/psutil/_pslinux.py", line 1470, in wrapper
    raise NoSuchProcess(self.pid, self._name)
NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=20)

Do you have an idea how to fix it?

[karpitsky]

I think I've found a solution for my case. My config has QUEUES = ("default",), so when I run a scheduler mrq-worker --scheduler it also runs a worker for the default queue. If this worker gets a job it will fail with the error KeyError 'scheduler', see the value of self.config below. I've removed the "default" queue from my config and it works.

self.config on the line 510 for the worker

{
'scheduler_interval': 60,
'add_network_latency': '0',
'no_import_patch': False,
'worker_class': 'mrq.worker.Worker',
'greenlets': 100,
'redis_max_connections': 1000,
'tasks': {},
'trace_greenlets': False,
'default_job_cancel_ttl': 86400,
'report_interval': 10,
'redis_timeout': 30,
'log_handler': 'mrq.logger.MongoHandler',
'log_level': 'DEBUG',
'mongodb_jobs': 'mongodb://127.0.0.1:30000/mrq?ssl=true&ssl_ca_certs=/opt/ssl/cacert.pem',
'print_mongodb': False,
'redis': 'redis://127.0.0.1:6379/2',
'max_memory': 0,
'version': False,
'redis_prefix': 'mrq',
'config': None,
'profile': False,
'scheduled_tasks': {},
'scheduler_tasks': [***],
'log_handlers': {},
'default_job_ttl': 15552000,
'max_jobs': 0,
'use_large_job_ids': True,
'default_job_timeout': 86400,
'project_dir': '/usr/src/app',
'report_file': '',
'processes': 1,
'trace_memory_type': '',
'name': None,
'trace_io': True,
'default_job_retry_delay': 3,
'default_job_abort_ttl': 86400,
'default_job_max_retries': 3,
'trace_memory_output_dir': 'memory_traces',
'quiet': False,
'mongodb_logs': '1',
'trace_memory': False,
'dequeue_strategy': 'sequential',
'paused_queues_refresh_interval': 10,
'log_format': '%(asctime)s [%(levelname)s] %(message)s',
'queues': ('default',),
'default_job_result_ttl': 604800
}

self.config on the line 510 for the scheduler

{
'ensure_indexes': False,
'add_network_latency': '0',
'no_import_patch': False,
'worker_class': 'mrq.worker.Worker',
'scheduler_interval': 60,
'greenlets': 100,
'redis_max_connections': 1000,
'worker_id': '',
'trace_greenlets': False,
'default_job_cancel_ttl': 86400,
'dequeue_strategy': 'sequential',
'admin_ip': '127.0.0.1',
'scheduled_tasks': {},
'redis_timeout': 30,
'log_handler': 'mrq.logger.MongoHandler',
'log_level': 'DEBUG',
'tasks': {},
'mongodb_jobs': 'mongodb://127.0.0.1:30000/mrq?ssl=true&ssl_ca_certs=/opt/ssl/cacert.pem',
'worker_profile': '',
'print_mongodb': False,
'redis': 'redis://127.0.0.1:6379/2',
'max_memory': 0,
'subqueues_refresh_interval': 60,
'version': False,
'redis_prefix': 'mrq',
'scheduler_tasks': [***],
'config': None,
'profile': False,
'report_interval': 10,
'max_jobs': 0,
'task_blacklist': '',
'admin_port': 0,
'max_latency': 1.0,
'log_handlers': {},
'default_job_ttl': 15552000,
'agent_id': '',
'scheduler': True,
'use_large_job_ids': True,
'default_job_timeout': 86400,
'project_dir': '/usr/src/app',
'worker_group': '',
'report_file': '',
'processes': 1,
'trace_memory_type': '',
'name': None,
'trace_io': True,
'default_job_retry_delay': 3,
'default_job_abort_ttl': 86400,
'default_job_max_retries': 3,
'trace_memory_output_dir': 'memory_traces',
'local_ip': '***',
'quiet': False,
'mongodb_logs': '1',
'external_ip': None,
'trace_memory': False,
'max_time': 0.0,
'paused_queues_refresh_interval': 10,
'log_format': '%(asctime)s [%(levelname)s] %(message)s',
'queues': ('default',),
'default_job_result_ttl': 604800,
'task_whitelist': ''
}

[sylvinus]

Hi @karpitsky !

Thanks for the report. I can't reproduce the behaviour with the latest master.

I don't see how "scheduler" wouldn't be in self.config. It should at least be False because it's in the config in all cases for mrq-worker: https://github.com/pricingassistant/mrq/blob/master/mrq/config.py#L365

Are you modifying the worker config somewhere maybe?

thanks!

[karpitsky]

Hi @sylvinus

Thanks for your time! I didn't see the error after changing QUEUES = ("default",) to QUEUES = [] I used the config as is, without modifying.

As I don't see the error anymore I close this ticket.