arachnys / cabot

Self-hosted, easily-deployable monitoring and alerts service - like a lightweight PagerDuty
MIT License
5.59k stars 593 forks source link

`Unhandled error. Please look at logs.` while deleting an instance #635

Open thomasleveil opened 6 years ago

thomasleveil commented 6 years ago

I have this error consistently when trying to delete an instance check (using Cabot 0.11.13)

cabot: 0.11.13 url: http://xxxxxxxxxxx/instance/delete/7/ result: Unhandled error. Please look at logs.

log:

ERROR 2018-08-09 09:00:39,753 exception 19 140601486752552 Internal Server Error: /instance/delete/7/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_respons
    response = self._get_response(request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/utils/decorators.py", line 67, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/utils/decorators.py", line 63, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/usr/local/lib/python2.7/site-packages/cabot/cabotapp/views.py", line 44, in dispatch
    return super(LoginRequiredMixin, self).dispatch(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/edit.py", line 269, in post
    return self.delete(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/views/generic/edit.py", line 264, in delete
    self.object.delete()
  File "/usr/local/lib/python2.7/site-packages/cabot/cabotapp/models/base.py", line 373, in delete
    self.icmp_status_checks().delete()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 616, in delete
    collector.collect(del_query)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/deletion.py", line 222, in collect
    elif sub_objs:
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __nonzero__
    return type(self).__bool__(self)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 254, in __bool__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/polymorphic/query.py", line 48, in _polymorphic_iterator
    real_results = queryset._get_real_instances(base_result_objects)
  File "/usr/local/lib/python2.7/site-packages/polymorphic/query.py", line 366, in _get_real_instances
    real_concrete_class = base_object.get_real_instance_class()
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 96, in get_real_instance_class
    and not issubclass(model, self.__class__._meta.proxy_for_model):
TypeError: issubclass() arg 2 must be a class or tuple of classes

db:

-- select * from cabotapp_instance where id=7;
id  name    alerts_enabled  last_alert_sent email_alert hipchat_alert   sms_alert   telephone_alert overall_status  old_overall_status  hackpad_id  runbook_link    address
7   IPO True    (null)  False   True    False   False   ERROR   ERROR   (null)      10.0.71.254

-- select sc.* from cabotapp_statuscheck sc inner join cabotapp_instance_status_checks isc ON sc.id=isc.statuscheck_id where isc.instance_id=7;
id  name    active  importance  frequency   debounce    calculated_status   last_run    cached_health   metric  check_type  value   expected_num_hosts  allowed_num_failures    endpoint    username    password    text_match  status_code timeout verify_ssl_certificate  max_queued_build_time   created_by_id   polymorphic_ctype_id
67  Default Ping Check for IPO  True    ERROR   5   1   failing 09/08/2018 11:27:15 -1,-1,-1,-1,-1,-1,-1,-1,-1,-1   (null)  (null)  (null)  0   0   (null)  (null)  (null)  (null)  200 30  True    (null)  (null)  22

-- 
select * from cabotapp_jenkinsstatuscheck WHERE statuscheck_ptr_id=67;
statuscheck_ptr_id  jenkins_config_id
67  1
thomasleveil commented 6 years ago

Note: if I manually delete the record from table cabotapp_jenkinsstatuscheck, there is no more error

dankonino commented 5 years ago

hi @thomasleveil please what is the way to check out the logs from cabot please?. I saw the same message but not sure where to look, when checking out the code I didn't find the location of them okay I think I got them /var/log/syslog

thomasleveil commented 5 years ago

@dankonino, I deployed my Cabot instance using the provided docker-compose.yml file. I was able to get the log with this command :

docker-compose logs web

You might also find interesting log entries for the worker service :

docker-compose logs worker