Graphs/Services with check periods greater than 1h will return server error 500 from graphite.

I have some services that are being checked every 2h. Any check_interval above 1h will cause this. This happens because the storage schema is set to 2h and the default view in graphite is 1h. Is it better to have a default storage schema of 1h for all services, even if you only check ever 2hrs?

Whisper files are populating as expected.

This is the error in icingaweb:

icingaweb_1       | [Sat Oct 07 22:44:50.240658 2023] [php7:notice] [pid 32132] [client 10.<omitted>] icingaweb2: ERROR - RuntimeException in /usr/share/icingaweb2/modules/graphite/library/vendor/iplx/Http/Client.php:173 with message: The requested URL returned error: 500 Internal Server Error\n
  #0 /usr/share/icingaweb2/modules/graphite/library/vendor/iplx/Http/Client.php(195): iplx\\Http\\Client->executeHandle()\n
  #1 /usr/share/icingaweb2/modules/graphite/library/Graphite/Graphing/GraphiteWebClient.php(83): iplx\\Http\\Client->send()\n
  #2 /usr/share/icingaweb2/modules/graphite/library/Graphite/Web/Widget/GraphImage.php(117): Icinga\\Module\\Graphite\\Graphing\\GraphiteWebClient->request()\n
  #3 /usr/share/icingaweb2/modules/graphite/library/Graphite/Graphing/Chart.php(105): Icinga\\Module\\Graphite\\Web\\Widget\\GraphImage->render()\n
  #4 /usr/share/icingaweb2/modules/graphite/application/controllers/GraphController.php(162): Icinga\\Module\\Graphite\\Graphing\\Chart->serveImage()\n
  #5 /usr/share/icingaweb2/modules/graphite/application/controllers/GraphController.php(86): Icinga\\Module\\Graphite\\Controllers\\GraphController->supplyImage()\n
  #6 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\\Module\\Graphite\\Controllers\\GraphController->serviceAction()\n
  #7 /usr/share/icingaweb2/library/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch()\n
  #8 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\\Web\\Controller\\Dispatcher->dispatch()\n
  #9 /usr/share/icingaweb2/library/Icinga/Application/Web.php(290): Zend_Controller_Front->dispatch()\n#10 /usr/share/icingaweb2/library/Icinga/Application/webrouter.php(105): Icinga\\Application\\Web->dispatch()\n#11 /usr/share/icingaweb2/public/index.php(4): require_once(String)\n
  #12 {main}, referer: http://10.<omitted>:9080/icingadb/services?host.name=NMS

This is the URL that is requested by icingaweb from graphite via render.

http://10.<omitted>:5080/render?from=1696805607&until=1696810407&bgcolor=282e39&fgcolor=ffffff&majorGridLineColor=5c5c5c&minorGridLineColor=282e39&width=967&height=220&hideLegend=&tz=Europe%2FLondon&_salt=1696810407.000&vTitle=Percent&lineMode=connected&drawNullAsZero=false&graphType=line&_ext=whatever.svg&areaAlpha=0.5&areaMode=all&lineWidth=2&title=used&yUnitSystem=none&target=alias%28color%28icinga2.NMS.services.4h_passive-disk4.dummy.perfdata.used.value%2C%20%27%231a7dd7%27%29%2C%20%27value%27%29%20HTTP/1.1

This is the graphite error:

Traceback (most recent call last):
  File "/opt/graphite/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/opt/graphite/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/graphite/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/errors.py", line 101, in new_f
    return f(*args, **kwargs)
  File "/opt/graphite/webapp/graphite/render/views.py", line 157, in renderView
    response = renderViewGraph(graphOptions, requestOptions, data)
  File "/opt/graphite/webapp/graphite/render/views.py", line 177, in renderViewGraph
    image = doImageRender(requestOptions['graphClass'], graphOptions)
  File "/opt/graphite/webapp/graphite/render/views.py", line 606, in doImageRender
    img = graphClass(**graphOptions)
  File "/opt/graphite/webapp/graphite/render/glyph.py", line 578, in __init__
    self.drawGraph(**params)
  File "/opt/graphite/webapp/graphite/render/glyph.py", line 1085, in drawGraph
    self.consolidateDataPoints()
  File "/opt/graphite/webapp/graphite/render/glyph.py", line 1427, in consolidateDataPoints
    bestXStep = numberOfPixels / divisor
ZeroDivisionError: float division by zero

Expected Behavior

I am not sure what the answer is. If you request a 1hr period graph for a service that is checked every 2hrs then what should be done?

Possible Solution

Changing the default period for all of graphite to 1 day fixes the majority of issues, but doesn't solve it for data you check less than once per day.

Is there a way to change the default graphite view per service?

Context

apply Service "passive-disk6" {
  import "generic-service"
  name = "2h_passive-disk6"
  check_interval = 2h
  check_command = "dummy"
  assign where host.vars.checkdisk == "test"
}

graphite storage-schemas.conf (relevant section shown only)

[icinga2_2h]
pattern = ^icinga2.*\.services\.2h_
retentions = 2h:400d

Environment

icinga2 - The Icinga 2 network monitoring daemon (version: v2.14.0)

Copyright (c) 2012-2023 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: Debian GNU/Linux Platform version: 11 (bullseye) Kernel: Linux Kernel version: 4.19.0-16-amd64 Architecture: x86_64

Build information: Compiler: GNU 10.2.1 Build host: buildkitsandbox OpenSSL version: OpenSSL 1.1.1n 15 Mar 2022

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid

Icinga Web 2 Version | 2.11.4 director 1.10.2 Configure icingadb 1.0.2 Configure graphite 1.2.2 Configure incubator 0.20.0 Configure Graphite-Web 1.1.10 (python 3.9)

Icinga / icingaweb2-module-graphite