Graphs randomly fail to load with RRDCached sidecar container

[hugalafutro]

Behaviour

Graphs load as red block randomly

Steps to reproduce this issue

1. use docker librenms install
2. refresh a page with any number of graphs (for me the most obvious culprits are the "Top Devices" widgets on dashboard, but it happens everywhere
3. observe as every refresh random number of the graphs doesn't load and look like red square, while some of the previously not loaded will load and display correct data

Expected behaviour

The graphs should all load

Actual behaviour

The graphs randomly do not load

Configuration

• Docker version (type docker --version) : Docker version 19.03.12, build 48a66213fe
• Docker compose version if applicable (type docker-compose --version) : docker-compose version 1.25.0, build unknown
• Platform (Debian 9, Ubuntu 18.04, ...) : Ubuntu 20.04
• System info (type uname -a) : Linux nms 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• dokcer-compose.yml and the .env files are literally the same as in the examples folder, I just changed port librenms listens on.

Docker info

Client:
 Debug Mode: false

Server:
 Containers: 21
  Running: 20
  Paused: 0
  Stopped: 1
 Images: 19
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-47-generic
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.661GiB
 Name: nms
 ID: ZYME:SMME:SOUR:BI62:XXOT:LR2H:THH4:FOAJ:IGJC:QXQZ:ZNW3:7RQE
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Logs

Nothing in any of librenms containers logs indicate any error.

Validate:

====================================
Component | Version
--------- | -------
LibreNMS  | 1.66
DB Schema | 2020_08_28_212054_drop_uptime_column_outages (173)
PHP       | 7.3.21
Python    | 3.8.5
MySQL     | 10.4.14-MariaDB-1:10.4.14+maria~focal
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.8
====================================

[OK]    Installed from the official Docker image; no Composer required
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  IPv6 is disabled on your server, you will not be able to add IPv6 devices.
[WARN]  Updates are managed through the official Docker image

There is also a discussion with some screenshots @ https://community.librenms.org/t/problems-with-graphs-red-only-missing-error-drawing-graph/10279