Grafana installation - volume permissions

petertjmills commented 12 months ago

Due diligence

My actions before raising this issue

I don't know if this is a faasd, grafana, containerd, or basic linux file permissions issue, I've been following the Serverless for everyone else book, but am having an extremely hard time trying to get grafana to use the volume, provided in the docker-compose.yaml file. I have the devops pro edition, and noticed the grafana.yaml and the config are different in that in the book it is implied to use 1000 as a user, and in the yaml there is an account named 'grafana'. I've tried to use both creating both a faasd user 1000, and a grafana user with uid 472. I've chown'd and chmod'd the folder in all number of different configurations, and I've chmod'd everything to 777. Grafana shows the same errors in the logs.

Why do you need this?

To use grafana with persistant storage, with faasd.

Who is this for?

Proof of concept for a commercial functions platform.

Expected Behaviour

Grafana creates the required directories in the mounted folder.

Current Behaviour

No files being created, grafana forgets all dashboards on restart.

Sep 09 14:02:14 ubuntu-4gb-nbg1-2 openfaas:grafana[335567]: logger=provisioning.datasources t=2023-09-09T14:02:14.366824422Z level=error msg="can't read datasource provisioning files from directory" path=/etc/grafana/provisioning/datasources error="open /etc/grafana/provisioning/datasources: no such file or directory"
Sep 09 14:02:14 ubuntu-4gb-nbg1-2 openfaas:grafana[335567]: logger=provisioning.plugins t=2023-09-09T14:02:14.367041377Z level=error msg="Failed to read plugin provisioning files from directory" path=/etc/grafana/provisioning/plugins error="open /etc/grafana/provisioning/plugins: no such file or directory"
Sep 09 14:02:14 ubuntu-4gb-nbg1-2 openfaas:grafana[335567]: logger=provisioning.notifiers t=2023-09-09T14:02:14.367231816Z level=error msg="Can't read alert notification provisioning files from directory" path=/etc/grafana/provisioning/notifiers error="open /etc/grafana/provisioning/notifiers: no such file or directory"
Sep 09 14:02:14 ubuntu-4gb-nbg1-2 openfaas:grafana[335567]: logger=provisioning.alerting t=2023-09-09T14:02:14.367488388Z level=error msg="can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error="open /etc/grafana/provisioning/alerting: no such file or directory"
Sep 09 14:02:14 ubuntu-4gb-nbg1-2 openfaas:grafana[335567]: logger=provisioning.dashboard t=2023-09-09T14:02:14.368004856Z level=error msg="can't read dashboard provisioning files from directory" path=/etc/grafana/provisioning/dashboards error="open /etc/grafana/provisioning/dashboards: no such file or directory"
Sep 09 14:02:14 ubuntu-4gb-nbg1-2 openfaas:grafana[335567]: logger=provisioning.dashboard t=2023-09-09T14:02:14.378494412Z level=error msg="can't read dashboard provisioning files from directory" path=/etc/grafana/provisioning/dashboards error="open /etc/grafana/provisioning/dashboards: no such file or directory"

List All Possible Solutions and Workarounds

Which Solution Do You Recommend?

Steps to Reproduce (for bugs)

git clone https://github.com/openfaas/faasd --depth=1
cd faasd
./hack/install.sh
cd /var/lib/faasd
mkdir grafana
chown -R 1000:1000 /var/lib/faasd/grafana # or 472:472, or faasd:faasd, or grafana:grafana
groupadd --gid 1000 faasd
useradd --uid 1000 --system \
  --no-create-home \
  --gid 1000 faasd
nano docker-compose.yaml
# grafana:
#   image: docker.io/grafana/grafana:latest
#   user: "1000" #or 472, or grafana, or root
#   environment:
#     - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin 
#     - GF_AUTH_ANONYMOUS_ENABLED=true
#     - GF_AUTH_BASIC_ENABLED=false
#   volumes:
#     # we assume cwd == /var/lib/faasd
#     - type: bind
#        source: ./grafana/
#        target: /etc/grafana/provisioning/
#   cap_add:
#     - CAP_NET_RAW 
#   depends_on:
#     - prometheus 
#   ports:
#     - "3000:3000"
sudo systemctl daemon-reload && sudo systemctl restart faasd

Your Environment

OS and architecture:
- Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-156-generic x86_64)

go version
n/a

containerd -version
containerd github.com/containerd/containerd v1.7.0 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d

uname -a
Linux ubuntu-4gb-nbg1-2 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

faasd version
faasd version: 0.18.0   commit: 53670e285466b60e24e9d0ade60e5fd5430d978e

petertjmills commented 12 months ago

The permissions wasn't the problem. It was my misunderstanding on grafana. I misunderstood what the docker-compose file was binding. I, completely and wrongfully, assumed /etc/grafana/provisioning was how it persisted data, but it's not, it's how it provisions files. In retrospect I should have seen this.

For persistent storage:

    volumes:
      # we assume cwd == /var/lib/faasd
      - type: bind
        source: ./grafana/
        target: /var/lib/grafana/

Apologies for opening an unnecessary issue. This may make a good addition to the book, for reference.

alexellis commented 11 months ago

Hi @petertjmills no worries.

What did you change vs. the eBook / DevOps Pro tier files?

Alex

openfaas / faasd