Grafana.env not beeing processed

[ingopmeyer]

Hi,

I have a problem that the grafana.env is not processed into the grafana settings. What I want to achive is to give the following parameter to grafana.ini

[Security]
allow_embedding=true

This will enable embedded grafana charts, which I need for Openhab. Per default they are forbidden by X-Frame-Options (for interested people see here https://community.openhab.org/t/influxdb-grafana-persistence-and-graphing/13761/751).

As stated in https://sensorsiot.github.io/IOTstack/Containers/Grafana/ I need to use the grafana.env for IOTstack. First thing I noticed is that the file does not exists. I had to create the grafana directory inside ~/IOTstack/services/ and then the file grafana.env. The content is then:

GF_SECURITY_ALLOW_EMBEDDING=true

But after restarting via

$ cd ~/IOTstack
$ docker-compose stop grafana
$ docker-compose up -d

it will not change anything in grafana. You can control if the setting is set using the admin console for grafana http://<your-iotstack-ip>:3000/admin/settings where I still see "allow_embedding" is false.

Is this broken somehow or have I done something wrong?

Thank in advance ~Ingo

[ingopmeyer]

Removing the container via portainer-ce and running

docker-compose build
docker-compose up -d

does not fixed it like recommended in https://github.com/gcgarner/IOTstack/issues/131

[Paraphraser]

Where to begin...

OK. Forget about grafana.ini because you can't get to it (at least, not easily). The mechanism, as you've noted, is environment variables.

Under "old menu", a container's chunk in docker-compose.yml pointed to an environment file in the container's ./services directory.

Under the new menu, environment files have been deprecated in favour of shoehorning all of this into docker-compose.yml.

IMO, this is a retrograde step because it adds clutter to docker-compose.yml and clutter is the enemy of comprehension.

To wrap your brain around the change, try these commands somewhere other than on your RPi running IOTstack (I'm doing it on my Mac):

$ mkdir fred
$ cd fred
$ git clone https://github.com/SensorsIot/IOTstack.git IOTstack
$ cd IOTstack/.templates/grafana/
$ cp service.yml new-service.yml 
$ git checkout old-menu
$ diff -y service.yml new-service.yml

"fred" is just "some junk directory you can throw away later" so don't get hung up on that name. Then, in words:

• get a fresh clone of SensorsIot/IOTstack (new menu is implied)
• go into the grafana template
• make a copy of the service definition (again, the "new menu" is implied)
• switch to the old menu
• compare the old and new service definitions

You'll note that the env_file definition went away and was replaced by an environment section with all the keys.

Next, focus on the fact that we are looking at the template. When you run the menu (old or new) and choose a service:

1. The service.yml in the .templates folder gets copied to the services directory. Using Grafana as the example, it's the equivalent of:

   $ cd ~/IOTstack
   $ mkdir -p services/grafana
   $ cp .templates/grafana/service.yml services/grafana/

2. All the individual service.yml files in the various subdirectories of the services directory are concatenated to form docker-compose.yml.

So, if you have run the menu (old or new) and selected grafana during the menu run but you don't have a services/grafana folder then you have a significant problem. You really should not have had to resort to creating services/grafana by hand.

Yes, I can easily see why following the now out-of-date doco on either gcgarner/IOTstack or SensorsIot/IOTstack might've led you to believe that you needed an environment file.

That's something @slyke really needs to fix so that the documentation reflects what the menu actually does.

But creating an environment file (which doesn't work because of the change to the services structure) is a long way from also having to create services/grafana. I don't want to blow this out of proportion but if you find the same pattern with other services then you should probably consider starting over.

Anyway, to get back to your original question, you've got two choices:

• The old way (environment file)
• The new way (environment variables).

Personally, I prefer environment files but it's your system. If you don't want to bother with the menu then, on your live system:

1. Clobber grafana properly:

   $ cd ~/IOTstack
   $ docker-compose stop grafana
   $ docker-compose rm -f grafana

   stop just stops the container. The next up will just start the stopped container again. If you want a clean slate where you can be certain things like environment files have been processed, you need to remove the stopped container too. This is what down does, save that it does it for all running containers.

2. Mimic what the menu will do to set up the service:

   $ mkdir -p services/grafana
   $ cp .templates/grafana/service.yml services/grafana/

   Yes, I know you already have services/grafana. I'm just being precise. The command is non-destructive if the path already exists.

3. Edit services/grafana/service.yml to be what you want. That might be zapping all the environment variables and reinstating the environment file, or just going with the environment variables.

4. Do a copy and paste of all of services/grafana/service.yml into docker-compose.yml, replacing the old grafana chunk.

5. Give it a whirl:

   $ docker-compose up -d grafana

Compared with running the menu, that might seem like a lot of hard work but you wind up understanding what the heck is going on, and you are then much better placed to diagnose and fix problems.

The reason for making the changes in both service.yml and docker-compose.yml is a small amount of insurance against the next time you run the menu. Providing you don't cause the grafana service to be re-pulled from its template, your settings will persist when the service.yml files are swept together to form the new docker-compose.yml.

[Slyke]

Just as an FYI, it's still possible to use env files with override: https://sensorsiot.github.io/IOTstack/Custom/

compose-override.yml:

services:
  grafana:
    environment: # Docker will complain if this is set to null. I'll update the YAML merger to remove keys that have null
    - "null" # For now putting this here will still allow it to build and run
    env_file: './services/grafana/grafana.env'

[mvadu]

You can use the custom grafana.ini using the bind files. You might have to manually edit the compose file or grafana template to add an additional volume, and map out your local path to grafana.ini path.

This is my currently working compose file with all configurations contained within a ini file, which you can use with stand alone grafana instance as well as in docker.

  grafana:
    container_name: grafana
    image: grafana/grafana
    restart: unless-stopped
    ports:
    - "3000:3000"
    volumes:
    - ./volumes/grafana/data:/var/lib/grafana
    - ./volumes/grafana/log:/var/log/grafana
    - ./volumes/grafana/config/custom.ini:/etc/grafana/grafana.ini # refer https://github.com/grafana/grafana/blob/master/conf/sample.ini, you just need to place your specific non default values in this file.

[gknops]

Is it me or does the new menu system feel misguided? Overrides that one has to remember to remove when not including a service feels rather a step backwards compared to editing files in a service specific directory.

And for someone upgrading from the old system, NOTHING about any of this is intuitive.

[Paraphraser]

Then don't upgrade. I don't mean that in any snippy or snarky sense. I mean it as honest advice.

I haven't upgraded. I'm still on old menu. git checkout old-menu. Slyke is working on a new-new menu so, honestly, I don't see any point in the "interim" new menu. I'll evaluate new-new menu when it's ready for business and decide, then, whether I want to upgrade.

On balance I think I prefer new-menu's inline environment variables in docker-compose.yml so I've started to adopt that, but I hate, loathe and detest the networks growing like weeds effect of the new menu. I'm never going to adopt that. Not on a bet.

I also like everything in one place - docker-compose.yml - so I have no truck with either the Docker-supplied docker-compose.override.yml or IOTstack-supplied compose-override.yml. Once you make that decision, there's not much purpose in running the new menu as a maintenance tool.

Bottom line: I still think the best approach with IOTstack is to use the menu (whichever one you choose) as a starting point only. Then, learn how it all hangs together and roll your own thereafter. I reckon you get a much quieter life.

Just my 2 cents.🤓