grafana_dashboard and grafana_datasource don't check if grafana is running

[Marcellii]

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 4.9.3
• Ruby: 2.4.0
• Distribution: Archlinux
• Module version: master

How to reproduce (e.g Puppet code you use)

class profiles::grafana {

  class{'::grafana':
  }

  grafana_dashboard{'example_dashboard':
    grafana_url      => 'http://localhost:3000',
    grafana_user     => 'admin',
    grafana_password => 'admin',
    content          => template('profiles/grafana_dashboard.json.erb'),
  }

  grafana_datasource{'influxdb':
    grafana_url      => 'http://localhost:3000',
    grafana_user     => 'admin',
    grafana_password => 'admin',
    type             => 'influxdb',
    url              => 'http://localhost:8086',
    user             => 'admin',
    password         => 'test123',
    database         => 'ressources',
    access_mode      => 'proxy',
    is_default       => true,
  }
}

What are you seeing

root@localhost /etc/puppetlabs/code/modules/grafana # puppet apply -e 'include profiles::grafana'
Warning: ModuleLoader: module 'profiles' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Notice: Compiled catalog for localhost.localdomain in environment production in 0.71 seconds
Notice: /Stage[main]/Grafana::Config/File[/etc/grafana.ini]/content: content changed '{md5}fb404386c6807d5424cab024babce097' to '{md5}cc5712b4a99d20259b9e045e1015cf13'
Notice: /Stage[main]/Grafana::Config/File[/etc/grafana.ini]/owner: owner changed 'root' to 'grafana'
Notice: /Stage[main]/Grafana::Config/File[/etc/grafana.ini]/group: group changed 'root' to 'grafana'
Notice: /Stage[main]/Grafana::Service/Service[grafana]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/Grafana::Service/Service[grafana]: Triggered 'refresh' from 1 events
Error: /Stage[main]/Profiles::Grafana/Grafana_dashboard[example_dashboard]: Could not evaluate: Failed to open TCP connection to localhost:3000 (Connection refused - connect(2) for "localhost" port 3000)
Error: /Stage[main]/Profiles::Grafana/Grafana_datasource[influxdb]: Could not evaluate: Failed to open TCP connection to localhost:3000 (Connection refused - connect(2) for "localhost" port 3000)
Notice: Applied catalog in 3.44 seconds
root@localhost /etc/puppetlabs/code/modules/grafana # systemctl status grafana.service

What behaviour did you expect instead

I expected to be working

Any additional information you'd like to impart

[bastelfreak]

Hi @Marcellii. thanks for the bugreport. we should ensure that grafana is actually reachable before the custom types and providers are started. The service may not be reachable if a refresh got triggered.

[fnoop]

Something like the elastic search connection validator might work although it's a bit of a heavy stick: https://forge.puppet.com/elastic/elasticsearch#connection-validator

Shame puppet doesn't have a generic connection validator, would be useful.

[fnoop]

OK this is a workaround:

    class { "::grafana": 
       ...
    } ->
    exec {"grafana-postdelay":
        command => "/usr/bin/wget --spider --tries 30 --waitretry=30 --retry-connrefused --no-check-certificate http://localhost:${webport}/public/img/grafana_icon.svg",
    } ->
    grafana_datasource { 'influxdb':
       ...
    }

[bastelfreak]

@fadenb do you want to check if https://forge.puppet.com/puppet/healthcheck would be suitable?

[fnoop]

@bastelfreak Oh thanks, that works great!

    http_conn_validator { 'grafana-postdelay' :
        host    => '127.0.0.1',
        port    => $webport,
        use_ssl => false,
        test_url => '/public/img/grafana_icon.svg',
    } ->

[bastelfreak]

cool cool. Would you mind to document this in our README.md as an example?

[ofbeaton]

I should mention that with grafana 5.x we can now use provisioning instead of the API for both dashboards and datasources. This does not suffer from this problem, as you can always write the files to the filesystem even if grafana isn't running yet.

http://docs.grafana.org/administration/provisioning/

In fact I had to switch from dashboard types to provisioning as I had all kinds of problems using the API (dashboards titles not being found despite them being installed, resulting in errors, dashboards being continuously deployed as they are not immutable...). Switching to provisioning solved all my problems.

May we should update the documentation for that too?