amonagent unable to authenticate with amon server at first

[kevinquinnyo]

I am installing amonagent (just the agent) manually via a saltstack state.

In order to have the amonagent authenticate and be viewable at /servers/ on my amon server, I've noticed I have to issue:

/opt/amonagent/amonagent -test

Right after installation. Before that the error I was getting was:

2016/02/19 14:47:56 Starting Amon Agent (version 0.4.9)
2016/02/19 14:47:56 INFO amonagent.log Agent Config: Interval:1m0s
2016/02/19 14:47:59 INFO amonagent.log Metrics collected (Interval:1m0s)
2016/02/19 14:47:59 INFO amonagent.log Flusher routine failed, exiting: Can't connect to the Amon API on Can't connect to the Amon API on Post /api/system/v2/?api_key=: unsupported protocol scheme ""

After running the -test option though, it runs though the checks, succeeds, and the servers immediately show up in the amon server UI.

Here is my amonagent salt state if you want to get an idea on the order / procedure i'm using to install amonagent by itself:

/srv/salt/amon/init.sls

#!py

def run():
    api_key = __pillar__['amon.api_key']
    amon_instance = __pillar__['amon.server']
    states = {}

    states['amonagent-repo'] = {
        'pkgrepo.managed': [
            {'humanname': 'Amon'},
            {'name': 'deb http://packages.amon.cx/repo amon contrib'},
            {'file': '/etc/apt/sources.list.d/amonagent.list'},
            {'keyid': 'AD53961F'},
            {'keyserver': 'hkp://keyserver.ubuntu.com:80'},
        ]
    }

    states['amonagent'] = {
        'pkg.installed': [
            {'require': [
                {'pkgrepo': 'amonagent-repo'}
            ]}
        ]
    }

    context = {
        'api_key': api_key,
        'amon_instance': amon_instance
    }
    states['/etc/opt/amonagent/amonagent.conf'] = {
        'file.managed': [
            {'user': 'root'},
            {'group': 'root'},
            {'mode': 640},
            {'source': 'salt://amon/files/amonagent.conf'},
            {'context': context},
            {'template': 'jinja'},
            {'require': [
                {'pkg': 'amonagent'}
            ]}
        ]
    }
    # PR https://github.com/amonapp/amonagent/pull/3 to fix typo upstream not merged yet
    states['fix-init-script-typo'] = {
        'file.replace': [
            {'name': '/etc/init.d/amonagent'},
            {'pattern': 'CONFING'},
            {'repl': 'CONFIG'},
            {'require': [
                {'pkg': 'amonagent'}
            ]}
        ]
    }

    states['amonagent-running'] = {
        'service.running': [
            {'name': 'amonagent'},
            {'require': [
                {'file': 'fix-init-script-typo'}
            ]}
        ]
    }

    # seems that it needs this to fire off an initial auth request?
    # it won't register in amon server until after i run this
    states['amonagent-test'] = {
        'cmd.run': [
            {'name': '/opt/amonagent/amonagent -test'},
            {'require': [
                {'service': 'amonagent-running'}
            ]}
        ]
    }

    return states

[kevinquinnyo]

I notice for the ones I installed manually, there is "No data for the selected period" under System for those servers.

Where are the docs for this?

[martinrusev]

@kevinquinnyo Can you please check if these manually installed agents are running:

systemctl amonagent status
(or)
service amonagent status

After running the -test option though, it runs though the checks, succeeds, and the servers immediately show up in the amon server UI.

This is a bug, I will look into it

[kevinquinnyo]

root@cache1:~# service amonagent status
amonagent Process is running [ OK ]

It looks like it's still producing the error about the scheme which I thought was od since the amonagent.conf file has http:// in the amon_instance key:

root@cache1:~# tail -n5 /var/log/amonagent/amonagent.log 

2016/02/20 07:40:38 INFO amonagent.log Metrics collected (Interval:1m0s)
2016/02/20 07:40:38 INFO amonagent.log Flusher routine failed, exiting: Can't connect to the Amon API on Can't connect to the Amon API on Post /api/system/v2/?api_key=: unsupported protocol scheme ""

root@cache1:~# date
Sat Feb 20 07:41:25 VET 2016

[kevinquinnyo]

root@cache1:~# cat /etc/opt/amonagent/amonagent.conf 
{"api_key": "my_key_here", "amon_instance": "http://my.ip.add.ress"}
root@cache1:~# curl -X POST http://my.ip.add.ress/api/system/v2/ --data "api_key=my_key_here"
{"detail":"Authentication credentials were not provided."}

Obviously I'm not testing the connection correctly. What data do I need to post to the server to test the connection manually? Or would that even help in troubleshooting the issue?

[martinrusev]

That is a strange error. The testing URL is: http://my.ip.add.ress/api/system/v2/?api_key=api_key_here.

Can you please run /opt/amonagent/amonagent -test once again and take a look at the test URL at the end