Not able to work with Ganglia

[GoogleCodeExporter]

I am trying to use nginx-sflow module with Ganglia-3.4.0 I cannot get it to 
work. 

Looking at ganglia code it looks like it expects HID to be passed to it. 
However, the nginx-sflow module never passes that. 

Original issue reported on code.google.com by umes...@gmail.com on 7 Sep 2012 at 11:52

[GoogleCodeExporter]

The HID structure comes from the hsflowd daemon which should be running as 
well.  Is hsflowd installed on the nginx servers?    The nginx sub-agent gets 
his config from /etc/hsflowd.auto which is written out by hsflowd.  In turn 
hsflowd can learn config from DNS-SD or /etc/hsflowd.conf.  So that way you can 
control everything via DNS-SD if you want, or via /etc/hsflowd.conf if you 
prefer to use config files.

One thing that may not be documented so well is that you can set a specific 
sampling rate just for HTTP with the sampling.http=<n> setting.  That goes in 
/etc/hsflowd.conf or the DNSSD TXT record,  depending on which method you 
choose.

Please let me know if any of this is not clear.

Neil

Original comment by neil.mck...@gmail.com on 8 Sep 2012 at 3:30

• Changed state: Accepted

[GoogleCodeExporter]

Thanks a lot for the quick response.

hsflowd was not installed on the servers running nginx. I configured 
/etc/hsflowd.auto manually and not rely on DNS. Based on the information above 
I should rather configure /etc/hsflowd.conf. 

I was going to rely on gmond for host statistics. I will try running hsflowd 
along side and see if that helps.

Original comment by umes...@gmail.com on 8 Sep 2012 at 4:56

[GoogleCodeExporter]

Just tried using nginx-sflow with hsflowd. Now it works great.

Thanks for the great module.

Original comment by umes...@gmail.com on 8 Sep 2012 at 6:56

[GoogleCodeExporter]

Thanks.

Closing ticket.

Neil

Original comment by neil.mck...@gmail.com on 10 Sep 2012 at 4:49

• Changed state: Done

[GoogleCodeExporter]

I cannot get this to work either. I do have hsflow running and I am seeing its 
metrics from gmond. 

Does nginx-sflow broadcast its metrics or are they polled? I thought that they 
were broadcast straight to gmond using the collector () information. But the 
fact that we need to have hsflow running makes me question that assumption. In 
which case do I need to define where and how that they are collected? 

Do I need to specify the 
 # option to listen for JSON sFlow-APPLICATION messages
  ######################################################
  # jsonPort = 36343
section of the hsflowd.conf file?

anyway to debug?

thanks for any pointers - based on the previous post, it looks like a user 
error :(

Original comment by jonjsil...@gmail.com on 4 Sep 2013 at 8:31

[GoogleCodeExporter]

nginx-sflow-module picks up config from /etc/hsflowd.auto,   but then sends 
it's data directly to the sFlow collector.    Best way to debug is probably to 
add another collector to /etc/hsflowd.conf like this:

collector {
  ip = 127.0.0.1
  udpport = 6344
}

and then restart hsflowd.

Confirm that /etc/hsflowd.auto has been updated with the new info,  and then 
run:

sflowtool -p 6344

Where sflowtool (src or binary) can be downloaded from here:

http://www.inmon.com/technology/sflowTools.php

In addition to the host-sflow CPU/mem/IO counters you should see HTTP 
transaction samples,  and periodic HTTP counter samples (e.g. "status2xx").   
It's those counter-samples that Ganglia gmond is looking for.

You don't need the jsonPort setting.  It's there to allow any other 
applications and scripts to contribute transaction-samples and counters too.

In the Ganglia gmond.conf you might need to specific "accept_http_metrics=yes".

And you might want to add "multiple_http_instances=yes" if you have more than 
one nginx instance per host (causes gmond to add a service-port suffix to all 
the metrics).

Let me know how it goes.

Neil

Original comment by neil.mck...@gmail.com on 4 Sep 2013 at 8:53

[GoogleCodeExporter]

Also,  do you have HTTP requests going to Nginx right now?   The 
nginx-sflow-module is missing a timer-driven callback to flush counters even if 
nothing is happening.   It relies on new transactions arriving all the time.   
Must fix that....I'll change the status of this thread back to "accepted".

Neil

Original comment by neil.mck...@gmail.com on 4 Sep 2013 at 9:30

• Changed state: Accepted

[GoogleCodeExporter]

no http metrics coming to flowtool. I am getting hsflow metrics though. 
I was not running any traffic through nginx, but now I am just running a couple 
of manual web requests, some good, some not. 

I do have all of the conf parms as you suggested. I also saw during the nginx 
build that the slow module was built. 

If I do an strace to nginx, I see it processing my page request and 
  stat("/etc/hsflowd.auto", {st_mode=S_IFREG|0644, st_size=241, ...}) = 0
so it looks like the module being called, but no http stats out in sflowtool 

any suggestions? 

Original comment by jonjsil...@gmail.com on 5 Sep 2013 at 3:43

[GoogleCodeExporter]

Got it working !!!!
Seems like it may have just been a timing issue and the number of http requests 
that I was making. I sent a whole flood in and I started seeing the stats. !!!!

How do I control their name? I see that they are prefixed with 80 as in 
    80.http_meth_get
    80.http_status_5XX
The stats do not have this prefix coming into sflowtool, so I'm not really sure 
where and how that application prefix gets controlled from. 

thanks again. 

Original comment by jonjsil...@gmail.com on 5 Sep 2013 at 5:36

[GoogleCodeExporter]

It sounds like maybe you have the "multiple_http_instances=yes" setting turned 
on in gmond.conf (?)

If so,  turning it off should cause the prefix to disappear.   However you'll 
have to clear out all the rrds in gmetad so that they don't stick around and 
get in the way.

I'll close this ticket and open another to remind me to fix the timer-tick 
thing.

Neil

P.S.  FYI,  you might also want to run the feed into "sFlowTrend" (freeware),  
because it can interpret the transaction-samples too.   Another suggestion 
would be to try out "sFlow-RT" (free for eval/non-commercial use) because it 
can also offer real-time monitoring -- it's metrics update asynchronously as 
new data arrives,  so you can trend with per-second updates if you want to.   
And "Traffic Sentinel" might be worth evaluating if you have many thousands of 
hosts and/or hundreds of network devices that support sFlow too.  One nice 
property of sFlow is that you can try all these tools in parallel,  driving 
them all from the same measurement infrastructure.

Original comment by neil.mck...@gmail.com on 5 Sep 2013 at 5:53

• Changed state: Done

[GoogleCodeExporter]

yes, I do have multiple_http_instances=yes because I was thinking about 
collecting metrics from node.js (which reports these same metrics). I was 
thinking about naming them with the application name instead of the server 
port, but I guess that I'll need to think about that for awhile. 

I was hoping that it was configurable somewhere so that I would have the 
choice, but I found it in the code, so I guess not.

anyways, 
it looks like a cool tool - and thanks for all of the help.
jon

Original comment by jonjsil...@gmail.com on 5 Sep 2013 at 6:54