Graphite datasource

[AGirin]

I am trying to use graphite searches and have a question on how to actually use it. I enabled graphite in the config.php and restarted httpd. Trying to create new search and not getting good result just this:

SearchException: Remote server returned 0: :

My config looks like this:

$config['graphite'] = [ 'host' => ['http://graphitetest'], ];

My query looks like this: ServerName.meminfo.used

What am I doing wrong here?

Thanks!

[kiwiz]

411 turns that into http://graphitetest/render?target=transformNull(ServerName.meminfo.used). Does that give results?

[AGirin]

:-) It does, but it returns png. You probably need to add this:

http://graphitetest/render?target=transformNull(ServerName.meminfo.used)&format=json

or

http://graphitetest/render?target=transformNull(ServerName.meminfo.used)&format=csv

[kiwiz]

Opps, we actually use &raw. Can you inspect the call here: https://github.com/etsy/411/blob/master/phplib/Search/Graphite.php#L54 and see what error message curl is returning?

[AGirin]

How can I do this? Sorry, I am not that familiar with php.

I used &format=raw and I do get result with curl.

curl -X GET 'http://graphitetest/render?target=transformNull(TestServer.meminfo.used)&format=raw' transformNull(TestServer.meminfo.used,0),1477506000,1477592400,60|5409239.33333,5376928.0,5377082.66667,5376348.66667,5377854.0,5403138.66667,5383543.33333,5379501.33333,5379480.0,5378435.33333,5416728.66667,5384055.33333,5382626.66667,5383032.0,5384245.33333,5411917.33333,5386283.33333,5384394.0,5384274.0,5385105.33333,5418946.66667,5391062.66667,5399434.66667,5398248.66667,5399042.0,5424368.0,5399430.0,5400075.33333,5399007.33333,5399214.66667,5432235.33333,5403503.33333,5400987.33333,5401531.33333,5402508.0,5424987.33333,5402220.66667,5402598.0,5404276.0,5404462.66667,5449074.66667,5410876.66667,5406648.66667,5406180.0,5407106.0,5434472.0,5408094.0,5409320.0,5409838.66667,5410074.0,5441948.66667,5411446.0,5412893.33333,5415073.33333,5415518.66667,5442176.0,5416188.0,5415534.66667,5413768.0,5409898.66667

[AGirin]

Found the issue - I guess [ ] are not supported in graphite config:

$config['graphite'] = [ 'host' => 'http://graphitetest', ];

NOT:

$config['graphite'] = [ 'host' => ['http://graphitetest'], ];

[AGirin]

Closed and reopened. So alerting works with graphite but... I do not see any alerts on 411 Alerts page and I also see these errors in Elasticsearch directly related to the graphite alert:

My guess 411 has troubles writing to ES because of the incorrect number format?

[AGirin]

Also saw a bunch of these errors when graphite search was active:

except [NONE] Errno 2048: "Declaration of ECL\Command\Elasticsearch\Builder::setSettings() should be compatible with ECL\Command\Elasticsearch::setSettings(array $settings)" at [/var/www/411/vendor/kiwiz/ecl/src/Command/Elasticsearch/Builder.php:9] 0:[include() called at [/var/www/411/vendor/composer/ClassLoader.php:414]] 1:[Composer\Autoload\includeFile() called at [/var/www/411/vendor/composer/ClassLoader.php:301]] 2:[Composer\Autoload\ClassLoader->loadClass() called at [:0]] 3:[spl_autoload_call() called at [/var/www/411/phplib/Search/ECL.php:96]] 4:[include_once() called at [/var/www/411/phplib/411bootstrap.php:48]] 5:[FOO{closure}() called at [:0]] 6:[spl_autoload_call() called at [/var/www/411/phplib/TypeModel.php:27]] 7:[FOO\TypeModel::getTypes() called at [/var/www/411/phplib/TypeModel.php:51]] 8:[FOO\TypeModel::getSchema() called at [/var/www/411/phplib/Model.php:574]] 9:[FOO\ModelFinder::generateWhere() called at [/var/www/411/phplib/Model.php:530]] 10:[FOO\ModelFinder::generateQuery() called at [/var/www/411/phplib/Model.php:492]] 11:[FOO\ModelFinder::getByQuery() called at [/var/www/411/phplib/REST/Models.php:130]] 12:[FOO\Models_REST->read() called at [/var/www/411/phplib/REST/Searches.php:48]] 13:[FOO\Searches_REST->GET() called at [/var/www/411/phplib/REST.php:107]] 14:[FOO\REST->route() called at [/var/www/411/htdocs/api/searches.php:6]], referer: http://411.test.com/alerts

[kiwiz]

Hmm, try composer update.

[AGirin]

That's new:

Loading composer repositories with package information Updating dependencies (including require-dev) Your requirements could not be resolved to an installable set of packages.

Problem 1

• Installation request for etsy/phan ^0.6.0 -> satisfiable by etsy/phan[0.6].
• etsy/phan 0.6 requires php >=7.0 -> your PHP version (5.6.26) does not satisfy that requirement.

Also - is this related to the errors in the logs? Or to the fact that no Alerts are created and 411 is having issues writing to ES?

[AGirin]

I removed require-dev from composer config. Updated. It looks like the errors in the log stopped. But graphite search still not creating any alerts on 411 page and I see errors in ES (NumberFormatExcpetion)

[AGirin]

Also we started getting this kind of emails very often now:

Then follow up message comes in:

Not sure why this is happening.

[kiwiz]

Those emails indicate that 411's occasionally getting an error when it tries to determine the current health of the Graphite server. Do you have any idea why that would be the case? (network, cpu, etc)

[AGirin]

I figured it out. We were restarting httpd on graphite server to limit memory used. So we can ignore those emails. What about alerts and NumberFormatException errors?

[kiwiz]

@AGirin Can you provide the mapping for your 411 index (probably 411_alerts_1)?

[AGirin]

@kiwiz - not sure where to get that info. Is this what you need?

[kiwiz]

Nope, I mean the mapping for ES. /_mapping/411_alerts_1 should give you it.

[AGirin]

Yes, this is what I have:

[kiwiz]

I need to see the details for each individual mapped field: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

[AGirin]

Does it look like what you need (not sure how to make formatting better)?

{ "411_alerts_1": { "mappings": { "logstash": {` "properties": { "alert_date": { "type": "date", "format": "epoch_second" }, "archived": { "type": "boolean" }, "assignee": { "type": "long" }, "assignee_type": { "type": "long" }, "category": { "type": "string" }, "content": { "properties": { "@version": { "type": "string" }, "Category": { "type": "long" }, "ComputerName": { "type": "string" }, "EventIdentifier": { "type": "long" }, "InsertionStrings": { "type": "string" }, "Logfile": { "type": "string" }, "RecordNumber": { "type": "long" }, "SourceName": { "type": "string" }, "TimeGenerated": { "type": "string" }, "TimeWritten": { "type": "string" }, "Type": { "type": "string" }, "User": { "type": "string" }, "host": { "type": "string" }, "message": { "type": "string" }, "metrics": { "properties": { "server_host": { "type": "string" }, "server_lag": { "type": "double" }, "server_time": { "type": "date", "format": "strict_date_optional_time||epoch_millis" } } }, "type": { "type": "string" } } }, "content_hash": { "type": "string" }, "create_date": { "type": "date", "format": "epoch_second" }, "escalated": { "type": "boolean" }, "id": { "type": "long" }, "notes": { "type": "string" }, "owner": { "type": "long" }, "priority": { "type": "long" }, "renderer_data": { "type": "object" }, "resolution": { "type": "long" }, "search_id": { "type": "long" }, "site_id": { "type": "long" }, "state": { "type": "long" }, "tags": { "type": "string" }, "update_date": { "type": "date", "format": "epoch_second" } } }, "default": { "properties": { "alert_date": { "type": "date", "format": "epoch_second" }, "assignee": { "type": "long" }, "assignee_type": { "type": "long" }, "content": { "type": "object" }, "content_hash": { "type": "string" }, "create_date": { "type": "date", "format": "epoch_second" }, "escalated": { "type": "boolean" }, "notes": { "type": "string" }, "resolution": { "type": "long" }, "search_id": { "type": "long" }, "state": { "type": "long" }, "update_date": { "type": "date", "format": "epoch_second" } } } } } }

[kiwiz]

That's perfect. Looking.

[AGirin]

Also I am finding that I am getting strange result when I try using "Less then" in Graphite search.

This is what I get through API:

xxxxxx.memory.available_bytes,1478809140,1478809200,10|1900199936.0,1900597248.0,1900032000.0,1900806144.0,1895153664.0,1895284736.0

For a test I specified number lot larger in a search expecting to see alert but sometimes I get this:

Or this

So it looks like query does not really work. From all my hundreds of tries I got it working twice I think (somehow it displayed the right value and alert). But other times I get the messages above.

It works fine however with greater than - no problems here.

[AGirin]

Well - I think after all updates to 411 it started creating alerts on the alerts page. But still graphite search is not reliable - at least when I click Test or Execute it gives me different results all the time (most of the time no results or count 0 but very rarely - the actual correct value).

I tried curl from the same server I run 411 and it returns all values just fine from graphite.

[kiwiz]

@Kennysan Could you take a look at this?

[kiwiz]

@AGirin Did you get this working? What was the url you were hitting with curl?

[kiwiz]

Closing due to inactivity.