TSDB basic authorization, gzip, batch, prefix

Grotax commented 7 years ago

Added some features to the TSDB Handler basic authorization: simple header with user and password for firewall protection gzip: added gzip support to compress metrics batch: added support to send metrics in batch prefix: you can add a prefix to all you metrics like diamond.myhostname.cpu.cpu_count

All these features can be disabled by not defining or setting a value lower than 1.

Sending metrics in batch can give you quite a nice performance boost and lower you cpu load. Compressing will reduce the buffer size you need to configure on the tsdb end.

On the negative side this will break the recently added tests and I was not able to recreate them. I would need some help as there is currently no handler (using urllib2) with tests. But I didn't want to keep these changes for my self. batch This graph shows you how long it took to send a metric to the db.

Grotax commented 7 years ago

Yea probably a good idea will take a look tomorrow

Am 21. März 2017 23:55:25 MEZ schrieb Grant Ridder notifications@github.com:

shortdudey123 commented on this pull request.

@@ -1,370 +1,375 @@

coding=utf-8

-

I was afraid you were going to say something like that. Is it possible to break into 2 commits then so its easier to see changes? 1st commit is the newline change and 2nd commit is your actual code changes?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/python-diamond/Diamond/pull/616#discussion_r107296451

shortdudey123 commented 7 years ago

With respect to the failing rests due to the urllib2 change, take a look at the elasticsearch, kafkastat, and nginx collector tests. They all mock out the urllib2 calls

Grotax commented 7 years ago

So after some more investigation I have added some tests and converted the old ones. Don't know why the build failed maybe travis needs to rebuild. Had some git problems but now it's sorted :)

shortdudey123 commented 7 years ago

Travis failed due to pep8 issues :) Once you get those fixed, i will review the PR more thoroughly

$ pep8 --config=.pep8 src bin/diamond bin/diamond-setup build_doc.py setup.py test.py
src/diamond/handler/test/testtsdb.py:27:81: E501 line too long (215 > 80 characters)
src/diamond/handler/test/testtsdb.py:40:81: E501 line too long (266 > 80 characters)
src/diamond/handler/test/testtsdb.py:56:81: E501 line too long (315 > 80 characters)
src/diamond/handler/test/testtsdb.py:68:81: E501 line too long (249 > 80 characters)
src/diamond/handler/test/testtsdb.py:80:81: E501 line too long (223 > 80 characters)
src/diamond/handler/test/testtsdb.py:94:81: E501 line too long (261 > 80 characters)
src/diamond/handler/test/testtsdb.py:96:81: E501 line too long (96 > 80 characters)
src/diamond/handler/test/testtsdb.py:109:81: E501 line too long (244 > 80 characters)
src/diamond/handler/test/testtsdb.py:112:5: E303 too many blank lines (2)
src/diamond/handler/test/testtsdb.py:112:81: E501 line too long (89 > 80 characters)
src/diamond/handler/test/testtsdb.py:126:81: E501 line too long (107 > 80 characters)
src/diamond/handler/test/testtsdb.py:139:81: E501 line too long (245 > 80 characters)
src/diamond/handler/test/testtsdb.py:141:81: E501 line too long (107 > 80 characters)
src/diamond/handler/test/testtsdb.py:155:81: E501 line too long (263 > 80 characters)
src/diamond/handler/test/testtsdb.py:157:81: E501 line too long (83 > 80 characters)
src/diamond/handler/test/testtsdb.py:172:81: E501 line too long (319 > 80 characters)
src/diamond/handler/test/testtsdb.py:174:81: E501 line too long (86 > 80 characters)
src/diamond/handler/test/testtsdb.py:190:81: E501 line too long (267 > 80 characters)
src/diamond/handler/test/testtsdb.py:192:81: E501 line too long (85 > 80 characters)
src/diamond/handler/test/testtsdb.py:207:81: E501 line too long (297 > 80 characters)
src/diamond/handler/test/testtsdb.py:209:81: E501 line too long (88 > 80 characters)
src/diamond/handler/test/testtsdb.py:225:81: E501 line too long (268 > 80 characters)
src/diamond/handler/test/testtsdb.py:227:81: E501 line too long (82 > 80 characters)
src/diamond/handler/test/testtsdb.py:242:81: E501 line too long (272 > 80 characters)
src/diamond/handler/test/testtsdb.py:244:81: E501 line too long (85 > 80 characters)
src/diamond/handler/test/testtsdb.py:260:81: E501 line too long (255 > 80 characters)
src/diamond/handler/test/testtsdb.py:262:81: E501 line too long (83 > 80 characters)
src/diamond/handler/test/testtsdb.py:277:81: E501 line too long (270 > 80 characters)
src/diamond/handler/test/testtsdb.py:279:81: E501 line too long (86 > 80 characters)
src/diamond/handler/test/testtsdb.py:295:81: E501 line too long (251 > 80 characters)
$