netdata release v1.7

[ktsaou]

Changelog for netdata v1.7:

---

This is release v1.7 of netdata.

netdata is still spreading fast: we are at 320.000 users and 132.000 servers! Almost 100k new users, 52k new installations and 800k docker pulls since the previous release 4 and a half months ago! netdata user base grows at about 1000 new users and 600 new servers per day! Thank you! You are awesome!

The next release (v1.8) will be focused on providing a global health monitoring service, for all netdata users, for free! Read more about it here. We need supporters for this cause. Join us!

key highlights of netdata v1.7

1. netdata is now a (very fast) fully featured statsd server and the only one with automatic visualization: push a statsd metric and hit F5 on the netdata dashboard: your metric visualized. It also supports synthetic charts, defined by you, so that you can correlate and visualize your application the way you like it.

2. netdata got new installation options - it is now easier than ever to install netdata - we also distribute a statically linked netdata x86_64 binary, including key dependencies (like bash, curl, etc) that can run everywhere a Linux kernel runs (CoreOS, CirrOS, etc).

3. metrics streaming and replication has been improved significantly. All known issues have been solved and key enhancements have been added. headless collectors and proxies can now send metrics to backends when data source = as collected.

4. backends have got quite a few enhancements, including host tags and metrics filtering at the netdata side; prometheus support has been re-written to utilize more prometheus features and provide more flexibility and integration options.

5. netdata now monitors ZFS (on Linux and FreeBSD), ElasticSearch, RabbitMQ, Go applications (via expvar), ipfw (on FreeBSD 11), samba, squid logs (with web_log plugin!).

6. netdata dashboard loading times have been improved significantly (hit F5 a few times on a netdata dashboard - it is now amazingly fast), to support dashboards with thousands of charts.

7. netdata alarms now support custom hooks, so you can run whatever you like in parallel with netdata alarms.

8. As usual, this release brings dozens of more improvements, enhancements and compatibility fixes.

netdata is now a fully featured statsd server

netdata is now a fully featured statsd server. It can collect statsd formatted metrics, visualize them on its dashboards, stream them to other netdata servers or archive them to backend time-series databases.

netdata statsd is fast. It can collect more than 1.200.000 metrics per second on modern hardware, more than 200Mbps of sustained statsd traffic. netdata statsd is inside netdata. This provides a distributed statsd implementation.

netdata also supports statsd synthetic charts: You can create dedicated sections on the dashboard to render the charts. You can control everything: the main menu, the submenus, the charts, the dimensions on each chart, etc.

Read more about netdata statsd

counters

• Scope: count the events of something (e.g. number of file downloads)
• Format: name:INTEGER|c or name:INTEGER|C or name|c
• statsd increments the counter by the INTEGER number supplied (positive, or negative).

gauges

• Scope: report the value of something (e.g. cache memory used by the application server)
• Format: name:FLOAT|g
• statsd remembers the last value supplied, and can increment or decrement the latest value if FLOAT begins with + or -.

histograms

• Scope: statistics on a size of events (e.g. statistics on the sizes of files downloaded)
• Format: name:FLOAT|h
• statsd maintains a list of all the values supplied and provides statistics on them.

The same chart with sum unselected, to show the detail of the dimensions supported:

meters

This is identical to counter.

• Scope: count the events of something (e.g. number of file downloads)
• Format: name:INTEGER|m or name|m or just name
• statsd increments the counter by the INTEGER number supplied (positive, or negative).

sets

• Scope: count the unique occurrences of something (e.g. unique filenames downloaded, or unique users that downloaded files)
• Format: name:TEXT|s
• statsd maintains a unique index of all values supplied, and reports the unique entries in it.

timers

• Scope: statistics on the duration of events (e.g. statistics for the duration of file downloads)
• Format: name:FLOAT|ms
• statsd maintains a list of all the values supplied and provides statistics on them.

The same chart with the sum unselected:

---

dashboard improvements

There have been significant optimizations to the loading times of the dashboard. The dashboard loads instantly now, even when there are several hundreds of charts in it (hit F5 on the dashboard - it is super fast).

For those who know: we eliminated most browser reflows, by refactoring the way the charts are initialized and splitting initialization in 2 phases. Unfortunately we had to re-shape gauge and easypiecharts, so pay some attention to your custom dashboards after updating.

We now use natural sorting on the dashboard elements (i.e. instead of 1, 10, 2, 3 we get 1, 2, 3, 10).

There have been dozens of performance improvements on the netdata dashboard. Like all the previous releases, this release makes netdata the fastest netdata so far!

new installation methods

• Single line installation on Linux
• Static 64bit packages for Linux
• Improved support for Red Hat Enterprise Linux @racciari,
• Improved support for Amazon Machine Image
• Improved support for Centos @n0coast
• Many more installer/updater improvements @nielsAD, @mfurlend

Streaming

• improved self cleanup of obsolete charts and hosts at a central netdata.
• host tags are now propagated from netdata to netdata while streaming metrics.
• log error when multiple clients are streaming the metrics of the same host.
• dozens more streaming improvements and bugfixes.

Backends

• New prometheus backend, supporting all the features of the others backends netdata supports. The new format changed the names of metrics, so if you use grafana or other tools you will have to update your queries.
• Prometheus and opentsdb now support host tags (advanced ephemeral nodes monitoring)
• Metrics sent to backends with data source average, sum or volume (from the netdata database) are now more accurate.
• Added contrib/nc-backend.sh, a script that can act as a fallback backend for graphite, opentsdb and compatibles.
• netdata nodes without a database (slaves and proxies) can now send as collected metrics to backends.

New and improved plugins

• Go apps monitoring via expvar ! @kralewitz
• ElasticSearch monitoring ! @l2isbad
• RabbitMQ monitoring ! @l2isbad
• ipfw monitoring under FreeBSD 11 ! @vlvkobal
• ZFS monitoring under FreeBSD (@vlvkobal) and Linux !
• samba monitoring ! @ntlug
• web_log plugin can now monitor squid logs too ! @l2isbad
• web_log plugin can now monitor apache cache logs too (removed old apache_cache plugin) @l2isbad
• many more web_log improvements - web_log is now a lot more powerful! @l2isbad
• python.d.plugin LogService now supports monitoring web log files matching a pattern @l2isbad
• disk monitoring under Linux now utilizes /dev/mapper names. It also has improved docker compatibility.
• haproxy improvements @l2isbad
• dns_query_time plugin to monitor the response time of nameservers @l2isbad
• Fronius Solar @BrainDoctor
• better support for monitoring Proxmox/qemu @efaden and libvirt/qemu VMs
• cpufreq improvements @l2isbad
• smartd_log improvements @pkoenig10
• bind_rndc rewritten @l2isbad
• lighttpd improvements (part of the apache plugin)
• isc_dhcpd improvements @l2isbad
• fping improvements
• apps.plugin improvements (added many more applications to monitor, notably hadoop and friends, improved compatibility)
• freeipmi improvements
• mdstat improvements @l2isbad
• mysql improvements @alibo
• redis improvements @l2isbad
• postgres rds fixes @facetoe
• fail2ban improvements @l2isbad
• idlejitter rewritten
• openvpn improvements @l2isbad
• numa improvements @Benje06

New and improved alarms

• alarm-notify.sh now supports custom notification methods (you can hook whatever you like to netdata alarms).
• email notifications are now multipart (have both HTML and text versions in them)
• low memory alarm now excludes ZFS ARC.
• improved discord notifications.
• improved telegraf notifications @alibo
• lighttpd alarm
• mongodb alarm @jnogol

Other improvements

• memory mode ram utilizes KSM (kernel memory deduper).
• many memory mode map improvements for faster operation with huge databases.
• netdata is now even faster on FreeBSD, thank to several optimization made by @vlvkobal
• netdata can now be compiled with clang, even on FreeBSD
• netdata can now be compiled on FreeBSD 10.3

[ktsaou]

I may have missed something. So please correct / complete me if something is not as it should or missing.

[elfgoh]

New Prometheus backend, supporting all the features of the others backends netdata supports.

Need to update the wiki once released!

[ktsaou]

Need to update the wiki once released

It is already updated: https://github.com/firehol/netdata/wiki/Using-Netdata-with-Prometheus

prometheus pulls data from netdata, so it has its own wiki page.

[ktsaou]

Added a link to it at the backends wiki page.

[ilyam8]

LogService: support for log file with pattern match

[ktsaou]

updated it. preparing the release now...

[ktsaou]

@philwhineray for the release commits, I use my fork or the official repo? (sorry I have forgotten)

[philwhineray]

you need to create a signed tag and push to the official repo. I don't think it matters where you create it. I use my local repo with the hooks installed to make sure everything is ok before the push.

On 16 Jul 2017 5:30 pm, "Costa Tsaousis" notifications@github.com wrote:

@philwhineray https://github.com/philwhineray for the release commits, I use my fork or the official repo? (sorry I have forgotten)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firehol/netdata/issues/2456#issuecomment-315620440, or mute the thread https://github.com/notifications/unsubscribe-auth/ABNlC4_cHZzP5aOoEFDMksal3FyP251dks5sOjq_gaJpZM4OWUcX .

[ktsaou]

@philwhineray have a look at the releases page:

1. Once I committed, it showed just this:

2. When travis finished, it added this:

3. I clicked on on v1.7.0 from point 1 above to save a draft release, and it added this:

How do you suggest to proceed?

[ktsaou]

Also, travis failed on macos.

[ktsaou]

I have not signed the results yet. Waiting for your input...

[philwhineray]

I am not sure what you are asking me?

[ktsaou]

ok.

If I publish the release, we will end up with 3 releases.

One solution could be to delete the one I created and release the one created by travis (created by your username [please fix this], that has the files attached).

Or, I could publish the one that has my name (the only one I edited), and move the files from the Draft one (that was created by travis, using your username).

What do you suggest?

[philwhineray]

So you are saying you edited "my" release and it created a separate one? That sounds like a problem with github to me. I see the prior release has your edited content but my name on it.

The draft and files are uploaded with whatever credential is stored in the travis build setup. You can replace it with your own if you want to have your name on it - I can't that for you, but the necessary notes are all in the infrastructure readme and I can help you with any problems.

[ktsaou]

So you are saying you edited "my" release and it created a separate one?

no, I edited the v1.7.0 tag that was created by github and it created a new one.

ok, I'll delete my draft and use yours. Then I'll deal with the travis issue.

[ktsaou]

ok released it. Can you point the docs for changing the username on travis for releases?

[philwhineray]

That is not the behaviour that I think I have seen before.

Did you do the edit before travis had done its upload? That might have confused some scripts I guess.

There is a section on travis integration with pointers to the travis docs here:

https://github.com/firehol/infrastructure/blob/master/README.md

The formatting looks messed up it is probably best to get the text version.

Really short version, you need to get the Travis CI command line client and run travis setup releases from the repo. You will need to update various keys that you share with travis itself.

[ktsaou]

Did you do the edit before travis had done its upload? That might have confused some scripts I guess.

hm... probably. I remember thinking "ok, lets do something while travis do this", so I might have...

[ktsaou]

Really short version, you need to get the Travis CI command line client and run travis setup releases from the repo. You will need to update various keys that you share with travis itself.

ok, I'll give it a shot...

[philwhineray]

hm... probably. I remember thinking "ok, lets do something while travis do this", so I might have...

The bit that does the upload looks like standard Travis functionality (i.e. I can't change it) so probably worth waiting next time to see if that helps.

[philwhineray]

ok, I'll give it a shot...

It was somewhat troublesome to get right IIRC, I will help as best I can but recommend reading around the topic a bit first.

[ktsaou]

ok. thanks.

Signed the release files too. So, netdata v1.7 is officially out!

[philwhineray]

Nice.

As part of updating the secrets and so on for the travis build, you will need to re-encrypt and store the private ssh key that is used to upload "master" branch builds to firehol.org. It is .travis/travis_rsa.enc in the netdata repo.

I put the unencrypted key (no password) here: costa@firehol.org/travis_rsa - if you could grab it and delete it once you are happy its working (should allow ssh to travis@firehol.org).

[philwhineray]

Sorry, that should be costa@firehol.org:travis_rsa (as in scp from your home dir).