barrucadu / lainonlife

RIP lainchan radio, taken out by HDD failure.
https://lainon.life
MIT License
54 stars 12 forks source link

Incorrect Peak Listeners Count #36

Closed pussinboot closed 7 years ago

pussinboot commented 7 years ago

Something's (potentially) up on icecast's end, stepped through check_status in radio.js and it's working as it should but the listener_peak from status-json.xsl doesn't match what the graph dashboard says.

from lainon life

vs

from graph page

barrucadu commented 7 years ago

If you fiddle with the graph axes, you can see that there actually were 243 listeners at one point. It was weird, so I looked into it and it was hundreds of connections from VLC on the same IP over about a second.

Although, 243 is still pretty far off 328, so there is an issue here, which is in this dubious bit of code:

            // Assume that the listeners of the ogg and mp3 streams
            // are disjoint and just add them.  Bigger numbers are
            // better, right?
            if (source.server_name !== undefined){
                let sname = source.server_name.substr(0, source.server_name.length - 6);
                if(sname == channel || sname == "[mpd] " + channel) {
                    listeners     += source.listeners;
                    listenersPeak += source.listener_peak;
                    description    = source.server_description;
                }
            }

So the deeper issue is that the peak listeners of the mp3 and the ogg streams are being summed, when they shouldn't be.

One fix would be to have the backend keep track of the current number of listeners (updating every minute or so), and then change it to "peak listeners in the last 12 hours" or something, which would solve the summing problem and get rid of this artificially huge number.

pussinboot commented 7 years ago

ok, I see now that listener-peak is "The maximum number of concurrent listeners of the stream since it was started." according to icecast docs, which is not quite what I though it'd be.

since having the backend keep track of max for the past 12 hours will require an additional ajax call from radio.js either way, how do you feel about adding alerts to grafata and then querying for their evalData to get the value of maximum listener count for past 12 hrs? i may be wrong but seems like auth isn't required to access said endpoint (doesn't give me permission denied error like eg datasources).

i feel like if you already have something calculating statistics it makes sense to use it, but on the other hand if you want to keep the backend as something standalone that can be duplicated by others it needs to do be self-contained.

barrucadu commented 7 years ago

I think Grafana alerts are fine, though the path to query should be in the config file as a template parameter rather than hard coded. I added another dashboard for alerts (because I didn't want the ugly red border on the main dashboard), and there are now two:

barrucadu commented 7 years ago

It seems the EvalData field of peak listeners alert is often null, I don't really understand why as the configuration is basically identical to the current listeners alert. I'll check the docs later.

pussinboot commented 7 years ago

thnx, the js was a quick fix but i thought maybe it needed 12 hrs before it started reporting peak it doesnt give any sort of execution error either which is strange

barrucadu commented 7 years ago

I can't find anything about why the EvalData would be null, so I figured it's probably easier to just pull this data from InfluxDB directly. It seems pretty fast, this script runs in 0.3s (which is including the overhead of starting python and printing):

#!/usr/bin/env python3

import datetime
import influxdb

client = influxdb.InfluxDBClient(database='lainon.life')

startTime = (datetime.datetime.now() - datetime.timedelta(hours=12)).replace(microsecond=0)

for channel in ['cafe', 'cyberia', 'everything', 'swing']:
    max_res  = client.query('select max({}) from channel_listeners where time >= \'{}Z\''.format(channel, startTime.isoformat()))
    last_res = client.query('select {} as last from channel_listeners order by time desc limit 1'.format(channel))
    the_max  = max_res.get_points().__next__()['max']
    the_last = last_res.get_points().__next__()['last']
    print(channel, the_max, the_last)