search

sapcc / octavia-f5-provider-driver

Apache License 2.0
15 stars 2 forks source link

Listener status-update reports wrong connection-full metric #133

Closed notandy closed 3 years ago

notandy commented 3 years ago

The octavia-f5-status-manager fetches statistics of every listener, and is supposed to report if the listener is satured (Max Connections allowed == Current Connections)

The implementation wrongly assumed that the clientside.MaxConns Metric is reporting back the maximum allowed conections, but it's just the "highest" number of connections F5 observed over all time.

https://github.com/sapcc/octavia-f5-provider-driver/blob/master/octavia_f5/controller/statusmanager/status_manager.py#L265-L267

Example F5 stats response for virtual object:

{
    "actualPvaAccel": {
        "description": "partial"
    }, 
    "clientside.bitsIn": {
        "value": 123000991688
    }, 
    "clientside.bitsOut": {
        "value": 916751184
    }, 
    "clientside.curConns": {
        "value": 2
    }, 
    "clientside.evictedConns": {
        "value": 0
    }, 
    "clientside.maxConns": {
        "value": 2
    }, 
    "clientside.pktsIn": {
        "value": 11854691
    }, 
    "clientside.pktsOut": {
        "value": 2430974
    }, 
    "clientside.slowKilled": {
        "value": 0
    }, 
    "clientside.totConns": {
        "value": 27
    }, 
    "cmpEnableMode": {
        "description": "all-cpus"
    }, 
    "cmpEnabled": {
        "description": "enabled"
    }, 
    "csMaxConnDur": {
        "value": 46801907
    }, 
    "csMeanConnDur": {
        "value": 24754300
    }, 
    "csMinConnDur": {
        "value": 78399
    }, 
    "destination": {
        "description": "10.180.0.7%2123:514"
    }, 
    "ephemeral.bitsIn": {
        "value": 0
    }, 
    "ephemeral.bitsOut": {
        "value": 0
    }, 
    "ephemeral.curConns": {
        "value": 0
    }, 
    "ephemeral.evictedConns": {
        "value": 0
    }, 
    "ephemeral.maxConns": {
        "value": 0
    }, 
    "ephemeral.pktsIn": {
        "value": 0
    }, 
    "ephemeral.pktsOut": {
        "value": 0
    }, 
    "ephemeral.slowKilled": {
        "value": 0
    }, 
    "ephemeral.totConns": {
        "value": 0
    }, 
    "fiveMinAvgUsageRatio": {
        "value": 0
    }, 
    "fiveSecAvgUsageRatio": {
        "value": 0
    }, 
    "mr.msgIn": {
        "value": 0
    }, 
    "mr.msgOut": {
        "value": 0
    }, 
    "mr.reqIn": {
        "value": 0
    }, 
    "mr.reqOut": {
        "value": 0
    }, 
    "mr.respIn": {
        "value": 0
    }, 
    "mr.respOut": {
        "value": 0
    }, 
    "oneMinAvgUsageRatio": {
        "value": 0
    }, 
    "status.availabilityState": {
        "description": "available"
    }, 
    "status.enabledState": {
        "description": "enabled"
    }, 
    "status.statusReason": {
        "description": "The virtual server is available"
    }, 
    "syncookie.accepts": {
        "value": 0
    }, 
    "syncookie.hwAccepts": {
        "value": 0
    }, 
    "syncookie.hwSyncookies": {
        "value": 0
    }, 
    "syncookie.hwsyncookieInstance": {
        "value": 0
    }, 
    "syncookie.rejects": {
        "value": 0
    }, 
    "syncookie.swsyncookieInstance": {
        "value": 0
    }, 
    "syncookie.syncacheCurr": {
        "value": 0
    }, 
    "syncookie.syncacheOver": {
        "value": 0
    }, 
    "syncookie.syncookies": {
        "value": 0
    }, 
    "syncookieStatus": {
        "description": "not-activated"
    }, 
    "tmName": {
        "description": "/net_4e3f0585_ba1d_40c1_8dec_146aef534de9/lb_b175b3ea-35ba-426e-bee6-f287a29910f5/listener_c763a4ed-7310-4cd5-b0dc-8ca83f4a32b6"
    }, 
    "totRequests": {
        "value": 0
    }
}

Proposal:

Evaluate if there is another metric usable to detect the "connection full" or - if thats not easily possible: remove status = 'FULL' report.

BenjaminLudwigSAP commented 3 years ago

As @m-kratochvil already mentioned, there is no metric for connection saturation. We will have to compare current connections to the maximum. However, there is also no metric for the maximum. The HM will have to either ask the DB or the current declaration on the device (which would mean another roundtrip)

BenjaminLudwigSAP commented 3 years ago

I have suggested a variant of calculating status=FULL by database in #135

BenjaminLudwigSAP commented 3 years ago

135 has been merged and deployed.