Open kaage opened 4 months ago
If I understand what you are trying to achieve correctly, I would say you don't really need delete events. All you need is knowing that a certain row in your table does not exist anymore after a certain timestamp. That way you can show the same table at different points (intervals?) in time highlighting the diffs.
Deletes are more for a system that ingests events and reacts to a change (e.g: trigger an alarm) Keep in mind that gNMI sample subscriptions do not include delete notifications.
Postgres will be config heavy, you need to create a schema and explicitly map each received path to a key/field.
I think I forgot to add one detail. In my solution data collection is based on stream mode on-change rather than sample. So we don't get full table data every interval. Those tables in network devices are changing relativety rarely. It doesn't make sense to subscribe interval based.
So in order to tell rows in the table at given point in time, I need to:
Or is there way to solve the issue without delete events?
I see your point, an additional tag that says delete=true
can be added when a delete is received.
Added in v0.37.0.
You can use the attribute delete-tag: xyz
under your influxdb output.
gNMIc will create entries with tag xyz: true
for delete notifications.
I tried this but for some reason when delete-tag
is in the output (gnmic.yaml):
-- clip --
outputs:
influxdb-output:
type: influxdb
url: http://influxdb:8086
bucket: telemetry
token: <removed>
batch-size: 1000
flush-timer: 10s
org: acme
override-timestamps: true
delete-tag: deleted
event-processors:
- remove_openconfig_origin
-- clip --
Nothing gets written into InfluxDB. Not even additions. gnmic logs look like this:
2024/07/03 08:11:22.082512 [gnmic] version=0.37.0, commit=05a3e785, date=2024-05-13T23:27:31Z, gitURL=https://github.com/openconfig/gnmic, docs=https://gnmic.openconfig.net
2024/07/03 08:11:22.082523 [gnmic] using config file "/app/gnmic.yaml"
2024/07/03 08:11:22.082598 [config] validating processor "remove_openconfig_origin" config
2024/07/03 08:11:22.082689 [gnmic] starting output type influxdb
2024/07/03 08:11:22.082842 [gnmic] added event processor 'remove_openconfig_origin' of type=event-strings to output
2024/07/03 08:11:22.082923 [influxdb_output:influxdb-output] initialized influxdb client: {"Cfg":{"URL":"http://influxdb:8086","Org":"acme","Bucket":"telemetry","Token":"<removed>","BatchSize":1000,"FlushTimer":10000000000,"UseGzip":false,"EnableTLS":false,"HealthCheckPeriod":0,"Debug":false,"AddTarget":"","TargetTemplate":"","EventProcessors":["remove_openconfig_origin"],"EnableMetrics":false,"OverrideTimestamps":true,"TimestampPrecision":"","CacheConfig":null,"CacheFlushTimer":0,"DeleteTag":"deleted"}}
2024/07/03 08:11:22.082961 [influxdb_output:influxdb-output] starting worker-0
2024/07/03 08:11:22.082986 [gnmic] queuing target "clab-tellab-xrd-telemetry-rtr1"
2024/07/03 08:11:22.082991 [gnmic] subscribing to target: "clab-tellab-xrd-telemetry-rtr1"
2024/07/03 08:11:22.082994 [gnmic] starting target "clab-tellab-xrd-telemetry-rtr1" listener
2024/07/03 08:11:22.085245 [gnmic] target "clab-tellab-xrd-telemetry-rtr1" gNMI client created
2024/07/03 08:11:22.085251 [gnmic] sending gNMI SubscribeRequest: subscribe='subscribe:{subscription:{path:{elem:{name:"Cisco-IOS-XR-ip-rib-ipv4-oper:rib"} elem:{name:"vrfs"} elem:{name:"vrf"} elem:{name:"afs"} elem:{name:"af"} elem:{name:"safs"} elem:{name:"saf"} elem:{name:"ip-rib-route-table-names"} elem:{name:"ip-rib-route-table-name"} elem:{name:"routes"}} mode:ON_CHANGE} encoding:PROTO}', mode='STREAM', encoding='PROTO', to clab-tellab-xrd-telemetry-rtr1
I'm trying to build solution based for gnmi data which is more like table formatted than metrics. This type of data can be routing table, ARP table, MAC address table etc.
In the first phase trying present the data in Grafana at selected point in time as a table. Later I will show difference of same table between two points in time. This means I need to get add, change and delete events to the database.
I saw in the gnmic changelog following:
Is this feature related to what I'm looking for? I updated gnmic and tried to see how delete events are written in to InfluxDB but couldn't find them. How it is supposed to work?
Ps. I have demostrated table and diff visualization with Grafana & Postgres. Not sure if same will work with InfluxDB (InfluxQL or Flux). Any plans to have Postgres output for gnmic?