search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
22.73k stars 3.31k forks source link

Netflow or/and pipeline (cisco logs) #1136

Open fredbcode opened 4 years ago

fredbcode commented 4 years ago

Hello,

I'm using ELK, logstash uses Netflow plugin for UDP logs from routers Maybe this is also something interesting for promtail

There is also a new ingest telemetry data from CISCO (pipeline), a plugin can works directly with prometheus or influxdb https://blogs.cisco.com/sp/introducing-pipeline-a-model-driven-telemetry-collection-service Unfortunately the plugin code is not updated https://github.com/cisco/bigmuddy-network-telemetry-pipeline

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

fredbcode commented 4 years ago

There is something planned in promtail for netflow ?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

randomchance commented 4 years ago

My understanding is that you would need to parse flows yourself (or use another tool) to reduce them down to labels | timestamp | text before they could be ingested.

My take on the current development focus is that the project is still fleshing out the core functionality of getting logs into Loki, and that formats that don't closely match "simple text logs" will probably end up in separate tools, similar to the different beats used by the ELK stack.

cyriltovena commented 4 years ago

We are focusing on other aspects but I’m happy to assist someone who wants to take a stab at this.

randomchance commented 4 years ago

Just noticed that you can ingest netflow files with the elastic FileBeat and have it output to rolling files, which promtail then ingests. I'm doing the same for windows logs and winlogbeat if it helps.

loganmc10 commented 3 years ago

This is generally already possible, I was able to get data from CloudFlare's goflow into Loki without any real difficulty.

There is still some pain points in terms of visualization though. There is no rate_over_time function to calculate the rate of an unwrapped metric.

There is also no way (that I can get working) in Grafana to show the top N series. Netflow generates a very high number of unique series, and I very quickly hit the Grafana Cloud limit of 500 series per query, with no clear way to select top N series

cstyan commented 7 months ago

It sounds like the original issue is already solved? If there's a secondary issue related to querying please open another issue.

Related to this issue in general, we're currently reevaluating promtails position as a project within Grafana Labs. Internally we're actually using the Agent for both metrics and logs collection at this point. Additionally, the agent team is more likely to have time to dedicate to your PR.

While we haven't made a formal decision yet, we expect in the near future that all new feature work will be done in the Agent's log collection pipelines rather than in Promtail.