Is the project dead?

[Elettronik]

Hello, I'm looking to use this project on some k8s but this project seems dead for about a year.

Is there any people from Heptio Labs taking care of this project?

I see a the current moment 11 PR opened during a year without a single answer from owners, and 30 issues.

Please says if the project is dead, or will be some housekeeping.

[alok87]

@timothysc @jbeda

[alok87]

We are using it and it works well for us in porting the logs to s3.

[Ghazgkull]

The description of the heptiolabs org (https://github.com/heptiolabs) now reads: "Former home of experimental projects from Heptio"

I wonder if these projects had to be abandoned when Heptio was bought by VMware.

[varac]

Yes, an update on this would be good.

[Elettronik]

Let me do the Devil's Advocate.

20 days - no official answers, no activities on the project.

[MatteoJoliveau]

I agree this project is probably on hold, if not abandoned. Would love to hear from VMWare/former Heptio folks if they plan to pick it up or donate the project to the community, or if we should just fork it/reimplement it.

[aantn]

This does indeed seem dead. I need the functionality that eventrouter provides for another project I'm working on. I'm going to be working on this fulltime for the near future. Please feel free to contact me if you need support/help.

I've decided to base my efforts on Kubewatch and not EventRouter. Kubewatch is a little more maintained, but it also seems somewhat dormant since the VMWare-Bitnami acquisition. Kubewatch doesn't yet stream all the events that EventRouter does, so I'll start by adding all event types to the Kubewatch output. Feel free to let me know which other EventRouter features you need in Kubewatch.

VMWare folks - I would love to merge my changes to Kubewatch/Eventrouter upstream. If not, I suppose I'll fork it, but that isn't my preference.

@MatteoJoliveau @Elettronik what are your needs beyond what Kubewatch/EventRouter currently provide? How are you interested in using them?

[MatteoJoliveau]

@aantn that seems like great news! Thank you for being willing to keep development going at least for now. I'll try to see if we can help as well.

For us it's mainly getting Kubernetes events into something Grafana can display. Using EventRouter we were going to have Loki and Promtail scrape up the logs from stdout/stderr and then have them streamed to Grafana as JSON lines. This way developers could select the event stream they wanted using the same query language they already use for selecting application log streams. e.g. { app_kubernetes_io_name="my-app", stream="events" }(or something along those lines).

But really any solution that allows us to collect and store events for later querying will do.

[aantn]

Got it. I'm working on a more generic platform for running Python code as a result of Kubernetes changes/Prometheus alerts and automating common responses. It will be open sourced eventually, but for now it is still in private beta. One easy use-case, for example, is to add annotations to grafana so that you can see exactly when a new version of an application was updated and can quickly eyeball the difference in the performance before and after.

For your use case, you're only interested in forwarding actual Kubernetes Events, right? In other words, if a pod is created/modified/dies, you don't need to forward that, but if there is a CrashLoopBackOff then you do need to forward it. Is that correct?

[MatteoJoliveau]

Correct. The reason being that, especially now with Operators, k8s Event objects are a very useful tool to track changes and issues in k8s resources. Currently we have logs, metrics and traces centrally accessible in Grafana, but if a developer wants to debug a crash loop they have to manually run kubectl describe. Having events stored in Loki would allow for quick querying and alerting over them using common tools devs are already used to

[aantn]

Great, I'll update you when I have something you can use for that purpose.

[MatteoJoliveau]

Looking forward to it, thanks @aantn!

[antoniocascais]

Maybe this is a bit OT, but... Is it possible to add metrics to the app? For example, first thing that pops into my mind: adding metrics for probe failures.

Asking this because it would be nicer to have graphs around metrics in grafana than around logs. But maybe this is out of scope :D

[aantn]

Maybe this is a bit OT, but... Is it possible to add metrics to the app? For example, first thing that pops into my mind: adding metrics for probe failures.

Asking this because it would be nicer to have graphs around metrics in grafana than around logs. But maybe this is out of scope :D

Yeah, this is the type of thing I'm working on. Can you help me understand exactly what you would like to achieve? I think you can already get probe failure metrics from kubelet into prometheus. Do you want to enrich grafana with extra info? To run some remediation or enrichment steps?

[aantn]

This while conversation is a little off topic, I guess. I think the eventrouter community could benefit from another open source project that is the spiritual successor of eventrouter, but if the discussion here bothers someone then let me know and I'll take it elsewhere.

[antoniocascais]

Maybe this is a bit OT, but... Is it possible to add metrics to the app? For example, first thing that pops into my mind: adding metrics for probe failures. Asking this because it would be nicer to have graphs around metrics in grafana than around logs. But maybe this is out of scope :D

Yeah, this is the type of thing I'm working on. Can you help me understand exactly what you would like to achieve? I think you can already get probe failure metrics from kubelet into prometheus. Do you want to enrich grafana with extra info? To run some remediation or enrichment steps?

I was not aware of such metrics! Thank you very much :D

[aantn]

I'm close to releasing my open source project which supports many of the features that people want added to eventrouter. Can everyone on this thread let me know what they're actually using eventrouter for today or what they want to use it for? Are you sending events to slack for online notifications? Are you just logging changes to ELK so that you can troubleshoot when something goes wrong? What use cases does event router solve for you?

[varac]

@aantn Good to hear! I'm using eventrouter only for exporting k8s events to loki (and grafana), using promtail which tails the eventrouter container logs. So the only feature I'd be interested in is having all k8s events in the (eventrouter|your project) container logs.

[aantn]

@varac do you have any interest in turning k8s events into grafana annotations? e.g. adding a dotted line to grafana whenever a deployment is updated and the image tags change? I'm using this to easily correlate upgrades w/ changes in CPU usage.

[aantn]

Something like this:

[varac]

@aantn That looks great, sure that would be a good feature.

[aantn]

Cool, I've already implemented that. I have a little more work before I release this, but it's coming along nicely. Let me know if there are more integrations you can think of which would be useful. I'm currently implementing two-way Slack integration. The typical usecase is something like this:

1. There is a prometheus alert (e.g. low disk space on a persistent volume)
2. The system sends a mesage to Slack with details and a recommended remediation (e.g. cleanup some logs and increase the volume size if it still is low on space)
3. You click a button in Slack approving that remediation.
4. The system receives your approval and executes some remediation commands

[cordoor]

I would like to use it to stream kubernetes events into a kafka topic so my system can make decisions based on some of these events (e.g. a job is finished, so do something).

[aantn]

@cordoor we can do that already. can you reach out to me privately to discuss in more detail (either aantny@gmail.com or on linkedin here: https://www.linkedin.com/in/natanyellin/)

[aakarshg]

hello @aantn I'm interested in your project.. Currently what my usecase involves is to stream changes occurring to a specific set of kubernetes objects such as pods that are part of a replicaset in a particular namespace into a kafka topic..

If you've a prototype or something in the works can you please point us to that so that we can start using it and communicate feedback and hopefully submit patches ourselves

[aantn]

@aakarshg sure, send me an email and I'll send you the beta version.

[sl4dy]

We are migrating to https://github.com/openshift/eventrouter

[zswanson]

@aantn - ditto on being interested in a maintained replacement. The grafana integration for annotations sounds great.

[aantn]

@zswanson sorry about the delay! We've finally released the first version! https://docs.robusta.dev/master/

@aakarshg @cordoor @varac @antoniocascais @MatteoJoliveau might be relevant for you guys too

If anyone wants to discuss, we're on Slack and happy to add features for anything you need. (Or just open a github issue)

[hjet]

we've also reimplemented this as a Grafana Agent integration here - you can run a standalone Agent that only runs this integration as a drop-in eventrouter replacement. to learn more please see the docs.

for now it supports shipping events directly to a Loki-compatible sink and solves the "duplicate events" bug that occurs when you restart eventrouter. from there you can create dashboard annotations and "metrics from logs" directly in Grafana.

please file an issue in the Agent repo and ping me directly if you encounter any bugs or have feature suggestions!

you may also want to checkout @joe-elliott's diff logger as well!