Configuration sources

[sundarkvp]

Hi,

Other than Cassendra, do you have any plans to add support for other configuration sources (Ex. elasticsearch) ?

[jshaughn]

The 2.0 release (master branch) uses Infinispan with HibernateSearch for persistence, removing the need for Cassandra. There is no plan to use any other backend. As for Elasticsearch, there is an alerting integration with ES, you can look here for more: ES example.

[mumoshu]

@jshaughn Would you mind if I contributed an Elasticsearch backend?

It seemed approachable to me, as it seems like a matter of implementing an alternative backend organized similar to ispn's. But I wasn't sure supporting another backends is within the scope of this project.

Btw, thanks anyway for creating and maintaining a great project!

[mumoshu]

Ah, let me provide some more context. I was going to use ES as a backend for all the events, logs, metrics and traces gathered from my K8S pods and deployments, services, and so on. Hawkular Alerts would take the role of alerting on any data flowing in ES.

I think treating an distributed search engine like Elasticsearch as an general-purpose datastore has been long considered an anti-pattern.

But the situation has started to change recently. You may already know but projects like Jaeger and Zipkin started to support ES as a backend. Elastic has started to treat ES as a distributed time-series DB while introducing Metricbeat. So, I found an use-case and a reason to use ES as a backend DB. Only thing I was unsure was if it fits within the scope of Hawkular Alerts.

[lucasponce]

@mumoshu project is open and any contribution is welcome :-).

I understand that the purpose to support ES as a backend is to unify, and not mix backends, the master branch (version 2.x) had that in purpose too, and the main reason to use ISPN is that we will use it in any case as it is necessary to give distributable features to Hawkular Alerting, as it is built-in so, you don't need external dependencies to plug your backend, and you only need to provide a folder to store the data files.

This is indenpendent that Hawkular Alerting can be integrated with ES or other source to receive events and notifications, but I guess that a ES backend would add a little overhead as you need to access to an external system (ES) to provide the internal info of Hawkular Alerting.

Note, that with the plugins you can store all events/alerting on ES also, without having to provide a new backend.

Having said that, if you want to work on a ES backend, don't hesitate to start something, we can create a different branch to see the evolution of this approach and also it can give you some hints about the design results. Note that in any case, you would need ISPN as it is used to provide clustering cappabilities to the engine.

Also, another good reason to use ISPN as a backend is to reduce the dependencies of pieces as we can use it here as an internal library, instead of a new server you need to add a container for your pod.

[mumoshu]

@lucasponce Thanks for the quick response. And sorry if my question sounded strange! I tend to ask it before contributing not to encourage scope creep.

to unify, and not mix backend

Exactly. That's also my purpose.

guess that a ES backend would add a little overhead as you need to access to an external system (ES) to provide the internal info of Hawkular Alerting.

Aha! I think I misunderstood what the current implementation was doing when I initially read issues and your comments.

(Disclaimer; I'm an Infinispan newb 😉) I guess - Under the hood, Hawkular Alerts uses Infinispan as a embedded distributable k-v store with indexing/searching support. It uses file-store for the k-v store part, and locally persisted lucene indices for the indexing/searching part. They are powered by file system based cache store and Directory Provider for Hibernate Search respectively.

So, Hawkular Alerts distributes kv pairs and indices across its cluster members, as far as I could see from the configuration file. There's no dependency to external kv-store or search-engine, which is nice. Am I following it correctly?

Then, yes - the ES backend should add a little overheard compared to it.

Note, that with the plugins you can store all events/alerting on ES also, without having to provide a new backend.

Made a lot of sense. So, I'd need to implement an ES-based cache store implementation as there's no such thing exists today. For the indexing/searching, I'd need to switch from Infinispan Directory Provider to the Elasticsearch one. Finally, I'd add the jars of the twoto Hawkular Alerts' classpath/config to externalize all the states formerly persisted on each cluster member.

Note that in any case, you would need ISPN as it is used to provide clustering cappabilities to the engine.

Noted. Thanks 👍

Also, another good reason to use ISPN as a backend is to reduce the dependencies of pieces as we can use it here as an internal library, instead of a new server you need to add a container for your pod.

Makes sense, too!

Just wanted to make sure, but for me externalizing all the states to an Elasticsearch/RDB/etc seemed to have two little benefits. One is that you can backup/restore the whole cluster state with used tools (ES snapshots, mysql dumps, ...). Another is that you can ensure your Hawkular Alerts deployment to be almost stateless, which is nice for operational perspective. You can quickly move Hawkular Alerts between K8S clusters without worrying about a bit lengthy downtime due to persistent volumes are detached/reattached between clusters.

// It isn't actually true as Hawkular Alerts has local cache provided by embedded Infinispan(=state), but seems better than self-managing states in disks.

I guess there's no winner. I just wanted to make it clear about my use-case and assumption. But any correction or comments are welcomed.

[lucasponce]

One is that you can backup/restore the whole cluster state with used tools (ES snapshots, mysql dumps, ...). Another is that you can ensure your Hawkular Alerts deployment to be almost stateless, which is nice for operational perspective.

Why don't work at persistence volumes levels here ?

Unless you want to backup data at App level, I think the easiest way would be to use same strategy you should use to your data today with ES (at persistence volumes) as it uses a similar way to store data configuring the data folders where you want.

Also, you can perform similar dumps in Hawkular Alerting with the export/import endpoints.

I'm just trying to understand the motivation :-), but in any case, if you want to provide a new backend option, no problem from my side.

I guess best option would be to add a new impl (elasticsearch) at this level

https://github.com/hawkular/hawkular-alerts/tree/master/engine/src/main/java/org/hawkular/alerts/engine/impl

Try to understand that code and how it is invoked, we can create a experimental branch for you.

But as it looks like a similar experience we had with Cassandra in 1.x version we suggest if you can think about to point your files into your persistence volumes, as we think you'd save time and would reach a similar result.

[mumoshu]

Thank you so much for your kindness ☺️ I do want to avoid implementing or encouraging meaningless solution. So, your comments are greatly appreciated!

Why don't work at persistence volumes levels here ?

Actually I have never considered it as the first option, probably due to my limited experience. Let me elaborate a bit more.

I try to use managed services from AWS whereas possible. To name a few, RDS for MySQL, AWS ES or ElasticCloud for Elasticsearch, so that I can outsource part of operational duty otherwise I had required to do my own.

If I needed to take a volume-level backup of Hawkular Alerts, I'd like to take a consistent one. An EBS snapshot is meant to be point-in-time - that's ok. But the application, in this case Hawkular Alerts, would need to flush all the buffered changes to local volumes and stops further writing until the EBS snapshot is taken. I'm thinking of a snapshot taken when a transaction was half-wrote to a volume. It is consistent at volume-level, but seemed inconsistent at application-level.

// Note that, as this is my theory, there may be incorrect assumptions from Hawkular's perspective. I see you have bunch of start/endBatch over the code-base but I wasn't sure if it automatically mean that every volumesnapshot or hawkular export is guaranteed to be consistent.

Perhaps you have a solution about this application-level consistent snapshotting problem in your organization?

Anyway, I personally had some experience in managing ES, and a managed service of a choice. At this point, I decided I'd prefer externalizing all the states to ES in regard to the backup/restore usecase.

But hey, the export/import facility you kindly pointed out sounds like an another solution to me, if it helps me to easily backup/restore Hawkular Alerts :) It may be a happy news as I don't need to implement anything other than some configuration files and Dockerfile and yamls then.

[jshaughn]

Hi mumoshu, Lucas has given you a lot of good information. Because supporting a new backend is not trivial, we just want to make sure it is worth your time. Our change from Cassandra to Infinispan was great simplifier, although ES would likely be an easier integration than Cassandra because the schema and query handling is not so strict. It is too bad ISPN does not natively support ES as a persistence option.

You will not find true consistency guarantees in the APIs.

As Lucas mentions, you do have the option to store all events and actions into ES with the current plugin. And although all of the trigger and action definitions would remain in ISPN, they can be imported and exported as JSON with the existing API.

But you are free to attempt an ES backend if you like.

[mumoshu]

@jshaughn I appreciate your detailed explanation and kind consideration.

I believe that I now better understand the context and the design decision of the project, and the consistency guarantee provided by APIs.

I'd attempt the ES backend implementation if it turned out necessary, but I really don't like to stick with my idea without reasons. Now, I have no specific reason to do so anymore, given the great insight provided by you two. More specifically, my use-case seems to be achievable with just export/import APIs with some consideration on consistency guarantee.

You saved my life!

[jshaughn]

I'm glad we are able to help you work in the most productive way. Thanks for your interest in the project.